Smoke tests fail with SDK version 2.13.7

[hab6]

About You:
Name: Michael Habiger
Company: Actian Corporation

Your question:
I am encountering a smoke test failure every time when attempting to run the TDVT tests using the latest 2.13.7 test suite.

My Environment:
Microsoft Windows [Version 10.0.19045.3930]
Tableau Desktop 2023.3.3 (20233.24.0202.1238)
Python 3.10.7

Package    Version
---------- -------
defusedxml 0.7.1
pip        24.0
setuptools 63.2.0
tdvt       2.13.7

Result:
When executing python -m tdvt.tdvt run mhds (with or without the --verify option), the execution always reports a smoke test failure :

2 smoke test(s) ran. 0 smoke tests disabled.
Smoke tests ran in 10.77 seconds.
1 smoke test(s) failed. Please check logs for information.
Tests for the following data source(s) will not be run: mhds

There are no errors in tdvt_log_combined.txt and I see no relevant errors in other files/logs.

Using SDK version 2.13.4 works:
Running the test suite with an exact same setup except using SDK version 2.13.4 instead of 2.13.7 allows the test suite to run properly.

Also, if the *.tds files that fail under version 2.13.7 are moved and used unaltered in an equivalent setup with version 2.13.4, the tests run successfully with a 98% pass rate:

Test Count: 872 tests
    Passed tests: 856
    Failed tests: 16
    Tests run: 872

Check of the SDK documentation Troubleshooting section:
I have reviewed the Frequently found issues and troubleshooting section of the TDVT documentation.

• The TDS files open perfectly fine in Tableau without being prompted for credentials or any additional fix.
• I am using a packaged and signed .taco file that is located in the "My Tableau Repository\Connectors" folder. The same .taco file is used with SDK 2.13.4 which runs successfully.
• The file tdvt_log_combined.txt reports no problems locating the tabquery tool.

INFO Found tabquery at [C:\Program Files\Tableau\Tableau 2023.3\bin\tabquerytool.exe]

Files:
tdvt_log_combined.txt
cast_calcs.mhds.tds.txt
Staples.mhds.tds.txt

[lukewrites]

Hi @hab6
One of the changes in the latest release was updating an expected for a smoke test; if you update to the latest release of Tableau (from the tab query path it looks like you're using 2023.3) it should go back to passing.

You can also ignore the failing smoke test and run all tests using python -m tdvt.tdvt run {ds name} --force-run.
If that doesn't work,
python -m tdvt.tdvt --verbose run {ds name} --verify
will produce a verbose tdvt_log_combined file for more debugging.

[lukewrites]

Sorry, that should have been --force-run, typing too fast 😄 I also fixed it above.

[hab6]

Thanks for the suggestions @lukewrites

The SDK repository README states that the latest, non-beta version of Tableau Desktop/Server is 2023.3, which is the version used when I originally encountered the problem (and still do).

Per your recommendation to update to the latest release, I found and installed Tableau Desktop version 2024.1 and the smoke test still fails. Verifying the versions used for this latest test:

Tableau Desktop: ./tds/cast_calcs.mhds.tds:  build 20241.24.0208.0337
Tableau Desktop: ./tds/Staples.mhds.tds:     build 20241.24.0208.0337
Connector SDK:   ./tdvt_log_combined.txt:    INFO TDVT version: 2.13.7

Using --force-run seems to work around the problem so that the TDVT test run ignores the failing smoke test (although the pass/fail rate is slightly worse than with SDK version 2.13.4).

C:\tdvtTest\wksp> python -m tdvt.tdvt run mhds --force-run
.....
2 smoke test(s) ran. 0 smoke tests disabled.
Smoke tests ran in 12.69 seconds.
1 smoke test(s) failed. Please check logs for information.
.....
Test Count: 873 tests
    Passed tests: 853
    Failed tests: 20
    Tests run: 873
    Disabled tests: 0
    Skipped tests: 0

The failing smoke test looks to be StaplesConnectionTestmhds, which appears to be due to trying to match strings to numbers (although the values are the same). Here is the DIFF file:
actual.setup.staples.connection.test_diff.xml.txt

[hab6]

Actian internal tracking issue II-13736

[lukewrites]

Sorry, Michael, I was wrong in the above. Are the newly failing tests in the lod.9_Fixed as Dimension and/or lod.calc.1.TopN?

You can ignore the failing smoke tests since those are caused by the new expected file.

[hab6]

(On my system) Here are the tests that succeed with SDK 2.13.4 and fail with 2.13.7

Test Name	Test Case	Comment
staples.connection.test	logicaltests/ setup/ connection_test/ setup.staples.connection.test	Smoke test: Succeeds with 2.13.4 Actual/Expected mismatch with 2.13.7
string.space.empty	SPACE(0)	New in 2.13.7. Did not exist in 2.13.4.
lod.9_Fixed as Dimension	logicaltests/ setup/ lod/ setup.lod.9_Fixed as Dimension	Succeeds with 2.13.4 Actual/Expected mismatch with 2.13.7
lod.calc.1.TopN	logicaltests/ setup/ lod/ setup.lod.calc.1.TopN	Succeeds with 2.13.4 Actual/Expected mismatch with 2.13.7

[lukewrites]

@hab6 the test-fix branch provides the old expecteds for the LOD tests and connection test, if you'd like to try it out. This was caused by a bug fixed in an upcoming point release. When the new release comes out I'll ping the thread and also remove the alternate expecteds.

The string.space.empty test is new; that's a test failure to verify.

Thanks!