[Unit Tests] All tests but one are passing whether project is built from CLI or with Qt Creator

[holly564]

Hello,

I am trying to run the unit tests with the command call: ctest -j4 --output-on-failure
But none of them passed
The output error suggests that there is a missing config
Test not available without configuration. (Missing "-C "?)

Environment: Windows 10 Pro

C:\Users\Holly\MuseScore\builds\Win-Qt5.15.2-msvc2019_64-VS16-RelWithDebInfo>ctest -j4 --output-on-failure
Test project C:/Users/Holly/MuseScore/builds/Win-Qt5.15.2-msvc2019_64-VS16-RelWithDebInfo
Start 1: global_tests
Test not available without configuration. (Missing "-C "?)
1/17 Test #1: global_tests .....................***Not Run 0.00 sec
Start 2: accessibility_tests
Test not available without configuration. (Missing "-C "?)
2/17 Test #2: accessibility_tests ..............***Not Run 0.00 sec
Start 3: audio_test
Test not available without configuration. (Missing "-C "?)
3/17 Test #3: audio_test .......................***Not Run 0.00 sec
Start 4: draw_tests
Test not available without configuration. (Missing "-C "?)
4/17 Test #4: draw_tests .......................***Not Run 0.00 sec
Start 5: mpe_test
Test not available without configuration. (Missing "-C "?)
5/17 Test #5: mpe_test .........................***Not Run 0.00 sec
Start 6: ui_tests
Test not available without configuration. (Missing "-C "?)
6/17 Test #6: ui_tests .........................***Not Run 0.00 sec
Start 7: diagnostics_tests
Test not available without configuration. (Missing "-C "?)
7/17 Test #7: diagnostics_tests ................***Not Run 0.00 sec
Start 8: engraving_tests
Test not available without configuration. (Missing "-C "?)
8/17 Test #8: engraving_tests ..................***Not Run 0.00 sec
Start 9: iex_bb_tests
Test not available without configuration. (Missing "-C "?)
9/17 Test #9: iex_bb_tests .....................***Not Run 0.00 sec
Start 10: iex_braille_tests
Test not available without configuration. (Missing "-C "?)
10/17 Test #10: iex_braille_tests ................***Not Run 0.00 sec
Start 11: iex_bww_tests
Test not available without configuration. (Missing "-C "?)
11/17 Test #11: iex_bww_tests ....................***Not Run 0.00 sec
Start 12: iex_capella_tests
Test not available without configuration. (Missing "-C "?)
12/17 Test #12: iex_capella_tests ................***Not Run 0.00 sec
Start 13: iex_midi_tests
Test not available without configuration. (Missing "-C "?)
13/17 Test #13: iex_midi_tests ...................***Not Run 0.00 sec
Start 14: iex_musicxml_tests
Test not available without configuration. (Missing "-C "?)
14/17 Test #14: iex_musicxml_tests ...............***Not Run 0.00 sec
Start 15: iex_guitarpro_tests
Test not available without configuration. (Missing "-C "?)
15/17 Test #15: iex_guitarpro_tests ..............***Not Run 0.00 sec
Start 16: plugins_tests
Test not available without configuration. (Missing "-C "?)
16/17 Test #16: plugins_tests ....................***Not Run 0.00 sec
Start 17: project_test
Test not available without configuration. (Missing "-C "?)
17/17 Test #17: project_test .....................***Not Run 0.00 sec

0% tests passed, 17 tests failed out of 17

Total Test time (real) = 0.48 sec

The following tests FAILED:
1 - global_tests (Not Run)
2 - accessibility_tests (Not Run)
3 - audio_test (Not Run)
4 - draw_tests (Not Run)
5 - mpe_test (Not Run)
6 - ui_tests (Not Run)
7 - diagnostics_tests (Not Run)
8 - engraving_tests (Not Run)
9 - iex_bb_tests (Not Run)
10 - iex_braille_tests (Not Run)
11 - iex_bww_tests (Not Run)
12 - iex_capella_tests (Not Run)
13 - iex_midi_tests (Not Run)
14 - iex_musicxml_tests (Not Run)
15 - iex_guitarpro_tests (Not Run)
16 - plugins_tests (Not Run)
17 - project_test (Not Run)
Errors while running CTest

Please know that I had previously ran the cmake clean command before launching the build for a second time
cmake -P build.cmake clean

I also updated the cmake file at the root of the project with:

set(ENV{QTDIR} "C:/Qt/5.15.2/msvc2019_64")
set(CMAKE_GENERATOR "Visual Studio 16 2019")
list(APPEND CONFIGURE_ARGS "-DBUILD_UNIT_TESTS=ON")
list(APPEND CONFIGURE_ARGS "-DCMAKE_BUILD_TYPE=RelWithDebInfo")
list(APPEND CONFIGURE_ARGS "-DMUE_COMPILE_BUILD_64=ON")
list(APPEND CONFIGURE_ARGS "TARGET_PROCESSOR_BITS=64")
list(APPEND CONFIGURE_ARGS "-DCMAKE_INSTALL_PREFIX=C:/Users/Holly/MuseScore/builds/Win-Qt5.15.2-msvc2019_64-VS16-RelWithDebInfo/install")
list(APPEND CONFIGURE_ARGS "VS_INSTALL_DIR=C:/Program Files (x86)/Microsoft Visual Studio/2019/Community")

Still, without success.
Can't find anything related to this on the wiki
Any ideas?

Thanks

[holly564]

Ah, I added a config directly to ctest and some tests are passing :-)

C:\Users\Holly\MuseScore\builds\Win-Qt5.15.2-msvc2019_64-VS16-RelWithDebInfo>ctest -C "RelWithDebInfo"
Test project C:/Users/Holly/MuseScore/builds/Win-Qt5.15.2-msvc2019_64-VS16-RelWithDebInfo
Start 1: global_tests
1/17 Test #1: global_tests .....................***Failed 2.93 sec
Start 2: accessibility_tests
2/17 Test #2: accessibility_tests .............. Passed 2.24 sec
Start 3: audio_test
3/17 Test #3: audio_test .......................Exit code 0xc0000135
***Exception: 2.04 sec
Start 4: draw_tests
4/17 Test #4: draw_tests ....................... Passed 2.09 sec
Start 5: mpe_test
5/17 Test #5: mpe_test ......................... Passed 2.09 sec
Start 6: ui_tests
6/17 Test #6: ui_tests ......................... Passed 2.45 sec
Start 7: diagnostics_tests
7/17 Test #7: diagnostics_tests ................ Passed 4.13 sec
Start 8: engraving_tests
....
59% tests passed, 7 tests failed out of 17

Total Test time (real) = 200.70 sec

The following tests FAILED:
1 - global_tests (Failed)
3 - audio_test (Exit code 0xc0000135
)
8 - engraving_tests (Failed)
10 - iex_braille_tests (Failed)
12 - iex_capella_tests (Failed)
14 - iex_musicxml_tests (Failed)
15 - iex_guitarpro_tests (Failed)

[holly564]

verbose

[----------] 4 tests from Global_IO_IODeviceTests
[ RUN ] Global_IO_IODeviceTests.Open_ReadOnly
C:\Users\Holly\MuseScore\src\framework\global\tests\iodevice_tests.cpp(60): error: Death test: { size_t size = buf.write(wrba); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("size", "0", size, 0))) ; else ::testing::internal::AssertHelper(::testing::TestPartResult::kNonFatalFailure, "C:\Users\Holly\MuseScore\src\framework\global\tests\iodevice_tests.cpp", 60, gtest_ar.failure_message()) = ::testing::Message(); }
Result: failed to die.
Error msg:
[ DEATH ]
[ FAILED ] Global_IO_IODeviceTests.Open_ReadOnly (194 ms)
[ RUN ] Global_IO_IODeviceTests.Open_WriteOnly
C:\Users\Holly\MuseScore\src\framework\global\tests\iodevice_tests.cpp(85): error: Death test: { ByteArray rba = buf.readAll(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(rba.empty())) ; else ::testing::internal::AssertHelper(::testing::TestPartResult::kNonFatalFailure, "C:\Users\Holly\MuseScore\src\framework\global\tests\iodevice_tests.cpp", 85, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "rba.empty()", "false", "true").c_str()) = ::testing::Message(); }
Result: failed to die.
Error msg:
[ DEATH ]
C:\Users\Holly\MuseScore\src\framework\global\tests\iodevice_tests.cpp(90): error: Death test: { ByteArray rba = buf.read(4); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar_ = ::testing::AssertionResult(rba.empty())) ; else ::testing::internal::AssertHelper(::testing::TestPartResult::kNonFatalFailure, "C:\Users\Holly\MuseScore\src\framework\global\tests\iodevice_tests.cpp", 90, ::testing::internal::GetBoolAssertionFailureMessage( gtest_ar_, "rba.empty()", "false", "true").c_str()) = ::testing::Message(); }
Result: failed to die.
Error msg:
[ DEATH ]
C:\Users\Holly\MuseScore\src\framework\global\tests\iodevice_tests.cpp(96): error: Death test: { uint8_t d = 0; size_t s = buf.read(&d, 1); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal::EqHelper::Compare("s", "0", s, 0))) ; else ::testing::internal::AssertHelper(::testing::TestPartResult::kNonFatalFailure, "C:\Users\Holly\MuseScore\src\framework\global\tests\iodevice_tests.cpp", 96, gtest_ar.failure_message()) = ::testing::Message(); }
Result: failed to die.

[cbjeukendrup]

Those with "failed to die" are failing because they are supposed to hit an assertion failure, which only happens in "debug" builds.

About the other failures: I believe those are exactly the tests that depend on the diff utility. Do you have that in your PATH?

[holly564]

Thanks a bunch for your suggestions.

Updated the build_type to debug
Added diff utility to my path

Actual Result: All tests but one are passing :-)

The following tests FAILED:
3 - audio_test (Exit code 0xc0000135
)
Errors while running CTest

[holly564]

Where could I find the reference to Exit code 0xc0000135?
It is not inside audioerrors.h

Or better, how could I move forward? adding verbose switches is not enough to debug the audio_test run failure

c:\Users\Holly\MuseScore\builds\Win-Qt5.15.2-msvc2019_64-VS16-Debug>ctest -j 4 -C "Debug" --rerun-failed --output-on-failure --verbose --extra-verbose
UpdateCTestConfiguration from :C:/Users/Holly/MuseScore/builds/Win-Qt5.15.2-msvc2019_64-VS16-Debug/DartConfiguration.tcl
UpdateCTestConfiguration from :C:/Users/Holly/MuseScore/builds/Win-Qt5.15.2-msvc2019_64-VS16-Debug/DartConfiguration.tcl
Test project C:/Users/Holly/MuseScore/builds/Win-Qt5.15.2-msvc2019_64-VS16-Debug
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 3
Start 3: audio_test

3: Test command: C:\Users\Holly\MuseScore\builds\Win-Qt5.15.2-msvc2019_64-VS16-Debug\src\framework\audio\tests\Debug\audio_test.exe
3: Working Directory: C:/Users/Holly/MuseScore/builds/Win-Qt5.15.2-msvc2019_64-VS16-Debug/src/framework/audio/tests
3: Test timeout computed to be: 10000000
1/1 Test #3: audio_test .......................Exit code 0xc0000135
***Exception: 0.53 sec

Thank you!

[cbjeukendrup]

I've Google'd that error code, and it seems to mean that some DLL is not found. I'll try if I can find out which; you could also try something, namely compiling the project in Qt Creator and running the test in the debugger.