testsuite adjustments - running with analyzers / code instrumentation

[GitMensch]

It would be good to:

• double check that all kind of variable types (in the case of binary variables also all sizes) are used in the testsuite
• try to run the tests with an analyzer active; on GNU/Linux valgrind's memcheck is really useful; as an alternative run with code instrumentation, for GCC that's possibly passing CPPFLAGS="-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2" CFLAGS="-O -fstack-protector" to the configure line and as another instrumentation (guess that works with GCC and clang) link against -lasan -lubsan. MSVC should have some code instrumentation options, too.

Originally posted by @GitMensch in #121 (comment)

[mridoni]

In v1.0.20dev the test suite has been adjusted as follows.

• Runs on .Net Core 6.0 (both on Windows and Linux)
• It can be run from Visual Studio, where the results are integrated in the Test Explorer, or directly from the command line
• The XML configuration has been split into two files: a local configuration where general settings, DB drivers and credentials are configured, and a "test matrix" that lists the tests to be performed (usually a module to be preprocessed, compiled and run)
• The local configuration can contain a command to run the tests under valgrind (Linux) or DrMemory (Windows)
• A GitHub Actions workflow is provided to automate testing. For now this is being developed and used mostly locally, using act

[GitMensch]

Nice, I did not know about ACT.

... but it seems the issue is still open as https://github.com/mridoni/gixsql/blob/main/.github/workflows/gixsql-linux-test.yml does not use any instrumentation so far.

for GCC that's possibly passing CPPFLAGS="-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2" CFLAGS="-O -fstack-protector" to the configure line and as another instrumentation (guess that works with GCC and clang) link against -lasan -lubsan.

The Windows action seems to not run any tests via actions so far - does / could it?

The local configuration can contain a command to run the tests under valgrind (Linux) or DrMemory (Windows).

Hm, valgrind possibly takes too long in the CI (not sure, the flags/linking option above would be fine), how much overhead is DrMemory? Do you execute these locally?

And related: Did you run with test coverage so far?

[mridoni]

The action is actually run with valgrind and is not that slow (this is why I did not add FORTIFY_SOURCE, etc.); that is referenced here. I am currently nor running it automatically, because at this stage the packaging process still needs a lot of adjustments, so it doesn't make sense to run the whole suite every time I build the packages to test them (and by "test", here I mean at a "layout" level, checking for missing files, runtime installs, dependencies, etc.).

From a more functional point of view I run specific tests locally to fix possible problems, then I run the whole suite locally before building the packages and once more on GitHub just before freezing the release.

Obviously, should the release cycle become faster (or the packaging process more stable), the test run will be integrated in the package building workflow.

Hm, valgrind possibly takes too long in the CI (not sure, the flags/linking option above would be fine), how much overhead is DrMemory? Do you execute these locally?

The problem here is that DrMemory, though being a great tool, tends to emit a lot of false positives coming from system libraries and there is no easy way to trap them. I use it to track Windows-specific problems (that might actually lead to an actual cross-platform bug) but as a general usage tool I do not find it really helpful for GixSQL (with Gix-IDE it is a bit different, given the scope of the project).