Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QCheck2 shrinker improvements to list, array, bytes, strings, and functions #319

Open
wants to merge 14 commits into
base: main
Choose a base branch
from

Conversation

jmid
Copy link
Collaborator

@jmid jmid commented Jan 31, 2025

This PR takes a first step towards improving the QCheck2 list shrinker.
The current one uses the input size tree and offers only prefixes, e.g., shrinking [1;2;3;4;5;6;7;8] to [], [1;2;3;4], [1;2;3;4;5;6], [1;2;3;4;5;6;7] (or something like that). This results in needlessly large counterexamples, if the property is essentially "lists do not contain the int 7". 😬

The PR replaces the list shrinker with a recursive one instead shrinking [1;2;3;4;5;6;7;8] to
[1;2;3;4], [5;6;7;8], [2;3;4;5;6;7;8] or [1;2;3;4;6;7;8], i.e.,

  • trying to keep either half or
  • dropping the head element from either half

The shrinker then continues size reduction recursively from either of these, and switches to element shrinking afterwards.
The approach is not perfect, but it is a good step forward over the current status, and together with #318 it gets QCheck2 working better (witness the differences in expect test outputs).

Secondly the PR switches the other list, array, bytes, string #157, and function generators/shrinkers #163 to use the improved list shrinker, so that these benefit from the improvement too.

The PR does not improve the QCheck2.Gen.{list_size, array_size, bytes_size, string_size} generators.
These take a size generator parameter, and I've not figured out how to cleanly respect that (read: preserve invariants), e.g., if a user provides a size generator of "even numbers", "multiples of 7", or "prime numbers". As such,
the shrinking behaviour and performance of those remain untouched.

QCheck2 still fares badly on < ocaml.5.0 because it needs Random.State.split and on OCaml4 that builds on a hack - that runs pretty slowly (on OCaml 5+ it works nice and fast). I have another patch to replace the hack with a faster one, that I'll submit as a separate PR.

The PR adds a bunch of QCheck2 shrinker unit tests to document the effect of the change.
Should anyone wish to, the PR can thus be read commit-by-commit as gradual improvements.

I'm keen to hear from any QCheck2 users if this helps them (polite ping to @vch9 and friends)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant