Fuzzing a list of strings does not generate list with only empty strings, length > 1

[mgold]

@ento made this comment on #190. I don't think it quite meets the criteria of "real world" since it's not clear why ["", ""] would have caused a bug. Where is the code that you were trying to test?

I appreciate that @ento tried to shrink things as far as possible, which is usually the correct approach, but in this case I'm asking for more context about what makes fuzzers effective. It's not automatically a bug that certain, seemingly trivial cases are not generated. I do not want to discuss how we can change fuzzers to generate such cases until we establish that doing so would allow them to better catch real world bugs.

I'm pulling an Evan and moving this to a new issue so the original issue stays clean. Original comment is below.

---

It looks like list string does not generate a list of two empty strings or more within 100 iterations. It may or may not within 10000 iterations. We didn't notice a possibly faulty test code because of this (we haven't gotten around to fixing it, so not 100% sure if it didn't lead to a user-facing bug).

node-test-runner: 0.18.7
elm-test: 4.1.1

Fancier version: (if the desired number of empty strings is generated, tests should fail; actually they pass)

module WhereIsTheFuzz exposing (..)

import Expect
import Fuzz exposing (..)
import Test exposing (Test, describe, fuzz)


spec : Test
spec =
    describe "list of strings" <|
        List.map expectFuzzToNotGenerateEmptyStrings (List.range 2 20)


expectFuzzToNotGenerateEmptyStrings : Int -> Test
expectFuzzToNotGenerateEmptyStrings count =
    fuzz (list string) ("does not generate " ++ toString count ++ " empty strings") <|
        \strings ->
            Expect.notEqual strings (List.repeat count "")

Barebones version:

module WhereIsTheFuzz exposing (..)

import Expect
import Fuzz exposing (..)
import Test exposing (Test, describe, fuzz)


spec : Test
spec =
    fuzz (list string) "does not generate two empty strings" <|
        \strings ->
            Expect.notEqual strings [ "", "" ]

elm-test --fuzz 10000 --seed 996044113 tests/WhereIsTheFuzz.elm successfully fails with this barebones version, while elm-test --fuzz 10000 --seed 1272159827 tests/WhereIsTheFuzz.elm did not.

I hope this is in line with what you're looking for.

[stoeffel]

We just encountered something related to this with just a Fuzz.string (no list).
the test was doing something like this:

            , fuzz Fuzz.string "the submit button is inactive only if there is no text" <|
                \commentText ->
                    Comment.view commentText
                        |> Query.fromHtml
                        |> Query.has
                            [ Selector.tag "button"
                            , if String.isEmpty commentText then
                                Selector.class "Inactive"
                              else
                                Selector.class "Primary"
                            , Selector.text "Save Comment"
                            ]

This test was flakey on CI, because the fuzzer didn't create empty strings (most of the time). It failed when we had a , because the view treated strings only with blanks as empty. I change the test to be two tests without the if and made sure that in one case we only produce empty strings and in the other case only none empty strings. That's a better way of writing this test anyways, but I was surprised to see that a value like " " doesn't get produced more frequently. I suspected to see a higher frequency for spaces in the string fuzzer. This might not be a bug, but a realworld problem we had with string fuzzing.

[mgold]

Not sure if it's relevant to this issue, but yes, it's valid to say that Fuzz.string should generate whitespace strings more often. They are a good edge case, and fuzzers intend to produce as many of those as reasonably possible.

That's a better way of writing this test anyways

Yes indeed. I should add something like that to the README Strategies section...