Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make the tests buildable and runnable with cabal and run said testsuites in CI #6

Closed
wants to merge 3 commits into from

Conversation

alpmestan
Copy link
Contributor

WIP -- the testsuites aren't green yet.

Current status:

Tests failed for test:encoding from fb-util-0.1.0.0.
Tests failed for test:rwvar from fb-util-0.1.0.0.
Tests failed for test:unit-tests from fb-util-0.1.0.0.
Tests failed for test:header-channel from thrift-lib-0.1.0.0.
Tests failed for test:dangling-pointer from thrift-server-0.1.0.0.
Tests failed for test:server from thrift-server-0.1.0.0.

unit-tests:

  tests/UnitTests.hs:44:10: 
  1) parseNaN
       couldn't parse NaN

  To rerun use: --match "/parseNaN/"

Randomized with seed 1332817988

rwvar:

Test suite rwvar: RUNNING...

readTest
writeTest
nestedReads
rwvar: thread blocked indefinitely in an MVar operation

Test suite rwvar: FAIL
Test suite logged to:
/hsthrift/dist-newstyle/build/x86_64-linux/ghc-8.4.4/fb-util-0.1.0.0/t/rwvar/test/fb-util-0.1.0.0-rwvar.log

header-channel:

Test suite header-channel: RUNNING...
WARNING: Logging before InitGoogleLogging() is written to STDERR
E1123 13:07:20.483438 32534 ThriftServer.cpp:386] Got an exception while setting up the server: failed to bind to async server socket: [::1]:0: Cannot assign requested address
I1123 13:07:20.484915 32535 Acceptor.cpp:467] Dropping all connections from Acceptor=0x7f8064002470 in thread 0x7f805c000d70
terminate called after throwing an instance of 'std::system_error'
  what():  failed to bind to async server socket: [::1]:0: Cannot assign requested address

Test suite header-channel: FAIL
Test suite logged to:
/hsthrift/dist-newstyle/build/x86_64-linux/ghc-8.4.4/thrift-lib-0.1.0.0/t/header-channel/test/thrift-lib-0.1.0.0-header-channel.log

dangling-pointer:

Test suite dangling-pointer: RUNNING...
WARNING: Logging before InitGoogleLogging() is written to STDERR
F1123 13:07:21.338804 32548 BaseThriftServer.h:574] Check failed: configMutable() 
*** Check failure stack trace: ***

Test suite dangling-pointer: FAIL
Test suite logged to:
/hsthrift/dist-newstyle/build/x86_64-linux/ghc-8.4.4/thrift-server-0.1.0.0/t/dangling-pointer/test/thrift-server-0.1.0.0-dangling-pointer.log

server:

Test suite server: RUNNING...
WARNING: Logging before InitGoogleLogging() is written to STDERR
F1123 13:07:22.108315 32568 BaseThriftServer.h:574] Check failed: configMutable() 
*** Check failure stack trace: ***

Test suite server: FAIL
Test suite logged to:
/hsthrift/dist-newstyle/build/x86_64-linux/ghc-8.4.4/thrift-server-0.1.0.0/t/server/test/thrift-server-0.1.0.0-server.log

@alpmestan alpmestan requested a review from simonmar November 23, 2020 13:11
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 23, 2020
common/util/fb-util.cabal Outdated Show resolved Hide resolved
common/util/fb-util.cabal Outdated Show resolved Hide resolved
lib/test/ClientTest.hs Outdated Show resolved Hide resolved
@@ -122,6 +123,29 @@ library
vector ^>=0.12.0.1

common test-common
hs-source-dirs:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we compiling these modules for each test rather than using a library? The tests worked for me I think.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The internal library thing was not working, at least with the cabal that we end up building with. I can get you the error a bit later, but this made me decide to stuff everything needed in the test-common section.

common test-common
hs-source-dirs: test, test/github, test/gen-hs2, ../lib/test/gen-hs2, ../lib/test

other-modules: Facebook.Init
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same question - why no library?

@alpmestan
Copy link
Contributor Author

alpmestan commented Nov 25, 2020

Following tips given in docker/docs#6075 -- I configured my local Docker service to be ipv6 capable and now all the lib/server tests pass! I'm down to just those 3 failing:

Preprocessing test suite 'encoding' for fb-util-0.1.0.0..
Building test suite 'encoding' for fb-util-0.1.0.0..
Running 1 test suites...
Test suite encoding: RUNNING...

regexTest FAILED [1]

Failures:

  tests/EncodingTest.hs:11:13: 
  1) regexTest
       regex match test

  To rerun use: --match "/regexTest/"

Randomized with seed 894481978

Finished in 0.0003 seconds
1 example, 1 failure

Test suite encoding: FAIL
Test suite logged to:
/hsthrift/dist-newstyle/build/x86_64-linux/ghc-8.4.4/fb-util-0.1.0.0/t/encoding/test/fb-util-0.1.0.0-encoding.log
0 of 1 test suites (0 of 1 test cases) passed.
Preprocessing test suite 'rwvar' for fb-util-0.1.0.0..
Building test suite 'rwvar' for fb-util-0.1.0.0..
Running 1 test suites...
Test suite rwvar: RUNNING...

readTest
writeTest
nestedReads
rwvar: thread blocked indefinitely in an MVar operation

Test suite rwvar: FAIL
Test suite logged to:
/hsthrift/dist-newstyle/build/x86_64-linux/ghc-8.4.4/fb-util-0.1.0.0/t/rwvar/test/fb-util-0.1.0.0-rwvar.log
0 of 1 test suites (0 of 1 test cases) passed.
Preprocessing test suite 'unit-tests' for fb-util-0.1.0.0..
Building test suite 'unit-tests' for fb-util-0.1.0.0..
Running 1 test suites...
Test suite unit-tests: RUNNING...

intToByteString
parseValueStrictTest
parseNaN FAILED [1]
useFFTW
withCStringLen
useByteStringsAsCStrings
bsListAsCStrLenArr
readText
insertCommasAndAnd

Failures:

  tests/UnitTests.hs:44:10: 
  1) parseNaN
       couldn't parse NaN

  To rerun use: --match "/parseNaN/"

Randomized with seed 34191424

Finished in 0.0010 seconds
9 examples, 1 failure

Test suite unit-tests: FAIL
Test suite logged to:
/hsthrift/dist-newstyle/build/x86_64-linux/ghc-8.4.4/fb-util-0.1.0.0/t/unit-tests/test/fb-util-0.1.0.0-unit-tests.log
0 of 1 test suites (0 of 1 test cases) passed.
cabal: Tests failed for test:encoding from fb-util-0.1.0.0.
Tests failed for test:rwvar from fb-util-0.1.0.0.
Tests failed for test:unit-tests from fb-util-0.1.0.0.

The bad news is that Github Actions won't support ipv6 capable Docker environments anytime soon: actions/runner-images#668

And Travis' pricing changed recently, not positively. Is there a way we could perhaps fallback to ipv4 without too much hassle, in the tests?

@alpmestan
Copy link
Contributor Author

Re: the parseNaN failure, here's the definition of that test case:

-- | We've patched the official aeson library to handle NaN. See #5183005
-- This test ensures that we notice if we forget to patch on upgrade.
parseNaN :: Test
parseNaN = TestCase $
  case decode "[NaN]" of
    Just [a :: Double] ->
      assertBool "NaN" $ isNaN a
    _ -> assertFailure "couldn't parse NaN"

Aeson has this test input:
https://github.com/bos/aeson/blob/550b03d62021c93da58d40014280486d1c82726e/tests/JSONTestSuite/test_parsing/n_number_NaN.json but doesn't seem to be using it for its own testsuite, just some comparative test of JSON libs.

The comment above presumably refers to the patch that you were talking about the other day. What do you want us to do here, omit this test for now?

@alpmestan
Copy link
Contributor Author

Regarding the regexTest failure...

regexTest :: Assertion
regexTest = assertBool "regex match test" $ regexMatchWord "lunedi"

-- ... in another module...
regexMatchWord :: Text -> Bool
regexMatchWord w = traceShow (ranges, w, Text.length w) $ ranges == [(0, Text.length w)]
  where
    regex = R.makeRegexOpts compOpts execOpts r
    compOpts = PCRE.defaultCompOpt + PCRE.compCaseless
    execOpts = PCRE.defaultExecOpt
    r = "luned(\x00ec|i)" :: String
    allMatches = R.match regex (Text.encodeUtf8 w) :: [[ByteString]]
    (ranges, _, _) = foldl' g ([], w, 0) allMatches
    g (ranges', s, offset) matches =
      case map Text.decodeUtf8 matches of
        [] -> (ranges', s, offset)
        ("":_) -> (ranges', s, offset)
        (text:_group) ->
          let (x,xs) = Text.breakOn text s
              m = offset + Text.length x
              n = Text.length text
              s' = Text.drop n xs
              newOffset = m + n
          in (ranges' ++ [range], s', newOffset)

I made this change to peek at what was going on:

diff --git a/common/util/tests/EncodingLib.hs b/common/util/tests/EncodingLib.hs
index da25169..83e66da 100644
--- a/common/util/tests/EncodingLib.hs
+++ b/common/util/tests/EncodingLib.hs
@@ -12,8 +12,10 @@ import Data.List (foldl')
 import Foreign.C hiding (peekCStringLen)
 import Data.Text.Foreign
 
+import Debug.Trace
+
 regexMatchWord :: Text -> Bool
-regexMatchWord w = ranges == [(0, Text.length w)]
+regexMatchWord w = traceShow (ranges, w, Text.length w) $ ranges == [(0, Text.length w)]

The traceShow prints ([(0,5)],"lunedi",6). It seems like it would all work nicely if the final check was ranges == [(0, Text.length w - 1)], which from a cursory glance at regexMatchWord would perhaps make sense, given that this list is apparently supposed to return offsets (see e.g how the code creates a new range to be returned: range = (m, newOffset), where m is itself another offset to which we added a length).

And the matching part of the input (all of it) does span from offset 0 ('l') to offset 5 (i), and has length 6.

However, I suspect this test is passing for you, so maybe my guess above is wrong, and somehow we should get [(0, 6)] and not [(0, 5)] ?

Makefile Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
@simonmar
Copy link
Contributor

And Travis' pricing changed recently, not positively. Is there a way we could perhaps fallback to ipv4 without too much hassle, in the tests?

We can #ifdef FACEBOOK and change ::1 to localhost (or perhaps it needs 127.0.0.1) if that works.

@simonmar
Copy link
Contributor

The comment above presumably refers to the patch that you were talking about the other day. What do you want us to do here, omit this test for now?

A fix for this will be incoming shortly.

@simonmar
Copy link
Contributor

However, I suspect this test is passing for you, so maybe my guess above is wrong, and somehow we should get [(0, 6)] and not [(0, 5)] ?

I believe this test needs the locale to be set. But in any case, it's not actually testing anything in this library, so we should omit it - I'll fix this up and remove the test from the repository.

Copy link
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@simonmar has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@alpmestan
Copy link
Contributor Author

With the tip of this branch, all tests but rwvar pass for me, and hopefully in CI too (ongoing, right now). I suspect the test driver we're using here (hspec) and the one used on the FB side are different enough in their approach to running tests (async or not, masking or not, etc). Using hspec results in the main thread being deadlocked too and receiving the exception from the RTS, whereas the one you use presumably protects itself against that type of problem.

(FWIW, looks like this problem appeared with hspec 2.5 -- see jcristovao/enclosed-exceptions#12)

facebook-github-bot pushed a commit that referenced this pull request Nov 27, 2020
Summary:
See #6

Github actions apparently has problems with IPv6, we weren't able to
get these tests working using ::1 as the localhost address, so this
diff makes the tests optionally use 127.0.0.1 instead. In fbcode we'll
continue to use ::1.

Reviewed By: anubhav94N

Differential Revision: D25196206

fbshipit-source-id: 0b3af3b32a421a843bc2c2eb6fc988c0d8c842d4
@simonmar simonmar mentioned this pull request Nov 27, 2020
@simonmar
Copy link
Contributor

simonmar commented Nov 27, 2020 via email

@simonmar
Copy link
Contributor

simonmar commented Nov 27, 2020 via email

facebook-github-bot pushed a commit that referenced this pull request Dec 1, 2020
Summary:
Pull Request resolved: #7

from #6

Reviewed By: ChrisKuklewicz

Differential Revision: D25196223

fbshipit-source-id: 105933fe84296e465db6be72c50909ec881b52c9
@simonmar
Copy link
Contributor

simonmar commented Dec 1, 2020

I believe this is all incorporated now.

@simonmar simonmar closed this Dec 1, 2020
@alpmestan alpmestan deleted the alp/ci-test branch January 22, 2021 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants