Investigate flaky sequential/test-net-connect-local-error on FreeBSD

[Trott]

• Version: v8.0.0-pre
• Platform: freebsd
• Subsystem: test

https://ci.nodejs.org/job/node-test-commit-freebsd/9081/nodes=freebsd10-64/consoleFull

not ok 1453 sequential/test-net-connect-local-error
  ---
  duration_ms: 0.294
  severity: fail
  stack: |-
    assert.js:92
      throw new AssertionError({
      ^
    
    AssertionError [ERR_ASSERTION]: 'EADDRINUSE' === 'ECONNREFUSED'
        at Socket.onError (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd10-64/test/sequential/test-net-connect-local-error.js:14:10)
        at Socket.<anonymous> (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd10-64/test/common/index.js:504:15)
        at emitOne (events.js:115:13)
        at Socket.emit (events.js:210:7)
        at emitErrorNT (net.js:1337:8)
        at _combinedTickCallback (internal/process/next_tick.js:102:11)
        at process._tickCallback (internal/process/next_tick.js:161:9)

The test was recently updated to check for ECONNREFUSED. However, we are seeing EADDRINUSE sometimes on FreeBSD in CI as seen above.

Possible bug in the FreeBSD operating system or quirk (relative to our other hosts) in the way it handles open/half-open/whatever ports? (I normally wouldn't jump to Blame-The-Operating-System as the likely cause right off the bat, but this is looking very familiar to me and I seem to recall it, but I can't find anything so maybe I'm imagining it?)

/cc @sebastianplesciuc @refack @nodejs/testing

[Trott]

@nodejs/platform-freebsd

[Trott]

Oh, meant to link to the recent changes: 0c2edd2

[sebastianplesciuc]

I can make another PR to remove that assert if need be or fix it any other way. Please let me know.

[Trott]

Possibly maybe kinda sorta related? nodejs/node-v0.x-archive#3796

[refack]

I run two "single-test" CI before #12964 (comment) so high probability is that this is test cross-talk. Well sequential !== sterile.
FWIW For Windows (and acording to rfc793) a closed socket will enter TIME_WAIT state for 2*MSL and will not SYN,ACK and will not be allowed to be reused by the OS.
So a previous test using common.PORT will cause EADDRINUSE...
It might happen more and more... So at first thought seems like common.PORT is problematic even in /sequential/.
two possible solutions:

• we salt common.PORT a little bit per process.
• we add a delay between tests in /sequential/ and verify the environment is more sterile (common.refreshTmpDir , verify common.PORT is not in TIME_WAIT, etc.)

[Trott]

@refack For this particular test, I wonder if the collision might be made less likely if we swap PORT and PORT + 1? EADDRINUSE must mean it's the localPort, right? In which case PORT + 1 is somewhat less likely to linger over from another test in the way you describe because it is used much less often than just PORT.

Not necessarily the greatest solution, but if it works, an easy one to implement quickly while we try to figure out something better (or not)?

[refack]

@Trott I'd go one step further and use PORT + 1 and PORT + 2 (which is just super simple slating), and document the reasoning.

@sebastianplesciuc you wanna do the honors?

[Trott]

I'd go one step further and use PORT + 1 and PORT + 2 (which is just super simple slating), and document the reasoning.

I worry that it might become an justification for doing PORT + 50 in some test and PORT + 45 in another. Maybe we can stick with PORT and PORT + 1 here and go to your proposal if we see continued problems?

[refack]

I worry that it might become an justification for doing PORT + 50 in some test and PORT + 45 in another. Maybe we can stick with PORT and PORT + 1 here and go to your proposal if we see continued problems?

NP

[sebastianplesciuc]

@refack on it! Thanks