Jetty 12 - LargeHeaderTest shows rare (error) response write bug where no response is produced

[joakime]

Jetty version(s)
12

Java version/vendor (use: java -version)
17.0.6

OS type/version
Linux

Description
When working on restoring the disabled test LargeHeaderTest it can be seen that some of the responses are not producing any response.
The connection is closed before a response is written (as evidenced by wireshark capture).

It happens about 1 time out of 200 full requests.

See branch fix/12.0.x/large-header-test for the current state of the LargeHeaderTest.

Note: the LargeHeaderTest creates 500 new connections, each making a simple request, every request reaches the Handler, which adds an excessively large response header, and then attempts to write a tiny response body. Each response that the client sees should be a 500: Response header too large.

[joakime]

@sbordet I managed to capture a full DEBUG log from this branch. (had to turn on logback to capture it successfully).

• test-debug.log.gz

Pay attention to the end of the log.

[ERROR] Failures:
[ERROR]   LargeHeaderTest.testLargeHeaderNewConnections:194 All tasks did not fail as expected.
Iterations: 500
Count (500 response status) [expected]: 492
Count (null response): 0
Count (throwables): 0
Count (other status codes): 8
 ==> expected: <500> but was: <492>


2023-04-24 17:23:57,807 DEBUG [ForkJoinPool-1-worker-1] o.e.j.s.LargeHeaderTest [LargeHeaderTest.java:189] -- Other Result: <status-bad-0-144>
2023-04-24 17:23:57,807 DEBUG [ForkJoinPool-1-worker-1] o.e.j.s.LargeHeaderTest [LargeHeaderTest.java:189] -- Other Result: <status-bad-0-221>
2023-04-24 17:23:57,807 DEBUG [ForkJoinPool-1-worker-1] o.e.j.s.LargeHeaderTest [LargeHeaderTest.java:189] -- Other Result: <status-bad-0-239>
2023-04-24 17:23:57,807 DEBUG [ForkJoinPool-1-worker-1] o.e.j.s.LargeHeaderTest [LargeHeaderTest.java:189] -- Other Result: <status-bad-0-307>
2023-04-24 17:23:57,808 DEBUG [ForkJoinPool-1-worker-1] o.e.j.s.LargeHeaderTest [LargeHeaderTest.java:189] -- Other Result: <status-bad-0-316>
2023-04-24 17:23:57,808 DEBUG [ForkJoinPool-1-worker-1] o.e.j.s.LargeHeaderTest [LargeHeaderTest.java:189] -- Other Result: <status-bad-0-344>
2023-04-24 17:23:57,808 DEBUG [ForkJoinPool-1-worker-1] o.e.j.s.LargeHeaderTest [LargeHeaderTest.java:189] -- Other Result: <status-bad-0-404>
2023-04-24 17:23:57,808 DEBUG [ForkJoinPool-1-worker-1] o.e.j.s.LargeHeaderTest [LargeHeaderTest.java:189] -- Other Result: <status-bad-0-469>

[joakime]

@gregw you might want to be aware of this

[gregw]

@joakime can you still reproduce this?

[gregw]

I get a long pause every 500 to 1000 runs when running in a loop in Intellij, but the test does not fail. No reports of empty responses and when a test for slow responses is added, I see no slow responses?

[gregw]

Hmmmm when I stop the test in the middle of a long pause and then immediately re-run it, then very first test goes back into a long pause. Thus I'm thinking it is an intellij or system issue rather than a test issue.

[joakime]

I consider this to be fixed by PR #9684 now