Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NPE in error handling leading to 100% CPU #11631

Closed
konstantin-mikheev opened this issue Apr 8, 2024 · 10 comments
Closed

NPE in error handling leading to 100% CPU #11631

konstantin-mikheev opened this issue Apr 8, 2024 · 10 comments
Assignees
@lorban
Labels
Bug For general bugs on Jetty side

Comments

@konstantin-mikheev
Copy link

konstantin-mikheev commented Apr 8, 2024 •
edited
Loading

Jetty version(s)
12.0.7

Jetty Environment
ee10

Java version/vendor (use: java -version)
OpenJDK Runtime Environment Corretto-21.0.0.35.1 (build 21+35-LTS)

OS type/version
Rocky Linux release 8.8 (Green Obsidian)

Description
Same as #11431 but Spring Boot 3.2.4 and Jetty 12.0.7

It is a reopening of the bug #11431. The commit have not fixed the issue but added logging therefore we have more logs.

There were circular references in logs so I am attaching the full stacktrace of the NPE in error handling as a text file. The later the log, the longer is the stacktrace.

stacktrace.txt

Overall here is the short version of the stacktrace

java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "f" is null
	at org.eclipse.jetty.http.HttpFields$Mutable.remove(HttpFields.java:1418) ~[jetty-http-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.ee10.servlet.ServletContextResponse.resetContent(ServletContextResponse.java:385) ~[jetty-ee10-servlet-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:512) ~[jetty-ee10-servlet-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1206) ~[jetty-server-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1199) ~[jetty-server-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.runInContext(ServletChannelState.java:1257) ~[jetty-ee10-servlet-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.asyncError(ServletChannelState.java:818) ~[jetty-ee10-servlet-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.accept(ContextHandler.java:1187) ~[jetty-server-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.server.handler.ContextRequest.lambda$addFailureListener$1(ContextRequest.java:53) ~[jetty-server-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.server.internal.HttpChannelState.lambda$onFailure$2(HttpChannelState.java:445) ~[jetty-server-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:191) ~[jetty-util-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.http2.server.internal.HttpStreamOverHTTP2.lambda$onFailure$2(HttpStreamOverHTTP2.java:590) ~[jetty-http2-server-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:971) ~[jetty-util-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1201) ~[jetty-util-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1156) ~[jetty-util-12.0.7.jar:12.0.7]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
	Suppressed: java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "f" is null
		... 16 common frames omitted

then there is multiple recurring

	Suppressed: java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "f" is null
		... 16 common frames omitted

then

Suppressed: org.eclipse.jetty.io.EofException: Reset cancel_stream_error
	at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onReset(HTTP2ServerConnectionFactory.java:158)
	at org.eclipse.jetty.http2.HTTP2Stream.notifyReset(HTTP2Stream.java:876)
	at org.eclipse.jetty.http2.HTTP2Stream.onReset(HTTP2Stream.java:586)
	at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:357)
	at org.eclipse.jetty.http2.HTTP2Session.onReset(HTTP2Session.java:346)
	at org.eclipse.jetty.http2.HTTP2Connection.onReset(HTTP2Connection.java:259)
	at org.eclipse.jetty.http2.parser.BodyParser.notifyReset(BodyParser.java:139)
	at org.eclipse.jetty.http2.parser.ResetBodyParser.onReset(ResetBodyParser.java:94)
	at org.eclipse.jetty.http2.parser.ResetBodyParser.parse(ResetBodyParser.java:61)
	at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:229)
	at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:156)
	at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:121)
	at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:349)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:512)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:410)
	... 4 common frames omitted

then a lot of

Suppressed: [CIRCULAR REFERENCE: java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "f" is null]

How to reproduce?
Same as #11431

We do not have the thread dump unfortunately but we have a heap dump this time.

We could not share heapdump for security reasons but if you could point out to specific things I could look it up
@konstantin-mikheev konstantin-mikheev added the Bug For general bugs on Jetty side label Apr 8, 2024
@lorban
Copy link
Contributor

lorban commented Apr 8, 2024

@konstantin-mikheev We're currently re-investigating this problem, and it seems there are at least a couple of issues at play; I managed to reproduce one of them and I'm working on a fix, but I still need to investigate the other one(s) to better understand what's going on.

I'll keep you posted shortly with my progress.

lorban added a commit that referenced this issue Apr 9, 2024
@lorban
#11631 abort instead of rethrow when the exception handler in handle(…
f73d413 
…) throws an exception

Signed-off-by: Ludovic Orban <[email protected]>
@lorban lorban mentioned this issue Apr 9, 2024
Merged
@lorban
Copy link
Contributor

lorban commented Apr 9, 2024

@konstantin-mikheev I've created a PR containing a simple change that should mitigate this problem, it's over here: #11637

For some reason I'm still investigating, the HTTP2 headers can become corrupted when a H2 reset frame is sent by the client with the wrong timing, causing the servlet handling code to trap itself in an infinite loop trying to constantly run the same piece of code against the corrupted headers, then handling the exception by adding it as a suppressed to the original one then retying.

How the H2 headers corruption happens still need more investigation, even if the stack traces you managed to collect seems to lead to a client sending a H2 reset frame with a bad timing.

But a least, with the linked patch, at least Jetty will break the infinite loop. I'd be glad if you could build it, give it a try and report the outcome. Thanks!

@konstantin-mikheev
Copy link
Author

Hey! Sorry for delay. This branch of yours is based also on the jetty version with the spring integration changes there so it is not compatible with the spring version we use

But we have cherry picked the commit changes and made a jetty version with those changes exclusively based on 12.0.8 release to be sure nothing else will intervene

I have deployed the version with this patch on prod today but we will need some time for the issue to manifest itself. I will update if and when there will be more info

joakime pushed a commit that referenced this issue Apr 23, 2024
@lorban
Fix for some invalid server behaviors when a client is aborting a req…
a1450f5 
…uest (#11637)

* #11631 abort instead of rethrow when the exception handler in handle() throws an exception
* fix ee10 state machine error when client aborts
* H3: wire client-sent reset packet on abort
* H3: only fail finished streams when fill interested is false

---------

Signed-off-by: Ludovic Orban <[email protected]>
@joakime joakime changed the title Reopening #11431: NPE in error handling leading to 100% CPU also in 12.0.7 NPE in error handling leading to 100% CPU Apr 26, 2024
@konstantin-mikheev
Copy link
Author

We got a bunch of those errors over the last couple of weeks with the @lorban changes applied over 12.0.8 release and those errors have not caused 100% CPU utilisation. It was reproduced for 30+ times on a few different DCs

Here is a sample stacktrace from one of those errors:

java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "field" is null
	at org.eclipse.jetty.server.internal.HttpChannelState$ErrorResponse.getResponseHttpFields(HttpChannelState.java:1623) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.server.internal.HttpChannelState$ChannelResponse.<init>(HttpChannelState.java:1088) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.server.internal.HttpChannelState$ErrorResponse.<init>(HttpChannelState.java:1610) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.server.internal.HttpChannelState$ChannelCallback.failed(HttpChannelState.java:1549) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.ee10.servlet.ServletChannel.onCompleted(ServletChannel.java:762) ~[jetty-ee10-servlet-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:421) ~[jetty-ee10-servlet-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1292) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1285) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.runInContext(ServletChannelState.java:1257) ~[jetty-ee10-servlet-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.asyncError(ServletChannelState.java:818) ~[jetty-ee10-servlet-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.accept(ContextHandler.java:1273) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.server.handler.ContextRequest.lambda$addFailureListener$1(ContextRequest.java:53) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.server.internal.HttpChannelState.lambda$onFailure$2(HttpChannelState.java:445) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:191) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.http2.server.internal.HttpStreamOverHTTP2.lambda$onFailure$1(HttpStreamOverHTTP2.java:594) ~[jetty-http2-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:211) ~[jetty-http2-common-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:158) ~[jetty-http2-common-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:449) ~[jetty-http2-common-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99) ~[jetty-io-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.io.ssl.SslConnection$SslEndPoint.onFillable(SslConnection.java:574) ~[jetty-io-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:390) ~[jetty-io-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:150) ~[jetty-io-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99) ~[jetty-io-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
	Suppressed: java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "f" is null
		at org.eclipse.jetty.http.HttpFields$Mutable.remove(HttpFields.java:1418) ~[jetty-http-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
		at org.eclipse.jetty.ee10.servlet.ServletContextResponse.resetContent(ServletContextResponse.java:385) ~[jetty-ee10-servlet-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
		at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:512) ~[jetty-ee10-servlet-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
		... 31 common frames omitted
		Suppressed: org.eclipse.jetty.io.EofException: Reset cancel_stream_error
			at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onReset(HTTP2ServerConnectionFactory.java:158)
			at org.eclipse.jetty.http2.HTTP2Stream.notifyReset(HTTP2Stream.java:876)
			at org.eclipse.jetty.http2.HTTP2Stream.onReset(HTTP2Stream.java:586)
			at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:357)
			at org.eclipse.jetty.http2.HTTP2Session.onReset(HTTP2Session.java:346)
			at org.eclipse.jetty.http2.HTTP2Connection.onReset(HTTP2Connection.java:259)
			at org.eclipse.jetty.http2.parser.BodyParser.notifyReset(BodyParser.java:139)
			at org.eclipse.jetty.http2.parser.ResetBodyParser.onReset(ResetBodyParser.java:94)
			at org.eclipse.jetty.http2.parser.ResetBodyParser.parse(ResetBodyParser.java:61)
			at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:229)
			at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:156)
			at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:121)
			at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:349)
			at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:512)
			at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)
			... 19 common frames omitted
			Suppressed: java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "f" is null
				at org.eclipse.jetty.http.HttpFields$Mutable.remove(HttpFields.java:1418)
				at org.eclipse.jetty.ee10.servlet.ServletContextResponse.resetContent(ServletContextResponse.java:385)
				at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:453)
				at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1292)
				at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1285)
				at org.eclipse.jetty.ee10.servlet.ServletChannelState.runInContext(ServletChannelState.java:1257)
				at org.eclipse.jetty.ee10.servlet.ServletChannelState.asyncError(ServletChannelState.java:818)
				at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.accept(ContextHandler.java:1273)
				at org.eclipse.jetty.server.handler.ContextRequest.lambda$addFailureListener$1(ContextRequest.java:53)
				at org.eclipse.jetty.server.internal.HttpChannelState.lambda$onFailure$2(HttpChannelState.java:445)
				at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:191)
				at org.eclipse.jetty.http2.server.internal.HttpStreamOverHTTP2.lambda$onFailure$1(HttpStreamOverHTTP2.java:594)
				at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
				at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
				at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
				... 19 common frames omitted

@joakime
Copy link
Contributor

joakime commented May 6, 2024 •
edited
Loading

@konstantin-mikheev can you try the staged 12.0.9 release? it contains many more patches than what you are doing.
See the most recent comments in the release issue for staging URL -

@mosabua mosabua mentioned this issue May 6, 2024
Closed
@lpandzic
Copy link

lpandzic commented May 9, 2024 •
edited
Loading

Hello, one question regarding this part:

For some reason I'm still investigating, the HTTP2 headers can become corrupted when a H2 reset frame is sent by the client with the wrong timing, causing the servlet handling code to trap itself in an infinite loop trying to constantly run the same piece of code against the corrupted headers, then handling the exception by adding it as a suppressed to the original one then retying.

Doesn't this make this issue a denial of service vulnerability which I expect would be of higher priority than a bug and more importantly - more people should be aware of it?

@joakime
Copy link
Contributor

joakime commented May 10, 2024

This issue is considered complete, and was handled by

And the fix is available in Jetty 12.0.9

@konstantin-mikheev
Copy link
Author

Tested on 12.0.9, there are no loop and here is exception stack trace

java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "field" is null
	at org.eclipse.jetty.server.internal.HttpChannelState$ErrorResponse.getResponseHttpFields(HttpChannelState.java:1636) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.server.internal.HttpChannelState$ChannelResponse.<init>(HttpChannelState.java:1104) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.server.internal.HttpChannelState$ErrorResponse.<init>(HttpChannelState.java:1623) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.server.internal.HttpChannelState$ChannelCallback.failed(HttpChannelState.java:1562) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.ee10.servlet.ServletChannel.onCompleted(ServletChannel.java:767) ~[jetty-ee10-servlet-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:426) ~[jetty-ee10-servlet-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1292) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1285) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.runInContext(ServletChannelState.java:1289) ~[jetty-ee10-servlet-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.asyncError(ServletChannelState.java:858) ~[jetty-ee10-servlet-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.accept(ContextHandler.java:1273) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.server.handler.ContextRequest.lambda$addFailureListener$1(ContextRequest.java:53) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.server.internal.HttpChannelState.lambda$onFailure$2(HttpChannelState.java:446) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:191) ~[jetty-util-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.http2.server.internal.HttpStreamOverHTTP2.lambda$onFailure$1(HttpStreamOverHTTP2.java:594) ~[jetty-http2-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478) ~[jetty-util-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441) ~[jetty-util-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293) ~[jetty-util-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201) ~[jetty-util-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311) ~[jetty-util-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979) ~[jetty-util-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209) ~[jetty-util-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164) ~[jetty-util-12.0.9.jar:12.0.9]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
	Suppressed: java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "f" is null
		at org.eclipse.jetty.http.HttpFields$Mutable.remove(HttpFields.java:1437) ~[jetty-http-12.0.9.jar:12.0.9]
		at org.eclipse.jetty.ee10.servlet.ServletContextResponse.resetContent(ServletContextResponse.java:385) ~[jetty-ee10-servlet-12.0.9.jar:12.0.9]
		at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:517) ~[jetty-ee10-servlet-12.0.9.jar:12.0.9]
		... 18 common frames omitted
		Suppressed: org.eclipse.jetty.io.EofException: Reset cancel_stream_error
			at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onReset(HTTP2ServerConnectionFactory.java:158)
			at org.eclipse.jetty.http2.HTTP2Stream.notifyReset(HTTP2Stream.java:876)
			at org.eclipse.jetty.http2.HTTP2Stream.onReset(HTTP2Stream.java:586)
			at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:357)
			at org.eclipse.jetty.http2.HTTP2Session.onReset(HTTP2Session.java:346)
			at org.eclipse.jetty.http2.HTTP2Connection.onReset(HTTP2Connection.java:259)
			at org.eclipse.jetty.http2.parser.BodyParser.notifyReset(BodyParser.java:139)
			at org.eclipse.jetty.http2.parser.ResetBodyParser.onReset(ResetBodyParser.java:94)
			at org.eclipse.jetty.http2.parser.ResetBodyParser.parse(ResetBodyParser.java:61)
			at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:229)
			at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:156)
			at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:121)
			at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:349)
			at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:512)
			at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)
			... 6 common frames omitted
			Suppressed: java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "f" is null
				at org.eclipse.jetty.http.HttpFields$Mutable.remove(HttpFields.java:1437)
				at org.eclipse.jetty.ee10.servlet.ServletContextResponse.resetContent(ServletContextResponse.java:385)
				at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:458)
				at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1292)
				at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1285)
				at org.eclipse.jetty.ee10.servlet.ServletChannelState.runInContext(ServletChannelState.java:1289)
				at org.eclipse.jetty.ee10.servlet.ServletChannelState.asyncError(ServletChannelState.java:858)
				at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.accept(ContextHandler.java:1273)
				at org.eclipse.jetty.server.handler.ContextRequest.lambda$addFailureListener$1(ContextRequest.java:53)
				at org.eclipse.jetty.server.internal.HttpChannelState.lambda$onFailure$2(HttpChannelState.java:446)
				at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:191)
				at org.eclipse.jetty.http2.server.internal.HttpStreamOverHTTP2.lambda$onFailure$1(HttpStreamOverHTTP2.java:594)
				at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
				at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
				at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
				... 6 common frames omitted

@konstantin-mikheev
Copy link
Author

Will close since the loop does not happen in 12.0.9 and there are no 100% CPU usage

@lorban
Copy link
Contributor

lorban commented May 13, 2024

The loop got sorted out but the original NPE still exists. I've opened #11785 to follow up on that one.

Again, if you could come up with a reproducer that would be tremendously helpful.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lorban lorban

Labels
Bug For general bugs on Jetty side
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@joakime @lorban @lpandzic @konstantin-mikheev