Hpack exception causes the connection drop with client

[ThiyagarajanManivannan]

We get below exception some time http2 server response to client, we use the pax-jetty version: 9.4.46.v20220331

HTTP/2 connection terminated with client after this exception

org.eclipse.jetty.http2.hpack.HpackException$SessionException: Could not hpack encode HTTP/2.0{s=201,h=7,cl=967}
	at org.eclipse.jetty.http2.hpack.HpackEncoder.encode(HpackEncoder.java:275)
	at org.eclipse.jetty.http2.generator.FrameGenerator.encode(FrameGenerator.java:56)
	at org.eclipse.jetty.http2.generator.HeadersGenerator.generateHeaders(HeadersGenerator.java:70)
	at org.eclipse.jetty.http2.generator.HeadersGenerator.generate(HeadersGenerator.java:57)
	at org.eclipse.jetty.http2.generator.Generator.control(Generator.java:86)
	at org.eclipse.jetty.http2.HTTP2Session$ControlEntry.generate(HTTP2Session.java:1170)
	at org.eclipse.jetty.http2.HTTP2Flusher.process(HTTP2Flusher.java:218)
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
	at org.eclipse.jetty.http2.HTTP2Session.frame(HTTP2Session.java:729)
	at org.eclipse.jetty.http2.HTTP2Session.frames(HTTP2Session.java:701)
	at org.eclipse.jetty.http2.HTTP2Stream.send(HTTP2Stream.java:115)
	at org.eclipse.jetty.http2.server.HttpTransportOverHTTP2.lambda$sendHeaders$0(HttpTransportOverHTTP2.java:200)
	at org.eclipse.jetty.http2.server.HttpTransportOverHTTP2$TransportCallback.send(HttpTransportOverHTTP2.java:387)
	at org.eclipse.jetty.http2.server.HttpTransportOverHTTP2$TransportCallback.access$100(HttpTransportOverHTTP2.java:367)
	at org.eclipse.jetty.http2.server.HttpTransportOverHTTP2.sendHeaders(HttpTransportOverHTTP2.java:191)
	at org.eclipse.jetty.http2.server.HttpTransportOverHTTP2.send(HttpTransportOverHTTP2.java:93)
	at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:1009)
	at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:1086)
	at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:285)
	at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:269)
	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:901)
	at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:301)
	at java.base/java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:187)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:278)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:232)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:247)
	at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:842)
	at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:389)
	at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:707)
	at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:373)
	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:363)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:857)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:889)
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:845)

Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
	at org.eclipse.jetty.http.PreEncodedHttpField.putTo(PreEncodedHttpField.java:118)
	at org.eclipse.jetty.http2.hpack.HpackEncoder.encode(HpackEncoder.java:376)
	at org.eclipse.jetty.http2.hpack.HpackEncoder.encode(HpackEncoder.java:214)
	... 47 common frames omitted

Same problem stated in this Issue #2048 · eclipse/jetty.project is fixed in jetty version 9.4.x, though fix is already available in below jetty version

Used version in our system:
karaf version: 4.3.7
Jetty version: 9.4.46

This happens intermittently, what could be the reason for this exception ?

[sbordet]

Can you detail what do you mean by "intermittently"?

I ask because for a single Jetty setup, either the PreEncodedHttpFields have all been discovered, or not, so your responses should all pass or all fail, not "intermittently".

Are you reloading the server between requests? Making requests to different Jetty, possibly configured differently?

[ThiyagarajanManivannan]

@sbordet Intermittently states that this issue happens some execution run in "Continuous Integration and deployment" staging job. When this happens all test fails. Below is the HTTP response received and not reloading the jetty server between requests.

When we re-triggered the failed job then all times it get failed with the same hpack exception.
It looks error occurs during the build and deployment time for specific execution, what cause this issue?

HTTP2_RESPONSE

version : h2_prior_knowledge,
HEADER: {
content-type : text/plain,
date : Tue, 10 May 2022 18:52:18 GMT,
server : envoy,
vary : Accept-Encoding
},
statusCode : 503,
message : ,
body : upstream connect error or disconnect/reset before headers. reset reason: connection termination

[sbordet]

IIUC, you start a job on your CI, and all the tests fail with that error.
If that's the case, then you don't have setup Jetty in the correct way, likely you lack the HTTP/2 libraries like reported in #2048.

In particular you probably lack the http2-hpack artifact, which is typically resolved as a transitive dependency.
In your case, you seem to get the error during the write of a response on the server, so you have to make sure that the http2-server artifact is available (which in turn depends on the http2-hpack artifact).

[joakime]

Jetty 9.x is now at End of Community Support.

• EoCS (End of Community Support) for Jetty 9 - June 2022 #7958

[github-actions]

This issue has been automatically marked as stale because it has been a
full year without activity. It will be closed if no further activity occurs.
Thank you for your contributions.