Thread got stuck while generating the header in jetty client

[MaheshJob]

Jetty Version
Jetty Version 11.0.20
Java Version
java version "17.0.11"
Question

• I am testing my application with high traffic and long response timeout messages.
• While testing this i can see that one of the thread is getting stuck for more than 15 secondswhile generating the http headers in the jetty client and causing the performance impact.
• I need the jetty client operation to be asynchronous to avoid any blocking calls. So from my business thread, using Webclient with Jetty reactor to achieve this. But during the handover process to jetty client it is getting stuck.

Please find the respective stack trace logs as follows.

[email protected]/java.lang.ref.Reference.waitForReferenceProcessing(Reference.java:292),
[email protected]/java.lang.ref.Reference$1.waitForReferenceProcessing(Reference.java:322),
[email protected]/java.nio.Bits.reserveMemory(Bits.java:164),
[email protected]/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:121),
[email protected]/java.nio.ByteBuffer.allocateDirect(ByteBu     ffer.java:332),
org.eclipse.jetty.util.BufferUtil.allocateDirect(BufferUtil.java:133),
org.eclipse.jetty.io.ByteBufferPool.newByteBuffer(ByteBufferPool.java:71),
org.eclipse.jetty.io.MappedByteBufferPool.acquire(MappedByteBufferPool.java:159),
org.eclipse.jetty.io.ByteBufferPool$Lease.acquire(ByteBufferPool.java:96),
org.eclipse.jetty.http2.generator.FrameGenerator.encode(FrameGenerator.java:53),
org.eclipse.jetty.http2.generator.HeadersGenerator.generateHeaders(HeadersGenerator.java:65),
org.eclipse.jetty.http2.generator.HeadersGenerator.generate(HeadersGenerator.java:52),
org.eclipse.jetty.http2.generator.Generator.control(Generator.java:105),
org.eclipse.jetty.http2.HTTP2Session$ControlEntry.generate(HTTP2Session.java:1289),
org.eclipse.jetty.http2.HTTP2Flusher.process(HTTP2Flusher.java:215),
org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:243),
org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224),
org.eclipse.jetty.http2.HTTP2Session$StreamsState.flush(HTTP2Session.java:2387),
org.eclipse.jetty.http2.HTTP2Session$StreamsState.createLocalStream(HTTP2Session.java:2285),
org.eclipse.jetty.http2.HTTP2Session$StreamsState.newLocalStream(HTTP2Session.java:2189),
org.eclipse.jetty.http2.HTTP2Session.newStream(HTTP2Session.java:645),
org.eclipse.jetty.http2.client.http.HttpSenderOverHTTP2.sendHeaders(HttpSenderOverHTTP2.java:129),
org.eclipse.jetty.client.HttpSender$ContentConsumer.onContent(HttpSender.java:496),
org.eclipse.jetty.client.util.AsyncRequestContent.notifyContent(AsyncRequestContent.java:264),
org.eclipse.jetty.client.util.AsyncRequestContent.produce(AsyncRequestContent.java:234),
org.eclipse.jetty.client.util.AsyncRequestContent.demand(AsyncRequestContent.java:109),
org.eclipse.jetty.client.HttpSender.demand(HttpSender.java:237),
org.eclipse.jetty.client.HttpSender.send(HttpSender.java:83),
org.eclipse.jetty.http2.client.http.HttpChannelOverHTTP2.send(HttpChannelOverHTTP2.java:109),
org.eclipse.jetty.client.HttpChannel.send(HttpChannel.java:127),
org.eclipse.jetty.client.HttpConnection.send(HttpConnection.java:110),
org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.send(HttpConnectionOverHTTP2.java:117),
org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:440),
org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:416),
org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:371),
org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:354),
org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:348),
org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:325),
org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:304),
org.eclipse.jetty.client.HttpClient.send(HttpClient.java:501),
org.eclipse.jetty.client.HttpRequest$$Lambda$1614/0x0000000800cd4cc8.accept(Unknown Source),
org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:879),
org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:867),
org.eclipse.jetty.reactive.client.internal.ResponseListenerProcessor.send(ResponseListenerProcessor.java:169),
org.eclipse.jetty.reactive.client.internal.ResponseListenerProcessor.onRequest(ResponseListenerProcessor.java:155),
org.eclipse.jetty.reactive.client.internal.AbstractSinglePublisher.request(AbstractSinglePublisher.java:90),

can you please let me know the following

1. what could be causing this issue ?

[sbordet]

Jetty 11 is at end of community support, see:

• EoCS (End of Community Support) for Jetty 10 / Jetty 11 - January 2024 #10485

Please upgrade to Jetty 12 and see if the problem persist.

From the stack trace, it is not a Jetty issue, but rather a GC / heap size issue.

[gregw]

This pause is in memory allocation of direct memory. So it is likely a garbage collection related problem. Even though direct memory is not directly GC'd the waitForReferenceProcessing method appears to be part of a GC mechanism.
Can you tell us more about the state of this JVM? How much memory is allocated, what operating system, what does the OS report for memory usage etc.
Can you run with -verbose:gc argument on the JVM and see if a GC even coincides with the pause?