HttpClient threads not getting cleared leading to a OutOfMemory crashing the Java process (java.lang.OutOfMemoryError: unable to create new native thread)

[satishtedla]

Jetty version(s)
9.4.43

Jetty Environment

Java version/vendor (use: java -version)
1.8.x

OS type/version
el7uek.x86_64

Description
Huge number of HttpClient threads lying around without getting cleaned up

How to reproduce?
Submit a bunch of application tasks in bulk:
In-house: which spawn 400-500 HttpClient threads. Some of them get cleared but a good chunk of these remain.
In Customer's env: It spawns thousands of HttpClient threads stay in this state without getting cleared leading to a JVM crash.
"java.lang.OutOfMemoryError: unable to create new native thread"

Analysis:
High-level anlalysis

1. HttpClient.start creates the threads and HttpClient.stop removes the threads
2. HttpClient.stop is called in the finalize method of HttpClientTerminus
3. Sometimes stop calls displays ClosedChannelException in DEBUG mode. At that time the threads are not removed.
   Would appreciate if you can comment if there were known issues in this area, any fixes, workaround to unblock the Customer.

NOTE: We can get into a meeting with the jetty team to share more data points & finer details of the anlysis

Sample stacktrace of HttpClient threads
HttpClient@74902a3f-141028" #141028 prio=5 os_prio=0 tid=0x00007f8ee90d6000
nid=0x13cfc runnable [0x00007f896b8f7000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

• locked <0x00000004c4d014f8> (a sun.nio.ch.Util$3)
• locked <0x00000004c4d014e8> (a java.util.Collections$UnmodifiableSet)
• locked <0x00000004c4d014a0> (a sun.nio.ch.EPollSelectorImpl)
  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
  at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:183)
  at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:190)
  at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.j

[sbordet]

Jetty 9 is at End of Community Support (#7958) as well as Jetty 10 and 11 (#10485).
Try with Jetty 12.0.x and see if the problem persists.

Having said that, likely it is not a Jetty problem.
The stack trace you reported is normal (it is a selector thread).

If you want commercial support from the Jetty team, contact Webtide.
See also: https://github.com/jetty/jetty.project?tab=readme-ov-file#commercial-support.

[satishtedla]

Thanks Simone for the quick respomse.

The stack trace you reported is normal (it is a selector thread).
HttpClient.stop() is throwing "ClosedChannelException in DEBUG mode" and this is the time when we see that threads are just staying for a long time..
When no error is thrown, we do not see any issue

Question: Were there any issues reported & fixed in the newer versions?

[joakime]

There have been ...

[jetty-client][jetty-11.0.x]$ git log --oneline jetty-9.4.53.v20231009...jetty-11.0.20 -- . | wc -l
696

696 commits to jetty-client between Jetty 9 and Jetty 11.

Then Jetty 12 underwent a reorganization and ...

[jetty-core/jetty-client][jetty-12.0.x]$ git log --oneline -- . | wc -l
145

There are an additional 145 commits on top of that.
Yes, there's MANY changes to jetty-client since Jetty 9.

And that's just the jetty-client artifact.
Which doesn't count the changes in the dependencies like jetty-io, jetty-http, jetty-util, etc ...

[satishtedla]

@joakime That's a lot of changes. We would definitely consider consuming the latest version in the product but it needs to go through a rigorous testing cycle (which would take time).

As the Customers are running into issues in their production env, would it be possible to suggest any workarounds in this area to unblock them?

[joakime]

As the Customers are running into issues in their production env, would it be possible to suggest any workarounds in this area to unblock them?

See #11366 (comment)

[joakime]

Also, see https://eclipse.dev/jetty/security_reports.php

Your version of Jetty is not up to date and is subject to several advisories.

[ekupcik]

2. HttpClient.stop is called in the finalize method of HttpClientTerminus

Just guessing here but anything relying on finalize() is just asking for trouble

[joakime]

2. HttpClient.stop is called in the finalize method of HttpClientTerminus

Just guessing here but anything relying on finalize() is just asking for trouble

Also, as of JDK 9 the method Object.finalize() was deprecated and flagged for removal in a future JDK release.

https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/lang/Object.html#finalize()

[jmcc0nn3ll]

If you need to talk about commercial support you can also contact me directly at sales @ webtide.com and I would be happy to discuss.