Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Java heap space #504

Open
HMubaireek opened this issue May 28, 2024 · 1 comment
Open

Java heap space #504

HMubaireek opened this issue May 28, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@HMubaireek
Copy link

Describe the bug
I'm using zeeqs with hazelcast & postgresql database

Occasionally, I get this Java heap space error, and I have to delete the pod to get this working again.

To Reproduce
Steps to reproduce the behavior:

  1. It's hard for me to find replication steps.

Expected behavior
I expect this to be handled. Probably there's a memory leak somewhere?

GraphQL request
N/A

Environment (please complete the following information):

  • Version [2.9.1]

Additional context

2024-05-28 12:56:56.826 ERROR 1 --- [nt_1.internal-3] c.h.s.i.executionservice.TaskScheduler   : hz.client_1 [dev] [5.3.6] Failed to execute java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@321a7a56[Completed exceptionally: java.lang.OutOfMemoryError: Java heap space]

java.lang.OutOfMemoryError: Java heap space

2024-05-28 12:56:56.826 DEBUG 1 --- [nio-9000-exec-1] o.s.w.c.request.async.WebAsyncManager    : Started async request
2024-05-28 12:56:56.826 DEBUG 1 --- [nio-9000-exec-1] o.s.w.c.request.async.WebAsyncManager    : Async error, dispatch to /graphql
2024-05-28 12:56:56.826 DEBUG 1 --- [nio-9000-exec-1] o.s.web.servlet.DispatcherServlet        : Exiting but response remains open for further handling
2024-05-28 12:56:56.826 DEBUG 1 --- [nio-9000-exec-1] o.s.web.servlet.DispatcherServlet        : "ASYNC" dispatch for POST "/graphql", parameters={}
2024-05-28 12:56:56.827 DEBUG 1 --- [nio-9000-exec-1] o.s.w.s.f.s.HandlerFunctionAdapter       : Resume with async result [java.lang.OutOfMemoryError: Java heap space]
2024-05-28 12:56:56.827 DEBUG 1 --- [nio-9000-exec-1] o.s.web.servlet.DispatcherServlet        : Unresolved failure from "ASYNC" dispatch: javax.servlet.ServletException: Async processing failed
2024-05-28 12:56:56.827 ERROR 1 --- [nio-9000-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] threw exception

java.lang.OutOfMemoryError: Java heap space

2024-05-28 12:56:56.827 ERROR 1 --- [nio-9000-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Async processing failed] with root cause

java.lang.OutOfMemoryError: Java heap space

2024-05-28 12:56:56.827 DEBUG 1 --- [nio-9000-exec-1] o.s.web.servlet.DispatcherServlet        : "ERROR" dispatch for POST "/error", parameters={}
2024-05-28 12:56:56.828 DEBUG 1 --- [nio-9000-exec-1] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController#error(HttpServletRequest)
2024-05-28 12:56:56.828 DEBUG 1 --- [nio-9000-exec-1] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Using 'application/json', given [*/*] and supported [application/json, application/*+json, application/json, application/*+json]
2024-05-28 12:56:56.828 DEBUG 1 --- [nio-9000-exec-2] o.s.w.s.f.s.HandlerFunctionAdapter       : Resume with async result [java.lang.OutOfMemoryError: Java heap space]
2024-05-28 12:56:56.828 DEBUG 1 --- [nio-9000-exec-2] o.s.web.servlet.DispatcherServlet        : Unresolved failure from "ASYNC" dispatch: javax.servlet.ServletException: Async processing failed
2024-05-28 12:56:56.828 ERROR 1 --- [nio-9000-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] threw exception

java.lang.OutOfMemoryError: Java heap space

2024-05-28 12:56:56.829 ERROR 1 --- [nio-9000-exec-2] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Async processing failed] with root cause

java.lang.OutOfMemoryError: Java heap space

2024-05-28 12:56:56.829 DEBUG 1 --- [nio-9000-exec-2] o.s.web.servlet.DispatcherServlet        : "ERROR" dispatch for POST "/error", parameters={}
2024-05-28 12:56:56.829 DEBUG 1 --- [nio-9000-exec-2] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController#error(HttpServletRequest)
2024-05-28 12:56:56.829 DEBUG 1 --- [nio-9000-exec-2] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Using 'application/json', given [*/*] and supported [application/json, application/*+json, application/json, application/*+json]
2024-05-28 12:56:56.830 DEBUG 1 --- [nio-9000-exec-2] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Writing [{timestamp=Tue May 28 12:56:56 UTC 2024, status=500, error=Internal Server Error, path=/graphql}]
2024-05-28 12:56:58.517 DEBUG 1 --- [     parallel-1] o.s.w.s.adapter.NativeWebSocketSession   : Closing StandardWebSocketSession[id=8ad6e8e3-b148-cf73-ec94-55861214bf71, uri=wss://zeebe-zeeq.staging.work-permit.cntxt.ai/graphql]
2024-05-28 12:56:58.518 DEBUG 1 --- [     parallel-1] s.w.s.h.LoggingWebSocketHandlerDecorator : StandardWebSocketSession[id=8ad6e8e3-b148-cf73-ec94-55861214bf71, uri=wss://zeebe-zeeq.staging.work-permit.cntxt.ai/graphql] closed with CloseStatus[code=4408, reason=Connection initialisation timeout]
2024-05-28 12:57:01.941 DEBUG 1 --- [nio-9000-exec-2] o.s.web.servlet.DispatcherServlet        : Exiting from "ERROR" dispatch, status 500
2024-05-28 12:56:56.830 DEBUG 1 --- [nio-9000-exec-1] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Writing [{timestamp=Tue May 28 12:56:56 UTC 2024, status=500, error=Internal Server Error, path=/graphql}]
2024-05-28 12:57:01.942 DEBUG 1 --- [nio-9000-exec-1] o.s.web.servlet.DispatcherServlet        : Exiting from "ERROR" dispatch, status 500
Exception in thread "Catalina-utility-2" java.lang.OutOfMemoryError: Java heap space
2024-05-28 12:57:58.225  WARN 1 --- [nt_1.internal-1] c.h.c.i.connection.tcp.HeartbeatManager  : hz.client_1 [dev] [5.3.6] Heartbeat failed over the connection: ClientConnection{alive=true, connectionId=4, channel=NioChannel{/10.xx.x.40:51665->/10.xx.x.x:5701}, remoteAddress=[10.xx.x.x]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=never}
2024-05-28 12:57:58.226  WARN 1 --- [nt_1.internal-1] c.h.c.i.c.tcp.TcpClientConnection        : hz.client_1 [dev] [5.3.6] ClientConnection{alive=false, connectionId=4, channel=NioChannel{/10.xx.x.40:51665->/10.xx.x.x:5701}, remoteAddress=[10.xx.x.x]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=2024-05-28 12:57:58.225} closed. Reason: Heartbeat timed out

com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=4, channel=NioChannel{/10.xx.x.40:51665->/10.xx.x.x:5701}, remoteAddress=[10.xx.x.x]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=never}
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.check(HeartbeatManager.java:104) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.run(HeartbeatManager.java:85) ~[hazelcast-5.3.6.jar:5.3.6]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) ~[hazelcast-5.3.6.jar:5.3.6]

2024-05-28 12:58:04.593 DEBUG 1 --- [nio-9000-exec-5] org.hibernate.SQL                        :
    select
        processins0_.key_ as col_0_0_
    from
        process_instance processins0_
    where
        processins0_.process_definition_key=?
        and (
            processins0_.state in (
                ? , ? , ?
            )
        )
2024-05-28 12:58:06.707 ERROR 1 --- [pool-1-thread-1] i.z.h.connect.java.ZeebeHazelcast        : Fail to read from ring-buffer at sequence '78462'. Will try again.

com.hazelcast.core.OperationTimeoutException: ClientInvocation{clientMessage = ClientMessage{connection=null, length=41, operation=Ringbuffer.ReadOne, isRetryable=true, correlationId=153252, messageType=170700, isEvent=false, isfragmented=false}, objectName = zeebe, target = partition 134, sentConnection = null} timed out because exception occurred after client invocation timeout 120000 ms. Current time: 2024-05-28 12:57:58.228. Start time: 2024-05-28 12:48:00.095. Total elapsed time: 598133 ms.
        at com.hazelcast.client.impl.spi.impl.ClientInvocation.notifyExceptionWithOwnedPermission(ClientInvocation.java:340) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:221) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onConnectionClose(TcpClientConnectionManager.java:849) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.close(TcpClientConnection.java:199) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.check(HeartbeatManager.java:104) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.run(HeartbeatManager.java:85) ~[hazelcast-5.3.6.jar:5.3.6]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) ~[hazelcast-5.3.6.jar:5.3.6]
        at ------ submitted from ------.() ~[na:na]
        at com.hazelcast.internal.util.ExceptionUtil.cloneExceptionWithFixedAsyncStackTrace(ExceptionUtil.java:348) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.spi.impl.AbstractInvocationFuture.wrapRuntimeException(AbstractInvocationFuture.java:1975) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.spi.impl.AbstractInvocationFuture.wrapOrPeel(AbstractInvocationFuture.java:1956) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.spi.impl.AbstractInvocationFuture$ExceptionalResult.wrapForJoinInternal(AbstractInvocationFuture.java:1438) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.spi.impl.AbstractInvocationFuture.resolveAndThrowForJoinInternal(AbstractInvocationFuture.java:601) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.spi.impl.AbstractInvocationFuture.joinInternal(AbstractInvocationFuture.java:585) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.proxy.ClientRingbufferProxy.invoke(ClientRingbufferProxy.java:220) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.proxy.ClientRingbufferProxy.readOne(ClientRingbufferProxy.java:156) ~[hazelcast-5.3.6.jar:5.3.6]
        at io.zeebe.hazelcast.connect.java.ZeebeHazelcast.readNext(ZeebeHazelcast.java:119) ~[zeebe-hazelcast-connector-1.4.0.jar:1.4.0]
        at io.zeebe.hazelcast.connect.java.ZeebeHazelcast.readFromBuffer(ZeebeHazelcast.java:111) ~[zeebe-hazelcast-connector-1.4.0.jar:1.4.0]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
Caused by: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out
        at com.hazelcast.client.impl.spi.impl.ClientInvocationServiceImpl.onConnectionClose(ClientInvocationServiceImpl.java:220) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onConnectionClose(TcpClientConnectionManager.java:849) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.TcpClientConnection.close(TcpClientConnection.java:199) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.check(HeartbeatManager.java:104) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.run(HeartbeatManager.java:85) ~[hazelcast-5.3.6.jar:5.3.6]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) ~[hazelcast-5.3.6.jar:5.3.6]
Caused by: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=4, channel=NioChannel{/10.xx.x.40:51665->/10.xx.x.x:5701}, remoteAddress=[10.xx.x.x]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=never}
        ... 10 common frames omitted

2024-05-28 12:58:06.708  INFO 1 --- [nt_1.internal-1] c.h.c.i.c.ClientConnectionManager        : hz.client_1 [dev] [5.3.6] Removed connection to endpoint: [10.xx.x.x]:5701:dd88754e-d76d-493f-8ad0-6e27f83b491e, connection: ClientConnection{alive=false, connectionId=4, channel=NioChannel{/10.xx.x.40:51665->/10.xx.x.x:5701}, remoteAddress=[10.xx.x.x]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=2024-05-28 12:57:58.225}
2024-05-28 12:58:06.709  WARN 1 --- [nt_1.internal-1] c.h.c.i.connection.tcp.HeartbeatManager  : hz.client_1 [dev] [5.3.6] Heartbeat failed over the connection: ClientConnection{alive=true, connectionId=3, channel=NioChannel{/10.xx.x.40:33243->/10.xx.5.5:5701}, remoteAddress=[10.xx.5.5]:5701, lastReadTime=2024-05-28 12:56:57.668, lastWriteTime=2024-05-28 12:57:12.392, closedTime=never}
2024-05-28 12:58:06.710  WARN 1 --- [nt_1.internal-1] c.h.c.i.c.tcp.TcpClientConnection        : hz.client_1 [dev] [5.3.6] ClientConnection{alive=false, connectionId=3, channel=NioChannel{/10.xx.x.40:33243->/10.xx.5.5:5701}, remoteAddress=[10.xx.5.5]:5701, lastReadTime=2024-05-28 12:56:57.668, lastWriteTime=2024-05-28 12:57:12.392, closedTime=2024-05-28 12:58:06.710} closed. Reason: Heartbeat timed out

com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=3, channel=NioChannel{/10.xx.x.40:33243->/10.xx.5.5:5701}, remoteAddress=[10.xx.5.5]:5701, lastReadTime=2024-05-28 12:56:57.668, lastWriteTime=2024-05-28 12:57:12.392, closedTime=never}
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.check(HeartbeatManager.java:104) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.run(HeartbeatManager.java:85) ~[hazelcast-5.3.6.jar:5.3.6]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) ~[hazelcast-5.3.6.jar:5.3.6]

2024-05-28 12:58:06.711  INFO 1 --- [nt_1.internal-1] c.h.c.i.c.ClientConnectionManager        : hz.client_1 [dev] [5.3.6] Removed connection to endpoint: [10.xx.5.5]:5701:e0923507-8901-4736-9a2b-84623e8dc861, connection: ClientConnection{alive=false, connectionId=3, channel=NioChannel{/10.xx.x.40:33243->/10.xx.5.5:5701}, remoteAddress=[10.xx.5.5]:5701, lastReadTime=2024-05-28 12:56:57.668, lastWriteTime=2024-05-28 12:57:12.392, closedTime=2024-05-28 12:58:06.710}
2024-05-28 12:58:06.712  WARN 1 --- [nt_1.internal-1] c.h.c.i.connection.tcp.HeartbeatManager  : hz.client_1 [dev] [5.3.6] Heartbeat failed over the connection: ClientConnection{alive=true, connectionId=2, channel=NioChannel{/10.xx.x.40:60723->/10.xx.1.17:5701}, remoteAddress=[10.xx.1.17]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=never}
2024-05-28 12:58:06.712  WARN 1 --- [nt_1.internal-1] c.h.c.i.c.tcp.TcpClientConnection        : hz.client_1 [dev] [5.3.6] ClientConnection{alive=false, connectionId=2, channel=NioChannel{/10.xx.x.40:60723->/10.xx.1.17:5701}, remoteAddress=[10.xx.1.17]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=2024-05-28 12:58:06.712} closed. Reason: Heartbeat timed out

com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=2, channel=NioChannel{/10.xx.x.40:60723->/10.xx.1.17:5701}, remoteAddress=[10.xx.1.17]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=never}
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.check(HeartbeatManager.java:104) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.client.impl.connection.tcp.HeartbeatManager$HeartbeatChecker.run(HeartbeatManager.java:85) ~[hazelcast-5.3.6.jar:5.3.6]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[hazelcast-5.3.6.jar:5.3.6]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) ~[hazelcast-5.3.6.jar:5.3.6]

2024-05-28 12:58:06.713  INFO 1 --- [nt_1.internal-1] c.h.c.i.c.ClientConnectionManager        : hz.client_1 [dev] [5.3.6] Removed connection to endpoint: [10.xx.1.17]:5701:c3154107-c8e8-4b1a-956b-3aa6771e6adf, connection: ClientConnection{alive=false, connectionId=2, channel=NioChannel{/10.xx.x.40:60723->/10.xx.1.17:5701}, remoteAddress=[10.xx.1.17]:5701, lastReadTime=2024-05-28 12:56:56.826, lastWriteTime=2024-05-28 12:57:12.392, closedTime=2024-05-28 12:58:06.712}
2024-05-28 12:58:06.713  INFO 1 --- [nt_1.internal-1] com.hazelcast.core.LifecycleService      : hz.client_1 [dev] [5.3.6] HazelcastClient 5.3.6 (20231109 - 9903dc9) is CLIENT_DISCONNECTED
2024-05-28 12:58:23.537  INFO 1 --- [nt_1.internal-3] c.h.c.i.c.ClientConnectionManager        : hz.client_1 [dev] [5.3.6] Trying to connect to cluster: dev
2024-05-28 12:58:23.539  INFO 1 --- [nt_1.internal-3] c.h.c.i.c.ClientConnectionManager        : hz.client_1 [dev] [5.3.6] Trying to connect to Member [10.xx.5.5]:5701 - e0923507-8901-4736-9a2b-84623e8dc861
2024-05-28 12:58:31.946 ERROR 1 --- [nt_1.internal-3] c.h.s.i.executionservice.TaskScheduler   : hz.client_1 [dev] [5.3.6] Failed to execute java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3125dee2[Completed exceptionally: java.lang.OutOfMemoryError: Java heap space]

java.lang.OutOfMemoryError: Java heap space

2024-05-28 12:58:31.947 ERROR 1 --- [nt_1.internal-2] c.h.s.i.executionservice.TaskScheduler   : hz.client_1 [dev] [5.3.6] Failed to execute java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@5e10b6eb[Completed exceptionally: java.lang.OutOfMemoryError: Java heap space]

java.lang.OutOfMemoryError: Java heap space
@HMubaireek HMubaireek added the bug Something isn't working label May 28, 2024
@saig0
Copy link
Contributor

saig0 commented Jun 12, 2024

@HMubaireek thank you for reporting. 👍

I didn't face an OutOfMemory error with ZeeQS before. To analyze the case, we must create a Java heap dump when the error occurs.

You should also check the resource limits and increase the memory if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@saig0 @HMubaireek