Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broker's web threads get stuck #4756

Closed
massakam opened this issue Jul 18, 2019 · 4 comments · Fixed by #6124
Closed

Broker's web threads get stuck #4756

massakam opened this issue Jul 18, 2019 · 4 comments · Fixed by #6124
Labels
type/bug The PR fixed a bug or issue reported a bug

Comments

@massakam
Copy link
Contributor

The other day, one of our broker servers became unable to respond to any HTTP request. The following is a part of the thread dump at that time.

threaddump.txt

It seems that many pulsar-web-x-x threads are blocked trying to get a list of subscriptions.

pulsar-web-28-70" #8062317 prio=5 os_prio=0 tid=0x00007fb54c001000 nid=0xd261 waiting on condition [0x00007fb48d0b4000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007fc91e010000> (a java.util.concurrent.CompletableFuture$Signaller)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
    at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
    at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
    at org.apache.pulsar.client.admin.internal.TopicsImpl.getSubscriptions(TopicsImpl.java:349)
    at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalGetSubscriptions(PersistentTopicsBase.java:532)
    at org.apache.pulsar.broker.admin.v1.PersistentTopics.getSubscriptions(PersistentTopics.java:230)
    at sun.reflect.GeneratedMethodAccessor131.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
    at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:53)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
    at org.apache.pulsar.broker.web.AuthenticationFilter.doFilter(AuthenticationFilter.java:74)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
    at org.eclipse.jetty.server.Server.handle(Server.java:524)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
    at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
    at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)

Pulsar version

  • broker: 2.2.1

We have already backported #4484 to Pulsar-2.2.1, but the REST API that returns a list of subscriptions does not seem to time out.

@massakam massakam added the type/bug The PR fixed a bug or issue reported a bug label Jul 18, 2019
@hrsakai
Copy link
Contributor

hrsakai commented Jul 19, 2019

It reproduces with the following command.

seq 1 100 | xargs -I ZZ -P 100 curl http://xxxxx:4080/admin/persistent/tenant1/global/ns1/partition-topic1/subscriptions

@sijie
Copy link
Member

sijie commented Jul 21, 2019

@massakam does #4765 fix this issue you described here?

@massakam
Copy link
Contributor Author

@sijie No, it doesn't. Probably, we need to modify the API for getting list of subscriptions as well as #4765 in order to fix this issue. I intend to send a pull-request for that later.

sijie pushed a commit that referenced this issue Jul 23, 2019
Master Issue: #4756

### Motivation

This is a continuation of #4765.

### Modifications

Added async rest handlers to the following APIs:
```
DELETE /admin/namespaces/{tenant}/{cluster}/{namespace}
PUT    /admin/namespaces/{tenant}/{cluster}/{namespace}/unload
POST   /admin/namespaces/{tenant}/{cluster}/{namespace}/clearBacklog
POST   /admin/namespaces/{tenant}/{cluster}/{namespace}/clearBacklog/{subscription}
POST   /admin/namespaces/{tenant}/{cluster}/{namespace}/unsubscribe/{subscription}

DELETE /admin/v2/namespaces/{tenant}/{namespace}
PUT    /admin/v2/namespaces/{tenant}/{namespace}/unload
POST   /admin/v2/namespaces/{tenant}/{namespace}/clearBacklog
POST   /admin/v2/namespaces/{tenant}/{namespace}/clearBacklog/{subscription}
POST   /admin/v2/namespaces/{tenant}/{namespace}/unsubscribe/{subscription}
```
merlimat pushed a commit that referenced this issue Jul 25, 2019
…Is (3) (#4795)

* Process requests asynchronously on some REST APIs (3)

* Add async rest handler to API for expiring message on single topic subscription
easyfan pushed a commit to easyfan/pulsar that referenced this issue Jul 26, 2019
Master Issue: apache#4756

### Motivation

This is a continuation of apache#4765.

### Modifications

Added async rest handlers to the following APIs:
```
DELETE /admin/namespaces/{tenant}/{cluster}/{namespace}
PUT    /admin/namespaces/{tenant}/{cluster}/{namespace}/unload
POST   /admin/namespaces/{tenant}/{cluster}/{namespace}/clearBacklog
POST   /admin/namespaces/{tenant}/{cluster}/{namespace}/clearBacklog/{subscription}
POST   /admin/namespaces/{tenant}/{cluster}/{namespace}/unsubscribe/{subscription}

DELETE /admin/v2/namespaces/{tenant}/{namespace}
PUT    /admin/v2/namespaces/{tenant}/{namespace}/unload
POST   /admin/v2/namespaces/{tenant}/{namespace}/clearBacklog
POST   /admin/v2/namespaces/{tenant}/{namespace}/clearBacklog/{subscription}
POST   /admin/v2/namespaces/{tenant}/{namespace}/unsubscribe/{subscription}
```
easyfan pushed a commit to easyfan/pulsar that referenced this issue Jul 26, 2019
…EST APIs (3) (apache#4795)

* Process requests asynchronously on some REST APIs (3)

* Add async rest handler to API for expiring message on single topic subscription
@shustsud
Copy link
Contributor

shustsud commented Aug 2, 2019

For now the stuck at getSubscriptions seems to be fixed by the following PRs:

#4765
#4778
#4795

However, besides the stuck at getSubscriptions, there is another stuck at getStats.
pulsar-web threads are blocked at NamespaceService#getWebServiceUrl at getStats, and I think the above PRs do not fix it:

"pulsar-web-28-66" #1702882 prio=5 os_prio=0 tid=0x00007fb558067000 nid=0xb753 waiting on condition [0x00007fb3b80e7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x00007fc91e030020> (a java.util.concurrent.CompletableFuture$Signaller)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.apache.pulsar.broker.namespace.NamespaceService.getWebServiceUrl(NamespaceService.java:191)
at org.apache.pulsar.broker.web.PulsarWebResource.validateTopicOwnership(PulsarWebResource.java:543)
at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalGetStats(PersistentTopicsBase.java:562)
at org.apache.pulsar.broker.admin.v1.PersistentTopics.getStats(PersistentTopics.java:242)
at sun.reflect.GeneratedMethodAccessor48.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1689)
at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:53)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.apache.pulsar.broker.web.AuthenticationFilter.doFilter(AuthenticationFilter.java:74)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:524)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

wolfstudy pushed a commit that referenced this issue Nov 20, 2019
Master Issue: #4756

### Motivation

This is a continuation of #4765.

### Modifications

Added async rest handlers to the following APIs:
```
DELETE /admin/namespaces/{tenant}/{cluster}/{namespace}
PUT    /admin/namespaces/{tenant}/{cluster}/{namespace}/unload
POST   /admin/namespaces/{tenant}/{cluster}/{namespace}/clearBacklog
POST   /admin/namespaces/{tenant}/{cluster}/{namespace}/clearBacklog/{subscription}
POST   /admin/namespaces/{tenant}/{cluster}/{namespace}/unsubscribe/{subscription}

DELETE /admin/v2/namespaces/{tenant}/{namespace}
PUT    /admin/v2/namespaces/{tenant}/{namespace}/unload
POST   /admin/v2/namespaces/{tenant}/{namespace}/clearBacklog
POST   /admin/v2/namespaces/{tenant}/{namespace}/clearBacklog/{subscription}
POST   /admin/v2/namespaces/{tenant}/{namespace}/unsubscribe/{subscription}
```

(cherry picked from commit 2cc34af)
wolfstudy pushed a commit that referenced this issue Nov 20, 2019
…Is (3) (#4795)

* Process requests asynchronously on some REST APIs (3)

* Add async rest handler to API for expiring message on single topic subscription

(cherry picked from commit 87c02a9)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
4 participants