Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory Leak in ZettaHttpServer - PeerSockets #294

Open
kevinswiber opened this issue Mar 17, 2016 · 5 comments
Open

Memory Leak in ZettaHttpServer - PeerSockets #294

kevinswiber opened this issue Mar 17, 2016 · 5 comments

Comments

@kevinswiber
Copy link
Member

When one Zetta peer connects to another, the receiver stores a PeerSocket in its ZettaHttpServer#peers object. When the initiating peer disconnects, its PeerSocket remains on the receiving peer. This was likely an intentional performance optimization for use cases that only ever have a few of the same peers connecting. In scenarios where hundreds, perhaps thousands of peers are connecting, the receiving peer's ZettaHttpServer#peers object continues to grow. Some of these peers may disconnect and never return.

We have a few options to address this:

  1. delete the peer from ZettaHttpPeers#peers on _peer/disconnect
  2. Set a buffer limit for dead PeerSocket instances. Prune this buffer on an interval.
  3. The same as Option 1, except with a timeout that is fired should the initiating peer not reconnect during a specified time duration.
@kevinswiber
Copy link
Member Author

@AdamMagaluk @mdobson Thoughts? This is currently an issue in our custom cloud deployment.

@landlessness, you may be experiencing some ill effects from this, too.

@kevinswiber
Copy link
Member Author

A workaround, in the meantime, implementing Option 1 above:

zetta()
  .name('cloud')
  .use(function(runtime) {
    runtime.pubsub.subscribe('_peer/disconnect', function(ev, msg) {
      delete runtime.httpServer.peers[msg.peer.name];
    });
  })
  .listen(process.env.PORT || 1337);

@AdamMagaluk
Copy link
Collaborator

@kevinswiber The main (possibly only reason, I forget) we are keeping the peer_sockets around is to keep the list of event subscriptions from clients/apps to that peer and subscribe to them when they reconnect.

Of course this can be done differently by using the event_broker or other internals to get a list of subscriptions for that peer. We could also delete the peer once all of it's subscriptions are gone (clients disconnecting), though I don't think that would ever be the case for our deployment.

@kevinswiber
Copy link
Member Author

@AdamMagaluk Thanks for the assessment. Let's abstract that away. What can we do about unbounded growth on leftover subscriptions? Do we need to institute a timeout between the lack of availability of connected device streams and the subscriptions to those streams? Is there a straightforward solution? Perhaps we could use a circular buffer that holds dead subscriptions and starts dropping them on overflow?

Unbounded growth is not only hurting memory consumption; it's also an attack vector.

@AdamMagaluk
Copy link
Collaborator

@kevinswiber I think the best approach would be to delete the peer_socket from .peers after a disconnect and use the data available from the clients/apps to re-subscribe to topics on the peer when a new peer_socket is created.

In this case zetta would only keep subscriptions and peers when they are currently connected. We would that just have to worry about an unbounded number of peers connecting, clients connecting, and subscriptions from clients but that solved pretty easily by some limits.

I think it would be pretty straightforward, may take a bit code and figuring out how to access the proper data but it's all there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants