Some subscriptions are lost when reconnecting

[Gutaicheng]

After I read the source code of jnatsdk, I found that the method of sending subscription information is different when connecting for the first time and reconnecting (I use Dispatcher to subscribe).

The first connection will execute queueOutgoing, and the reconnection will execute queueInternalOutgoing;

There is no limit to the MessageQueue when reconnecting. When all clients reconnect and resubscribe, the subscription will be lost.
I have observed this under the following conditions:

1. 5 client connections, each client connection subscribes to 300,000 topics
2. When connecting to the server for the first time, each client can subscribe to 300,000
3. After closing and opening the server, the client began to reconnect and re-subscribe, but the number of subscribed topics was less than 300,000

I found that the connectionTimeout time set by the client will affect the result of resubscribing. If I change the default 2s to 5s, then 300,000 topics will be fully subscribed when reconnecting.

I think it is unwise to modify the connectionTimeout to ensure that the subscription is not lost when reconnecting. Is there any other solution to ensure that the subscription is not lost when reconnecting?

[derekcollison]

Looping in @scottf

[scottf]

@Gutaicheng So I have a few questions, but let's consider your use case first. You have 300,000 re-subscriptions trying to happen, the server simply cannot do that amount of work in the short time given. I think the making the timeout longer is exactly why it's available, for you to tune your specific case to actual behavior. How big is your cluster? The more servers that this work is split between will make a difference.
Can you explain how you setup your subjects and give an example of how each connection subscribes to that many subjects? Are you subscribing via wildcard, or are there literally 300,000 subscription instances (60,000 per connection)
What kind of consumers are these? Push/pull consumers? Are they durable consumers or are they ephemeral consumers with default or some other inactivity threshold? Is the stream file or memory based and what is the replication factor on the stream and the consumer? These are all pieces of the tuning puzzle.

[Gutaicheng]

@scottf thanks ！I use the Publish-Subscribe mode of Core NATS, and I have a cluster of three nodes.
At present, it is still in the research stage, and it is tested in accordance with the leader's idea. I use Java code to create 5 client connections to connect to one of the nodes and reconnect only to this node, and will not try to connect to other nodes of the cluster. Each client connection is subscribed through the following code:
for (String s : subjects) { //This subjects list contains 300000 different topics from topic-0 to topic-299999 dispatcher.subscribe(s); }
I didn't start to push the message, because I have to wait for the client subscription to complete. All clients will subscribe to 300,000 topics after the first successful connection. If the node is restarted, the client will not fully subscribe when it reconnects to the server, about 200,000 topics. As for the type of consumer you mentioned, etc., I'm not taking any restrictions on subscribing clients, everything is default.

emm，If I understand that sending subscription information when resubscribing is equivalent to pushing messages from the publisher client, it should not take so long

[scottf]

Individually it does not take that long. It's all the work being done at once.

[Gutaicheng]

@scottf I also think it is very fast, about 5-10MB/s, but the size of the subscription information sent when reconnecting is not large, so why some subscriptions become invalid, I am very concerned...

Emmm, I confirm that it is because connectionTimeout affects the integrity of the subscription when reconnecting. The restart program will only cancel the reconnection status after the subscription cycle is completed. If the connectionTimeout time is reached, subsequent subscriptions will be cancelled.

`this.dispatchers.forEach((nuid, d) -> {
                        if (!d.isDraining()) {
                            d.resendSubscriptions();
                        }

                    });

                    try {
                        this.flush(this.options.getConnectionTimeout());
                    } catch (Exception var10) {
                        this.processException(var10);
                    }

                    this.writer.setReconnectMode(false);
                    this.processConnectionEvent(Events.RESUBSCRIBED);`

`void resendSubscriptions() {
        this.subscriptionsUsingDefaultHandler.forEach((id, sub) -> {
            this.connection.sendSubscriptionMessage(sub.getSID(), sub.getSubject(), sub.getQueueName(), true);
        });
        this.subscriptionsWithHandlers.forEach((sid, sub) -> {
            this.connection.sendSubscriptionMessage(sub.getSID(), sub.getSubject(), sub.getQueueName(), true);
        });
    }`

It is unwise to set a long connectionTimeout in a production environment. Is this a place that can be optimized? or are there other measures to ensure that a large number of subscriptions will be intact when reconnecting

[scottf]

1. Why is it unwise to set a long connection timeout in a production environment? For secure connections where handshake and certificates are involved, we often suggest the timeout be raised to 10 seconds.
2. Consider if you have a program that has with a very large number of requests. It takes time to execute them, there is no physical way it could not. There are multiple connections to a server, the server is processing, in order, multiple requests. As you said it is able to handle most of them within a short timeframe, but some cannot finish in that timeframe. The only way is to increase the time allowed.