Intermittent invalid sdk key error when restarting relay proxy

[simonkotwicz]

Describe the bug
If you use the relay proxy automatic configuration profile and restart the relay proxy while some clients are running you sometimes get a 401 invalid SDK key error. Seems to be some kind of race condition where the relay proxy server doesn't wait and properly initialize itself before starting to listen for connections.

To reproduce

• Start the relay proxy using an automatic configuration profile (eg. docker run --rm -p 8030:8030 --name ld-relay -e AUTO_CONFIG_KEY="***" launchdarkly/ld-relay)
• Start an ld client that talks to the relay proxy
• Restart the relay proxy
• The client sometimes gets an error ERROR: Error in stream connection (giving up permanently): HTTP error 401 (invalid SDK key)

Relay version
launchdarkly/ld-relay@sha256:2a3bdc8d8adf774a1631d7f13bacac5a8e9e89cdb2ce84f662104a3bb4be11fa

Client code

import (
	"time"

	ld "gopkg.in/launchdarkly/go-server-sdk.v5"
	"gopkg.in/launchdarkly/go-server-sdk.v5/ldcomponents"
)

func main() {
	var config ld.Config
	relayURI := "http://localhost:8030"
	config.DataSource = ldcomponents.StreamingDataSource().BaseURI(relayURI)
	ld.MakeCustomClient("***", config, 5*time.Second)
}

may be related to #112

[eli-darkly]

Hmm. Even though the symptoms are similar, I don't think this is related to the other problem you reported— that one had a very specific cause. Since this one is intermittent it may be harder to reproduce, but I think I know the area of the code to look at; investigating.

[simonkotwicz]

Shouldn't be too difficult to reproduce, it happens fairly often for me (>10% of the time)

[eli-darkly]

@simonkotwicz I do know the cause and can reproduce it with test-instrumented versions of the Relay components, but I think it'll inevitably be harder for us to reproduce with a real Relay instance in our test environment than it is for you, because we have very low latency for requests to our own LD servers. The longer it takes for Relay to contact LD and obtain the auto-config data, the more likely it is for a client to be able to get in with a request during that interval.

This one is slightly harder to fix than the other bug, because we need to add a kind of synchronization that wasn't present before, but we're aiming to include fixes for both of these in the next patch release soon.

[simonkotwicz]

Oh ok, it should be possible to reproduce just on a local machine running the ld relay docker image and client locally.

[eli-darkly]

We've released version 6.1.1, which should fix this. The behavior now should be that if an SDK client makes a request while the Relay Proxy is still starting up and hasn't received its configuration from LD yet, so it does not know about any SDK keys yet, the response will be a 503 error rather than a 401. A 503 will cause the client to retry using a backoff. This is consistent with how the Relay Proxy behaves (regardless of whether automatic configuration is enabled) if a client gets a request in early during startup when we do know the SDK for the environment, but haven't yet received the flags for that environment from LD.