Settings.InjectTopLevelFallback race condition can cause failures

[Aaronontheweb]

Version Information
Version of Akka.NET? v1.5.15
Which Akka.NET Modules? Akka.Cluster.Tools, others too presumably

Describe the bug

Consider the following method in the ClusterSingletonManagerSettings class:

akka.net/src/contrib/cluster/Akka.Cluster.Tools/Singleton/ClusterSingletonManagerSettings.cs

Lines 28 to 36 in 55644b1

	public static ClusterSingletonManagerSettings Create(ActorSystem system)
	{
	system.Settings.InjectTopLevelFallback(ClusterSingletonManager.DefaultConfig());
	var config = system.Settings.Config.GetConfig("akka.cluster.singleton");
	if (config.IsNullOrEmpty())
	throw ConfigurationException.NullOrEmptyConfig<ClusterSingletonManagerSettings>("akka.cluster.singleton");
	return Create(config).WithRemovalMargin(Cluster.Get(system).DowningProvider.DownRemovalMargin);

It uses system.Settings.InjectTopLevelFallback in order to make sure that its serialization definitions are available for Akka.Remote so all of the Cluster.Singleton-specific messages can be deserialized.

However, the way system.Settings.InjectTopLevelFallback works is inherently racy - multiple components all starting up at the same time might inject their configs all at once and we can end up "missing" a config injection, which causes the following line to fail occasionally:

akka.net/src/contrib/cluster/Akka.Cluster.Tools/Singleton/ClusterSingletonManagerSettings.cs

Lines 33 to 34 in 55644b1

	if (config.IsNullOrEmpty())
	throw ConfigurationException.NullOrEmptyConfig<ClusterSingletonManagerSettings>("akka.cluster.singleton");

Expected behavior

Plugins should be able to add their configurations to Akka.NET in a thread-safe, reliable manner that is also CTOR-friendly (no-Task).

Actual behavior

There is a small chance that the plugin configuration never gets properly registered and it crash - it's also possible for the default HOCON to get quite large as a result of many plugins multiplicatively registering their configuration values.

Additional context

We had talked about solving this problem in v1.4, when we attempted to replace our built-in HOCON implementation with https://github.com/akkadotnet/HOCON. The suggestion at the time was to have some means looking for known configuration sections and then staging them altogether inside the ActorSystem.

Given that #7246 is a huge goal for v1.6, any sort of assembly-scanning solution is off the table - what we probably need is something functionally similar to InjectTopLevelFallback now but with thread-safety.