Alternative to GC.Collect to free memory after calling AssemblyLoadContext.Unload

[dave-yotta]

A bit of reference in this issue here: #44679

We've now got the problem that rapidly loaded and unloaded AssemblyLoadContexts associated with a high volume of actions in our app leads to spikes in memory usage before the GC can clear it.

I understand it's not good to second guess the GC here - but we've added a full blocking collection following each call to Unload, the difference is a flat 300mb v.s. spiking upto around 2gb.

Normally I would not say this is a problem, but we're running this app in a containerised/kubernetes environment and it's causing problems:

• Tests running into the issue look like they are hitting memory limits and causing swapping to start (I get 100% 500mb/s on my machine) which grinds things to a halt
• kubernetes pods with resource limits are being terminated for violating them

Any advice?

Edit: Actually - there are no resource.limit set for the pod; after setting it the GC does respect it. I do have one worry some clever person might be able to address though:

When the GC collects the Assemblies that have been released by the AssemblyLoadContext after calling Unload, not only does this release the memory associated but also some file handles - does the GC also free stuff based on file handle pressure? We've had issues with "Too many open files" so it troubles me - although so long as it tracks the memory pressure maybe the file handle limit is a "higher ceiling" 🤔

[acaly]

does the GC also free stuff based on file handle pressure?

In terms of file handles, I think it might be related to SafeHandles, which wraps the OS file handle and release them when being collected. These handles are not treated specially by GC. They are just objects with finalizers (I have to say SafeHandle uses critical finalizer, but it has no effect on how GC collect the objects). I don't think the runtime should be using many handles for each assembly itself. Have you tried to Dispose all the objects you use as soon as possible, instead of leaving them for GC collection?

In general the case of frequently loading and unloading contexts might not have been considered when the runtime is optimized. If there are too many small, separate tasks, which requires small assemblies (for example, dynamically generated assemblies), it might be better to consider executing them in new processes.

And for the most important question, I don't think there is any alternatives to GC.Collect. It should be the only place you can ask GC to collect. However, in some cases the Dispose mentioned above might help a lot.

Also note that GC.Collect not only does the collection, but also promotes live objects to higher generation, which can be harmful to performance. If you do it too often soon all your objects will be moved to Gen2, and all future collections must be full collections in order to actually collect something.

[dave-yotta]

Thanks for the info!

I don't think the runtime should be using many handles for each assembly itself.

These are script snippets distinctly associated with each write operation for example, so we do reach easily 1000/s of these operations. I think it's more to do with this volume.

Have you tried to Dispose all the objects you use as soon as possible, instead of leaving them for GC collection?

These are held by assemblies loaded in the AssemblyLoadContext - they will get collected once Unload and releases the final strong reference to the loaded assemblies. I believe these come from MetadataReference to things like mscorlib.dll System.xxx.dll and so on that are made when we emit the assembly from our Roslyn compilation into the assembly load context - this is stuff that can only be released when the unload happens - I don't think Roslyn exposes and of those handles in a way we could try to close before the unload either.

it might be better to consider executing them in new processes

I have mentioned this one to the team - it's in the pipeline - also poignant from a security point of view. Bit tricky since it needs to be a daemon due to strict performance requirements (sync part of write pipeline) creating 30k processes would likely lead to other similar issues!

Also note that GC.Collect not only does the collection, but also promotes live objects to higher generation, which can be harmful to performance.

Yeah we're talking 1000s of collections per second so it's fairly crippling to performance - won't be doing this unless I do confirm any problem with those handles and then only as a last ditch effort to save production.

I might speak to the Roslyn team and see if there's any way to dispose resources associated with an emitted assembly before that assembly is finalized.

[acaly]

I might speak to the Roslyn team and see if there's any way to dispose resources associated with an emitted assembly before that assembly is finalized.

Roslyn has nothing to do with the execution of the assembly. Once the DLL is generated, the CLR will take over and does all the loading, compiling (to machine code), etc. Anything from the Roslyn compilation won't be needed and can be safely disposed after you get the DLL generated. This can be much earlier than the finalization of the assembly. That being said, I guess Roslyn classes hold mostly managed resources so I guess there won't be many to Dispose. They still add much pressure to the GC.

If using Roslyn cannot meet your performance requirements, another way is probably to replace Roslyn with a manual emitter with Mono.Cecil or System.Reflection.Emit. They are must faster than generating C# codes as string and then compiling with Roslyn. The DynamicMethod class can be used to emit collectable methods directly into the CLR without a AssemblyLoadContext. With System.Reflection.Emit (including the DynamicMethod class), you can directly use types in your assemblies (not as references), so you shouldn't need any file handle. I recommend you have a look at it if possible. As long as the script you are generating isn't too complicated, it should be a better choice.

[dave-yotta]

Ok thanks again - I'd have to check the dumps more closely but in that case those handles are likely indeed held by the assemblies loaded by the CLR - everything Roslyn related that can be disposed is done as early as possible.

We're using the syntax trees from Roslyn to do some inference for example about the returned type, and to do whitelisting on "allowed syntax". The scripts are not generated but rather defined by the user also. Will definitely keep this in mind though.

The performance I was worried about needing a daemon for was the process start-up time associated with trying to do one process per script - Roslyn is managing to compile and run these in ~10ms which is not bad considering. There's plenty to do on parameterization though so we don't compile as frequently as 1:1, and out-of-process is planned but needs to be thought through carefully is all :)