Memory leak when running long GPU inferences

[RandomDefaultUser]

I have encountered odd behavior when running long (over 200) GPU inferences in a row on the GPUs on hemera. After around 200-300 inferences, I get a memory overload error, which seems to originate somewhere on the python side. This looks like a memory leak.

I have tried to identify the problem, but standard profiling and investigating didn't give much insight. I suspect it could be related to our LAMMPS interface, but I have no idea. I will have to investigate further.