Cobaya: Memory Leak when Using External Likelihood

[Umang Kumar]

Hi!

I am using Cobaya as an MCMC sampler for a user-defined likelihood and I am observing a memory-leak.

The likelihood is a native python code and doesn't exhibit any signs of memory leak when run without Cobaya but as soon as I use it within Cobaya I face a memory leak.

The likelihood python code has two functions, one computes the model and the other computes the log likelihood which is then fed into the Cobaya input file as mentioned in the documentation. I have also tried using del keyword and gc.collect() in the likelihood file with no improvements in the memory leak.

Cobaya version: 3.4.1
OpenMPI version: 4.1.5
Chains run: 8
System memory: 32GB

Thanks,
Umang.

[Antony Lewis]

You can try tools like tracemalloc to diagnose. Cobaya memory will gradually increase as it stores more points, but not to an extent that it is likely to be a problem.

[Umang Kumar]

I had never faced such issues with Cobaya when using Planck likelihood which prompted me to think that there was a problem in the likelihood function that I had written. The major difference being that, I was using numba jitted code and as soon as I removed the jit commands the memory leak problem was resolved.

So, I think as of now there is a compatibility issue between Cobaya and numba jitted external likelihood.