Cobaya: Memory Leak when Using External Likelihood

Use of Cobaya. camb, CLASS, cosmomc, compilers, etc.
Post Reply
Umang Kumar
Posts: 6
Joined: March 15 2023
Affiliation: Ashoka University

Cobaya: Memory Leak when Using External Likelihood

Post by Umang Kumar » February 24 2024

Hi!

I am using Cobaya as an MCMC sampler for a user-defined likelihood and I am observing a memory-leak.

The likelihood is a native python code and doesn't exhibit any signs of memory leak when run without Cobaya but as soon as I use it within Cobaya I face a memory leak.

The likelihood python code has two functions, one computes the model and the other computes the log likelihood which is then fed into the Cobaya input file as mentioned in the documentation. I have also tried using del keyword and gc.collect() in the likelihood file with no improvements in the memory leak.

Cobaya version: 3.4.1
OpenMPI version: 4.1.5
Chains run: 8
System memory: 32GB

Thanks,
Umang.

Antony Lewis
Posts: 1943
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: Cobaya: Memory Leak when Using External Likelihood

Post by Antony Lewis » February 26 2024

You can try tools like tracemalloc to diagnose. Cobaya memory will gradually increase as it stores more points, but not to an extent that it is likely to be a problem.

Umang Kumar
Posts: 6
Joined: March 15 2023
Affiliation: Ashoka University

Re: Cobaya: Memory Leak when Using External Likelihood

Post by Umang Kumar » February 26 2024

I had never faced such issues with Cobaya when using Planck likelihood which prompted me to think that there was a problem in the likelihood function that I had written. The major difference being that, I was using numba jitted code and as soon as I removed the jit commands the memory leak problem was resolved.

So, I think as of now there is a compatibility issue between Cobaya and numba jitted external likelihood.

Post Reply