CosmoMC Killed for memory usage

[Thomas Clarke]

I've been running CosmoMC and find that my chains keep getting killed. When I run with MPI and 3 chains it stops after about 10,000 steps on each. I then ran without MPI and got about 30,000 chains before my run was killed (it just displays "Killed" after a chain report and stops) . When I checked back CosmoMC was using over 6GB of virtual memory when the run was killed. What could cause this to happen? I have made some modifications to CAMB but only very minor changes to CosmoMC that I'm pretty sure aren't causing this.

[Antony Lewis]

Is this using camb 1.1? Does the memory increase continuously as the program starts to run?

[Thomas Clarke]

I modified CAMB from the default version in CosmoMC which I downloaded on 15/11/2019 which I think makes it CAMB version 1.0.7.

It looks like the memory does increase continuously (it's currently using about 1GB 5000 steps into a chain).

[Thomas Clarke]

Just as an update to my reply, the memory usage definitely increases continuously (and it is the swap space that has decreased most - if this is helpful information!).

[Antony Lewis]

Version 1.0.6 or later should be OK for known memory leaks. Is it increasing only in the main MPI instance, or in all of them? Some increase is expected as the sample collection grows, but not by very much - at least for the default sampling method.

Do you get the same thing running the default code with the supplied test.ini with action=0? What compiler?

[Thomas Clarke]

I'll try and check on your answer to the question about MPI this weekend.

In response to your second question, I get the same behaviour for a completely default code running test.ini. The compiler is ifort 15.0.1.

[Antony Lewis]

I would try a more up to date ifort version. 17+ works OK I think.