Lenspix nside=2048 run

Use of Healpix, camb, CLASS, cosmomc, compilers, etc.
Post Reply
Samuel Leach
Posts: 16
Joined: October 15 2004
Affiliation: SISSA, Trieste
Contact:

Lenspix nside=2048 run

Post by Samuel Leach » December 13 2005

Hi Antony (and other lenspix users),

As you know, I'm trying to run lenspix code at nside=2048 and here is the status. I'm going to very much appreciate your general advice, since naturally you have an unrivalled knowledge of the lenspix code.

- So far I managed to run nside=2048, lmax=2000. The resources needed were 90 CPU hours, 54GB max memory and 74GB max swap, on 64 processors (1GB per proc.). It looks like it's getting a bit heavy for 64 processors, perhaps? By the way, the spectra looked fine (modulo the usual stuff near lmax).

- Ideally, the goal is to be able to run an nside=2048, lmax=3000 run (on the same 64 processors if possible, or perhaps 80 or so).

- The problem I face is that code crashes when trying to allocate memory:

POLALM2MAP: Sending to farm
POLALM2MAP Time: 34.2203190326691
SCALALM2MAP: Sending to farm
SCALALM2MAP Time : 14.4978890419006
ALM2GRADIENTMAP: Sending to farm
ALM2GRADIENTMAP> can not allocate memory for array : map2

What's your feeling: is it out of the question running on 64 processors for this lmax=3000 run?

- A final thing: for nside=1024, lmax=2000, I required 7 CPU hours, 12GB max mem and 13BG max swap on 16 processors.

Thanks for your comments and tips.

Antony Lewis
Posts: 1522
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: Lenspix nside=2048 run

Post by Antony Lewis » December 13 2005

What interpolation factor are you using? This obviously has a big effect on the time to run and shouldn't need to be very large except for BB if the main nside is large.

I think the main thread needs more memory than the others. On the CITA cluster there are a few nodes with 2/3GB each, so arranging for the main thread to run on one of these nodes can help a lot with memory. So if you can put an extra 1GB into one of your nodes and use that for the main thread, that might help. Or if there is more than one processor per node, make sure you only use one CPU on the head node to keep max memory available. Otherwise you may need to fiddle with the code or use a bigger computer.

Samuel Leach
Posts: 16
Joined: October 15 2004
Affiliation: SISSA, Trieste
Contact:

Re: Lenspix nside=2048 run

Post by Samuel Leach » December 13 2005

Antony Lewis wrote:What interpolation factor are you using? This obviously has a big effect on the time to run and shouldn't need to be very large except for BB if the main nside is large.
Good point, I'm using interp_factor=8 (the default for nside=1024 as you know).

So I should try setting interp_factor=4 for a first try with nside=2048?

Antony Lewis
Posts: 1522
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: Lenspix nside=2048 run

Post by Antony Lewis » December 13 2005

That should certainly help (though you may still have problems with the main thread).

Samuel Leach
Posts: 16
Joined: October 15 2004
Affiliation: SISSA, Trieste
Contact:

Lenspix nside=2048 run

Post by Samuel Leach » December 17 2005

Just reporting back.

Things are much more manageable now. The nside=2048 runs across 32 nodes with

CPU time : 34950.93 sec.
Max Memory : 24551 MB
Max Swap : 26630 MB

I also fixed a mistake in my program main thread by deallocating the unlensed map (~500MB) once I'm finished with it.

Thanks!

Post Reply