mpirun exceeds requested CPU usage

Use of Cobaya. camb, CLASS, cosmomc, compilers, etc.
Post Reply
Christian Neissner
Posts: 4
Joined: September 13 2024
Affiliation: Port de Informacio Cientifica

mpirun exceeds requested CPU usage

Post by Christian Neissner »

I'm trying to run cobaya with

Code: Select all

mpirun -n 8 cobaya-run test.yaml
which I run from inside a script that is sent to an HTCondor node. The submit file looks like this:

Code: Select all

executable = cobaya.sh
request_memory = 32000
request_cpus = 8
queue
The job runs for a while, but constantly execeeds the CPU usage. After some grace period the job is stopped because of it.

The run uses CAMB, is there something I should put as a parameter during the compilation/installation with pip?
Christian Neissner
Posts: 4
Joined: September 13 2024
Affiliation: Port de Informacio Cientifica

Re: mpirun exceeds requested CPU usage

Post by Christian Neissner »

On the mpirun level I tried the mapping parameter:

Code: Select all

mpirun --map-by slot:PE=1 -n 8 cobaya-run test.yaml
but CAMB still exceeds the limits.
Antony Lewis
Posts: 1983
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: mpirun exceeds requested CPU usage

Post by Antony Lewis »

Make sure you set OMP_NUM_THREADS for the number of threads per MPI process
Christian Neissner
Posts: 4
Joined: September 13 2024
Affiliation: Port de Informacio Cientifica

Re: mpirun exceeds requested CPU usage

Post by Christian Neissner »

Hi!

Thanks for the suggestion but it didn't work:

Code: Select all

%CPU  %MEM     TIME+ COMMAND                                                                                                                                  
 355.3   0.0   7:54.72 cobaya-run                                                                                                                               
 315.1   0.0   8:19.71 cobaya-run                                                                                                                               
 274.6   0.0   7:17.33 cobaya-run                                                                                                                               
 266.9   0.0   6:18.57 cobaya-run                                                                                                                               
 254.3   0.0   6:35.39 cobaya-run                                                                                                                               
 253.4   0.0   5:56.99 cobaya-run                                                                                                                               
 219.0   0.0   5:01.43 cobaya-run                                                                                                                               
 211.6   0.0   5:57.86 cobaya-run
Even in a single run:

Code: Select all

cobaya-run test.yaml

Exceeds the 100% CPU usage.
Antony Lewis
Posts: 1983
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: mpirun exceeds requested CPU usage

Post by Antony Lewis »

Looks like OMP_NUM_THREADS=4. May also be other threading parameters, ask the cluster admin for advice in your specific set up for OpenMP+MPI runs.
Post Reply