CosmoMC: How long can chains converge?

[Jun-Qing Xia]

But there are still eight files even if I set num_threads = 0 or 1 or 2 or 4.
In the output there are eight lines for

Random seeds: *****, *****rand_inst: (1,...,8)

I think it means the program run eight chains.

In CAMB Makefile:
#Intel 8 , -openmp toggles mutli-processor:
#(do "module load dev8" to get v8 compilers on COSMOS)
F90C = efc
FFLAGS = -openmp -ip -O2 -Vaxlib -W0 -WB -fpp2

And in source Makefile
#Intel fortran 8
F90C = mpif90
FFLAGS = -O2 -Vaxlib -ip -W0 -WB -openmp -fpp -DMPI
LAPACKL = -L/clusterware/mathlib/mkl61/lib/64 -lmkl_lapack -lmkl_ipf -limf
INCLUDE = -I../camb

I find these are all compiled with -openmp.
I submit the job using:
bsub -n 8 (number of CPUs 2×4) -R "span[ptile=4]" (number of CPUs used on each node) ......

I don't know how the difference come from?
I think the Makefile in source maybe wrong. 8 CPUs can only run 8 chains when the Makefile is written like this. If I want 8 CPUs to run only 2 chains the Makefile should be modified. But I don't know how to modify the Makefile. Can anyone help me?

[Antony Lewis]

I'm not sure how this works on LSF. But I think it's a question of environment/bsub options, not how you compile.. e.g. see

http://www.lanl.gov/asci/bluemtn/LSF/MPI.shtml

(so maybe n=8, np=2, ptile=1 ?).

See the supplied runMPI.pl script for a sample of how to run on a cluster running PBS (set up for 2CPUs per node by default). See also

http://www.damtp.cam.ac.uk/cosmos/faq.html#cosmomc

[shadab alam]

I want to run one chain on 4 processor.
I am running script with 1 node and 4 processor.
I have tried following option and it always gives me 4 chain in place of 1 chain.

In params.ini:
num_threads=0
num_threads=4

I am calling the script with following option using a pbs script:

mpirun -machinefile machines -np $NCPU ./cosmomc params.ini > out.txt

According to the discussion one of the above setting should give me one chain?? can someone help me with this?

Thanks,
Shadab

[Sheng Li]

I want to run one chain on 4 processor.
I am running script with 1 node and 4 processor.
I have tried following option and it always gives me 4 chain in place of 1 chain.

In params.ini:
num_threads=0
num_threads=4

I am calling the script with following option using a pbs script:

mpirun -machinefile machines -np $NCPU ./cosmomc params.ini > out.txt

According to the discussion one of the above setting should give me one chain?? can someone help me with this?

Thanks,
Shadab

One chain (file) would suggest you to use this style:

Code: Select all

mpirun -np 1 program [arguments]

where the option

Code: Select all

-np 1

means the NUM_MPI_PROCESS.

In you PBS (I assume you submit your jobs on a PBS job management system) script, you need to configure like

Code: Select all

#PBS -l nodes=4:ppn=1

as you requested.

Here,

Code: Select all

nodes=4

requests 4 nodes/cpus
and

Code: Select all

ppn=1

starts ONE mpi_thread per node/cpu.

Hope this helps.

[Sheng Li]

But there are still eight files even if I set num_threads = 0 or 1 or 2 or 4.
In the output there are eight lines for

Random seeds: *****, *****rand_inst: (1,...,8)

I think it means the program run eight chains.

In CAMB Makefile:
#Intel 8 , -openmp toggles mutli-processor:
#(do "module load dev8" to get v8 compilers on COSMOS)
F90C = efc
FFLAGS = -openmp -ip -O2 -Vaxlib -W0 -WB -fpp2

And in source Makefile
#Intel fortran 8
F90C = mpif90
FFLAGS = -O2 -Vaxlib -ip -W0 -WB -openmp -fpp -DMPI
LAPACKL = -L/clusterware/mathlib/mkl61/lib/64 -lmkl_lapack -lmkl_ipf -limf
INCLUDE = -I../camb

I find these are all compiled with -openmp.
I submit the job using:
bsub -n 8 (number of CPUs 2×4) -R "span[ptile=4]" (number of CPUs used on each node) ......

I don't know how the difference come from?
I think the Makefile in source maybe wrong. 8 CPUs can only run 8 chains when the Makefile is written like this. If I want 8 CPUs to run only 2 chains the Makefile should be modified. But I don't know how to modify the Makefile. Can anyone help me?

If I was not misunderstood, I would think the number of the output files (also, the log files) is determined by the NUM_MPI_PROCESS which is the option in

Code: Select all

** -np $NUM_MPI_PROCESS **

in your command line/script when you submit your job.


And in a PBS system, M chains CAN be spread to N cpus, for example here N=8, if you set

Code: Select all

#PBS -l nodes=8:ppn=M/8

where M>=8 or M=8*K where K is integer larger or equal than 1. then your command line will be

Code: Select all

mpirun -np tot_mpi_process  ./prog [args]

to run your program with in total M chains, but the number of output file is decided from the argument *tot_mpi_process*.

So, as you proposed, you can run 2 chains (that means total number of mpi jobs/processes) but instead you need to set *tot_mpi_process=2* following the option -np, and leaving ppn=1.