CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Use of Cobaya. camb, CLASS, cosmomc, compilers, etc.
Ira *Wolfson
Posts: 69
Joined: January 24 2013
Affiliation: MPA
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Ira *Wolfson » January 27 2014

Hello

I am running CosmoMC compiled with WMAP 9 year option data + PLANCK data.

I currently running on 6 processors, running a "vanilla" run.
Run time is over 4 weeks and going strong.
output file is >230 Mb.

Does this seems to make sense anyone?
or do I have a problem with some loop gone wild?

did not make any changes to CosmoMC (yet) - just "factory" version.

System is:

Code: Select all

[i]Intel core i7 (8 processors).
8 Mb of RAM
Ubunto 12.04 X64 bit
ifort 2013.1.117 + associated LAPACK.
mpiifort 4.1.2.040
Planck dependencies installed.[/i]
This is the .ini file used for the run:

Code: Select all

DEFAULT(batch1/CAMspec_defaults.ini)
DEFAULT(batch1/lowl.ini)
DEFAULT(batch1/lowLike.ini)

#planck lensing
DEFAULT(batch1/lensing.ini)

#Other Likelihoods
DEFAULT(batch1/BAO.ini)
#DEFAULT(batch1/HST.ini)
#DEFAULT(batch1/Union.ini)
#DEFAULT(batch1/SNLS.ini)
#DEFAULT(batch1/WiggleZ_MPK.ini)
#DEFAULT(batch1/MPK.ini)

#general settings
DEFAULT(batch1/common_batch1.ini)

#high for new runs
MPI_Max_R_ProposeUpdate = 30

propose_matrix= planck_covmats/base_planck_lowl_lowLike.covmat

high_accuracy_default=T
start_at_bestfit =F
feedback=1
use_fast_slow = T
num_threads=6
checkpoints=T

#sampling_method=7 is a new fast-slow scheme good for Planck
sampling_method = 1
dragging_steps  = 3
propose_scale = 2

indep_sample=1

use_clik=T

#Folder where files (chains, checkpoints, etc.) are stored
root_dir = chains/

#Root name for files produced
file_root=output/V2
action = 0

#these are just small speedups for testing
get_sigma8=F

#Uncomment this if you don't want one 0.06eV neutrino by default
#num_massive_neutrinos=3
#param[mnu] = 0 0 0 0 0
Thank you![/code]

Antony Lewis
Posts: 1941
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: CosmoMC +WMAP +PLANCK : projected convergance file sizes

Post by Antony Lewis » January 27 2014

Even worst case multi-parameter non-flat should run fine in two days; you should set a maximum wall time to kill bad jobs before they waste too much CPU. You can also inspect output chains before the job has finished running to see if things are working OK.

However you may not be using MPI, I suggest you do so - then you will multiple chains and it can automatically assess and stop when converged.

Ira *Wolfson
Posts: 69
Joined: January 24 2013
Affiliation: MPA
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Ira *Wolfson » January 27 2014

Thanks for the swift(!!!) reply

I am using mpi as stated in the README file

Code: Select all

mpirun -np 6 ./cosmomc test.ini
I can even see the feedback from these 6 processors as well as run system analysis to see they are working full capacity.

Can you please point me to where I should look into putting those constraints?

Thanks!

Jason Dossett
Posts: 97
Joined: March 19 2010
Affiliation: The University of Texas at Dallas
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Jason Dossett » January 28 2014

Hi,
What is the current convergence statistic for your run? As Antony stated the vanilla CosmoMC should not take anywhere near that long.

One potential problem I can see is that you have num_threads = 6 in your ini file. That line is actually saying: for each chain, make 6 threads. So you are actually trying to execute 36 parallel tasks on an 8 core machine. I have not tried doing this myself on a single machine (I usually run on a cluster), but I could see how it might hurt your performance.

Ira *Wolfson
Posts: 69
Joined: January 24 2013
Affiliation: MPA
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Ira *Wolfson » January 28 2014

Hi
thanks for the reply :)

Since it's a first run I have nothing to compare this to,

I am working in a kind of a void, there's no one in my vicinity to teach me this, so excuse me if I ask any novice questions.

Having said that -
Currently the output .txt file is over 230 Mb in size, if that is what you are asking.

If not, where would I go and look for the convergence statistics?

As for the 6 threads & 6 CPU's -
It's good to know!!
But still, even with 6 threads why would it take the code over a MONTH to converge?!?!

-- Ira

Antony Lewis
Posts: 1941
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: CosmoMC +WMAP +PLANCK : projected convergance file sizes

Post by Antony Lewis » January 28 2014

If you don't use MPI the program will not automatically stop on convergence, just when you total sample number limit is reached.

You can run getdist to see convergence statistics.

Jason Dossett
Posts: 97
Joined: March 19 2010
Affiliation: The University of Texas at Dallas
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Jason Dossett » January 28 2014

Hi,

Do you only have a single .txt file in your chains directory? If that is so, it seems that for some reason your job never actually started running in mpi, but rather ran as a single chain. If that is the case, as Antony said the job will never converge, but rather continue to run until it has 4000000 samples or you stop it.

Go ahead and run getdist as Antony suggests and see what kind of convergence statistic you have. After that you may want to troubleshoot why your job didn't run mpi.

Ira *Wolfson
Posts: 69
Joined: January 24 2013
Affiliation: MPA
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Ira *Wolfson » January 28 2014

Thanks a bunch!!!

I re-compiled cosmoMC with a -DMPI flag,
and now I get the desired 6 chains.

I STILL don't know how to read the convergence statistics, but I'm working on it :)

Jason Dossett
Posts: 97
Joined: March 19 2010
Affiliation: The University of Texas at Dallas
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Jason Dossett » January 28 2014

No worries. At this point, now that you have it running MPI, the convergence stats will print out in the output file periodically. You can still run getdist on that really big chain to see what kind of constraints you got. It probably sampled the parameter space well.

Akhilesh Nautiyal(akhi)
Posts: 72
Joined: June 13 2007
Affiliation: Malaviya National Institute of Technology Jaipur

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Akhilesh Nautiyal(akhi) » March 23 2014

Hi,

I am also trying to run cosmomc with WMAP(+PLANCK. I have complied it with setting the path of WMAP9 likelihood and cfitsio in Makefile. the program is running fine but the output is not showing any line indicating that it is taking the WMAP data into account (in Pre PLANCK time I used to get the line "Initialising WMAP data"). So do I need to change anything elsewhere in cosmomc .ini files or source. I am able to run it with only PLANCK.

Reagards,
akheilsh

Jason Dossett
Posts: 97
Joined: March 19 2010
Affiliation: The University of Texas at Dallas
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Jason Dossett » March 24 2014

In your ini file add:

Code: Select all

DEFAULT(batch1/WMAP.ini)
I would however suggest not using Planck and WMAP at the same time since the two datasets are, of course, sampling the same sky. There will be a significant amount of covariance between the two datasets that will not be accounted for when evaluating the likelihood.

-Jason

Akhilesh Nautiyal(akhi)
Posts: 72
Joined: June 13 2007
Affiliation: Malaviya National Institute of Technology Jaipur

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Akhilesh Nautiyal(akhi) » March 24 2014

Hi Jason,

Thanks for the reply.
I did it. But after running for 2 minutes the code is getting stopped after writing lines "Initializing WMAP likelihood code".
Can you please help me.

Regards,
akhielsh

Jason Dossett
Posts: 97
Joined: March 19 2010
Affiliation: The University of Texas at Dallas
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Jason Dossett » March 24 2014

Hi Akhilesh,

I suspect it might be that you need to put DEFAULT(batch1/WMAP.ini) somewhere after DEFAULT(batch1/CAMspec_defaults.ini), but if that is not the case. I suggest you set your feedback =3 (in your ini file), add -g -traceback to your compiling flags and see from the outputs where the program is stopping. With the outputs from that I should be able to off you more help.

Best,
Jason

Akhilesh Nautiyal(akhi)
Posts: 72
Joined: June 13 2007
Affiliation: Malaviya National Institute of Technology Jaipur

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Akhilesh Nautiyal(akhi) » March 25 2014

Hi Jason,

Thanks for the reply.
I tried to put DEFAULT(batch1/WMAP.ini) in the section "Other likelihoods" of test.ini. I also used Feedback=3 and added -g -traceback in the ifort flag. I am unable to resolve the issue. I am getting the following output.

Checking likelihood './data/clik/lowlike_v222.clik' on test data. got -1007.04 expecte
d -1007.04 (diff -1.97381e-05)
----
TT from l=0 to l= 32
EE from l=0 to l= 32
BB from l=0 to l= 32
TE from l=0 to l= 32
----
clik version 5887
gibbs d462e865-e178-449a-ac29-5c16ab9b38f5
Checking likelihood './data/clik/commander_v4.1_lm49.clik' on test data. got 3.2784 ex
pected 3.2784 (diff -2.55579e-10)
----
TT from l=0 to l= 49
reading BAO data set: DR9
reading BAO data set: DR7
reading BAO data set: 6DF
read BAO data sets
Initialising BBN Helium data...
Done. Interpolation table is 48 by 13
----
clik version 5887
gibbs d462e865-e178-449a-ac29-5c16ab9b38f5
Calling CAMB
Checking likelihood './data/clik/commander_v4.1_lm49.clik' on test data. got 3.2784 ex
pected 3.2784 (diff -2.55579e-10)
----
TT from l=0 to l= 49
reading BAO data set: DR9
reading BAO data set: DR7
reading BAO data set: 6DF
read BAO data sets
Initialising BBN Helium data...
Done. Interpolation table is 48 by 13
Checking likelihood './data/clik/lowlike_v222.clik' on test data. got -1007.04 expecte
d -1007.04 (diff -1.97381e-05)
----
TT from l=0 to l= 32
EE from l=0 to l= 32
BB from l=0 to l= 32
TE from l=0 to l= 32
Calling CAMB
Checking likelihood './data/clik/lowlike_v222.clik' on test data. got -1007.04 expecte
d -1007.04 (diff -1.97381e-05)
----
TT from l=0 to l= 32
EE from l=0 to l= 32
BB from l=0 to l= 32
TE from l=0 to l= 32
Checking likelihood './data/clik/lowlike_v222.clik' on test data. got -1007.04 expecte
d -1007.04 (diff -1.97381e-05)
----
TT from l=0 to l= 32
EE from l=0 to l= 32
BB from l=0 to l= 32
TE from l=0 to l= 32
Checking likelihood './data/clik/lowlike_v222.clik' on test data. got -1007.04 expecte
d -1007.04 (diff -1.97381e-05)
----
TT from l=0 to l= 32
EE from l=0 to l= 32
BB from l=0 to l= 32
TE from l=0 to l= 32
----
clik version 5887
gibbs d462e865-e178-449a-ac29-5c16ab9b38f5
Checking likelihood './data/clik/lowlike_v222.clik' on test data. got -1007.04 expecte
d -1007.04 (diff -1.97381e-05)
----
Likelihood uses the following Cls:
TT from l=0 to l= 32
EE from l=0 to l= 32
BB from l=0 to l= 32
TE from l=0 to l= 32
Using clik with likelihood file ./data/clik/commander_v4.1_lm49.clik
Initialising clik...
Checking likelihood './data/clik/lowlike_v222.clik' on test data. got -1007.04 expecte
d -1007.04 (diff -1.97381e-05)
----
TT from l=0 to l= 32
EE from l=0 to l= 32
BB from l=0 to l= 32
TE from l=0 to l= 32
Checking likelihood './data/clik/commander_v4.1_lm49.clik' on test data. got 3.2784 ex
pected 3.2784 (diff -2.55579e-10)
----
TT from l=0 to l= 49
reading BAO data set: DR9
reading BAO data set: DR7
reading BAO data set: 6DF
read BAO data sets
Initialising BBN Helium data...
Done. Interpolation table is 48 by 13
----
clik version 5887
gibbs d462e865-e178-449a-ac29-5c16ab9b38f5
----
clik version 5887
gibbs d462e865-e178-449a-ac29-5c16ab9b38f5
Calling CAMB
----
clik version 5887
gibbs d462e865-e178-449a-ac29-5c16ab9b38f5
----
clik version 5887
gibbs d462e865-e178-449a-ac29-5c16ab9b38f5
----
clik version 5887
gibbs d462e865-e178-449a-ac29-5c16ab9b38f5
Checking likelihood './data/clik/commander_v4.1_lm49.clik' on test data. got 3.2784 ex
pected 3.2784 (diff -2.55579e-10)
----
Likelihood uses the following Cls:
TT from l=0 to l= 49
reading BAO data set: DR9
reading BAO data set: DR7
reading BAO data set: 6DF
read BAO data sets
Doing non-linear Pk: F
Doing CMB lensing: T
Doing non-linear lensing: F
TT lmax = 2500
EE lmax = 2000
ET lmax = 2000
BB lmax = 2000
lmax_computed_cl = 2500
Computing tensors: T
lmax_tensor = 600
max_eta_k = 6625.000
transfer kmax = 0.8000000
adding parameters for: lowlike_v222.clik
adding parameters for: WMAP
WMAP data param indices: 22
adding parameters for: BICEP2
adding parameters for: DR7
adding parameters for: 6DF
adding parameters for: DR9
adding parameters for: commander_v4.1_lm49.clik
adding parameters for: CAMspec_v6.2TN_2013_02_26_dist.clik
CAMspec_v6.2TN_2013_02_26_dist.clik data param indices: 23 24
25 26 27 28 29 30
31 32 33 34 35 36
Fast divided into 1 blocks
21 parameters ( 7 slow ( 3 semi-slow), 14 fast ( 0 semi-fast))
Checking likelihood './data/clik/commander_v4.1_lm49.clik' on test data. got 3.2784 ex
pected 3.2784 (diff -2.55579e-10)
----
TT from l=0 to l= 49
reading BAO data set: DR9
reading BAO data set: DR7
Checking likelihood './data/clik/commander_v4.1_lm49.clik' on test data. got 3.2784 ex
pected 3.2784 (diff -2.55579e-10)
----
TT from l=0 to l= 49
Initialising BBN Helium data...
reading BAO data set: DR9
reading BAO data set: DR7
reading BAO data set: 6DF
Done. Interpolation table is 48 by 13
read BAO data sets
reading BAO data set: 6DF
read BAO data sets
Checking likelihood './data/clik/commander_v4.1_lm49.clik' on test data. got 3.2784 ex
pected 3.2784 (diff -2.55579e-10)
----
TT from l=0 to l= 49
reading BAO data set: DR9
Initialising BBN Helium data...
reading BAO data set: DR7
Initialising BBN Helium data...
Done. Interpolation table is 48 by 13
Done. Interpolation table is 48 by 13
reading BAO data set: 6DF
read BAO data sets
Initialising BBN Helium data...
Done. Interpolation table is 48 by 13
Checking likelihood './data/clik/commander_v4.1_lm49.clik' on test data. got 3.2784 ex
pected 3.2784 (diff -2.55579e-10)
----
TT from l=0 to l= 49
reading BAO data set: DR9
reading BAO data set: DR7
reading BAO data set: 6DF
read BAO data sets
Initialising BBN Helium data...
Done. Interpolation table is 48 by 13
Calling CAMB
Calling CAMB
Calling CAMB
Calling CAMB
Calling CAMB
CAMB done
CAMB done
CAMB done
CAMB done
lowlike_v222.clik lnlike = 1007.15351028039
Initializing WMAP likelihood, version v5
lowlike_v222.clik lnlike = 1007.15351028039
Initializing WMAP likelihood, version v5
CAMB done
CAMB done
CAMB done
CAMB done
lowlike_v222.clik lnlike = 1007.15351028039
Initializing WMAP likelihood, version v5
lowlike_v222.clik lnlike = 1007.15351028039
Initializing WMAP likelihood, version v5
lowlike_v222.clik lnlike = 1007.15351028039
Initializing WMAP likelihood, version v5
lowlike_v222.clik lnlike = 1007.15351028039
Initializing WMAP likelihood, version v5
lowlike_v222.clik lnlike = 1007.15351028039
Initializing WMAP likelihood, version v5
lowlike_v222.clik lnlike = 1007.15351028039
Initializing WMAP likelihood, version v5
MPI: MPI_COMM_WORLD rank 0 has terminated without calling MPI_Finalize()
MPI: aborting job
MPI: Received signal 9

I will be grateful if you can kindly help me.

Regards,
akhilesh

Jason Dossett
Posts: 97
Joined: March 19 2010
Affiliation: The University of Texas at Dallas
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Jason Dossett » March 26 2014

Hi Akhilesh,
Are those all of the outputs? Did you not get any error outputs? That is what would really help narrow it down. If that is all the outputs you got, I can try a test run on my end to see if I can help narrow it down.

It also may be best, when testing to not run using mpirun/mpiexec, just run from command line ./cosmomc test.ini instead and see what you get then. Lets see if the error becomes a little more obvious.

Sorry I am not of more immediate help.


-Jason

Post Reply