CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Use of Cobaya. camb, CLASS, cosmomc, compilers, etc.
Akhilesh Nautiyal(akhi)
Posts: 72
Joined: June 13 2007
Affiliation: Malaviya National Institute of Technology Jaipur

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Akhilesh Nautiyal(akhi) » March 26 2014

Hi Jason,

thanks for the reply.
I tried to run with -g -check all -traceback, but it is not even running without WMAP with these flags. Then I tried to run with -g -traceback and with WMAP.ini it showed the allocation error of ninvplninv2. I changed the definition of ninvplninv2 in WMAP likelihood code from "real(8)" to "double precision". then there was no error but the program again stopped with the same output I posted in my previous Post. I have to run comomc with MPI in my cluster. tomorrow I will try it in my Laptop without mpi.


Regards,
akhilesh

Jason Dossett
Posts: 97
Joined: March 19 2010
Affiliation: The University of Texas at Dallas
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Jason Dossett » March 28 2014

Hi Akhilesh,

I have tested on my end and everything seems to be running fine for me. Is the WMAP likelihood code you are using one you have compiled yourself, and with the latest version of the intel compilers?

That might be causing an issue.

-Jason

Akhilesh Nautiyal(akhi)
Posts: 72
Joined: June 13 2007
Affiliation: Malaviya National Institute of Technology Jaipur

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Akhilesh Nautiyal(akhi) » March 31 2014

Hi Jason,

thank you for your kind reply. Yes. I installed wmap likelihood +cfitsio again with ifort 14 and tried to run cosmomc by using ./cosmomc test.ini.
And I got the following output.
Checking likelihood './data/clik/lowlike_v222.clik' on test data. got -1007.04 expected -1007.04 (diff -1.97381e-05)
----
Likelihood uses the following Cls:
TT from l=0 to l= 32
EE from l=0 to l= 32
BB from l=0 to l= 32
TE from l=0 to l= 32
Using clik with likelihood file ./data/clik/commander_v4.1_lm49.clik
Initialising clik...
----
clik version 5887
gibbs d462e865-e178-449a-ac29-5c16ab9b38f5
Checking likelihood './data/clik/commander_v4.1_lm49.clik' on test data. got 3.2784 expected 3.2784 (diff -2.55579e-10)
----
Likelihood uses the following Cls:
TT from l=0 to l= 49
reading BAO data set: DR9
reading BAO data set: DR7
reading BAO data set: 6DF
read BAO data sets
Doing non-linear Pk: F
Doing CMB lensing: T
Doing non-linear lensing: F
TT lmax = 2500
EE lmax = 2000
ET lmax = 2000
BB lmax = 2000
lmax_computed_cl = 2500
Computing tensors: T
lmax_tensor = 600
max_eta_k = 6625.000
transfer kmax = 0.8000000
adding parameters for: lowlike_v222.clik
adding parameters for: WMAP
WMAP data param indices: 22
adding parameters for: BICEP2
adding parameters for: DR7
adding parameters for: 6DF
adding parameters for: DR9
adding parameters for: commander_v4.1_lm49.clik
adding parameters for: CAMspec_v6.2TN_2013_02_26_dist.clik
CAMspec_v6.2TN_2013_02_26_dist.clik data param indices: 23 24
25 26 27 28 29 30
31 32 33 34 35 36
Fast divided into 1 blocks
21 parameters
starting Monte-Carlo
Initialising BBN Helium data...
Done. Interpolation table is 48 by 13
Calling CAMB
CAMB done
lowlike_v222.clik lnlike = 1007.56235773523
Initializing WMAP likelihood, version v5
forrtl: error (65): floating invalid
Image PC Routine Line Source
cosmomc 00000000006BD8D9 Unknown Unknown Unknown
cosmomc 00000000006BC1AE Unknown Unknown Unknown
cosmomc 0000000000661E12 Unknown Unknown Unknown
cosmomc 000000000060E523 Unknown Unknown Unknown
cosmomc 0000000000616AC1 Unknown Unknown Unknown
libpthread.so.0 00007FFFE664B5D0 Unknown Unknown Unknown
cosmomc 00000000006BA4CA Unknown Unknown Unknown
cosmomc 00000000005F7498 Unknown Unknown Unknown
cosmomc 00000000005E6F1A Unknown Unknown Unknown
cosmomc 0000000000520E50 cmblikelihoods_mp 154 CMB.f90
cosmomc 00000000004CE129 likelihood_cosmol 46 Likelihood_Cosmology.f90
cosmomc 0000000000538B83 calclike_mp_theor 310 calclike.f90
cosmomc 00000000005381BB calclike_mp_theor 241 calclike.f90
cosmomc 0000000000536922 calclike_mp_getlo 129 calclike.f90
cosmomc 00000000004B341E montecarlo_mp_tsa 94 MCMC.f90
cosmomc 00000000004B47FA montecarlo_mp_tme 279 MCMC.f90
cosmomc 00000000004B3892 montecarlo_mp_tch 144 MCMC.f90
cosmomc 00000000004C368B generalsetup_mp_t 137 GeneralSetup.f90
cosmomc 000000000053E591 MAIN__ 255 driver.F90
cosmomc 000000000040EDD6 Unknown Unknown Unknown
libc.so.6 00007FFFE60F8BC6 Unknown Unknown Unknown
cosmomc 000000000040ECC9 Unknown Unknown Unknown
Aborted (core dumped)


This time it is showing the above eror message. Can you please help me in figuring it out.

Regards,
akhilesh

Jason Dossett
Posts: 97
Joined: March 19 2010
Affiliation: The University of Texas at Dallas
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Jason Dossett » March 31 2014

Hi,

So, as expected, the error is happening somewhere in the wmap likelihood itself. I suggest that you recompile your wmap likelihood and add the same -g -traceback flags so we can trace the issue to inside that part of the code. Also, make sure to test your WMAP likelihood compilation by compiling WMAP with:

Code: Select all

make check
This will automatically run the test program inside the WMAP likelihood directory.


-Jason

Akhilesh Nautiyal(akhi)
Posts: 72
Joined: June 13 2007
Affiliation: Malaviya National Institute of Technology Jaipur

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Akhilesh Nautiyal(akhi) » March 31 2014

HI,

make check is running fine in WMAP code. I compiled WMAP Likelihood again wit -g -traceback and here is the output of ./cosmomc test.ini

lowlike_v222.clik lnlike = 1007.13710733334
Initializing WMAP likelihood, version v5
forrtl: error (65): floating invalid
Image PC Routine Line Source
cosmomc 00000000006BC3A9 Unknown Unknown Unknown
cosmomc 00000000006BAC7E Unknown Unknown Unknown
cosmomc 00000000006608E2 Unknown Unknown Unknown
cosmomc 000000000060CFF3 Unknown Unknown Unknown
cosmomc 0000000000615591 Unknown Unknown Unknown
libpthread.so.0 00007FFFE664B5D0 Unknown Unknown Unknown
cosmomc 00000000006B8F9A Unknown Unknown Unknown
cosmomc 00000000005F7159 wmap_teeebb_lowl_ 550 WMAP_9yr_teeebb_pixlike.F90
cosmomc 00000000005E7630 wmap_likelihood_9 406 WMAP_9yr_likelihood.F90
cosmomc 0000000000520E50 cmblikelihoods_mp 154 CMB.f90
cosmomc 00000000004CE129 likelihood_cosmol 46 Likelihood_Cosmology.f90
cosmomc 0000000000538B83 calclike_mp_theor 310 calclike.f90
cosmomc 00000000005381BB calclike_mp_theor 241 calclike.f90
cosmomc 0000000000536922 calclike_mp_getlo 129 calclike.f90
cosmomc 00000000004B341E montecarlo_mp_tsa 94 MCMC.f90
cosmomc 00000000004B47FA montecarlo_mp_tme 279 MCMC.f90
cosmomc 00000000004B3892 montecarlo_mp_tch 144 MCMC.f90
cosmomc 00000000004C368B generalsetup_mp_t 137 GeneralSetup.f90
cosmomc 000000000053E591 MAIN__ 255 driver.F90
cosmomc 000000000040EDD6 Unknown Unknown Unknown
libc.so.6 00007FFFE60F8BC6 Unknown Unknown Unknown
cosmomc 000000000040ECC9 Unknown Unknown Unknown
Aborted (core dumped)

It is showing the problem in WMAP_9yr_teeebb_pixlike.F90. Can you please suggest me some think.

Thanks,
Akhilesh

Akhilesh Nautiyal(akhi)
Posts: 72
Joined: June 13 2007
Affiliation: Malaviya National Institute of Technology Jaipur

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Akhilesh Nautiyal(akhi) » March 31 2014

Hi Jason,

I just sorted out the issue. It is happening due to the flag -fpe0. Once I turned this flag OFF in my Makefile in source directory of cosmomc, my code is running fine by running ./cosmomc test.ini.

Unfortunately my institute cluster is busy right now so I am unable to run it using MPI.

Cheers,
akhilesh

Jason Dossett
Posts: 97
Joined: March 19 2010
Affiliation: The University of Texas at Dallas
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Jason Dossett » March 31 2014

Yeah that flag will do it. I am glad you fixed it. If it is running test likelihoods then it should run fine in MPI. Let me know if you run into any more issues.

Best,
Jason

Akhilesh Nautiyal(akhi)
Posts: 72
Joined: June 13 2007
Affiliation: Malaviya National Institute of Technology Jaipur

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Akhilesh Nautiyal(akhi) » March 31 2014

Hi Jason,

Sorry to bother you again.
I tried to run it in cluster as it became available just now using MPI and it is getting stopped again. Here is the output.

reading BAO data set: DR9
reading BAO data set: DR7
reading BAO data set: 6DF
read BAO data sets
Doing non-linear Pk: F
Doing CMB lensing: T
Doing non-linear lensing: F
TT lmax = 2500
EE lmax = 2000
ET lmax = 2000
BB lmax = 2000
lmax_computed_cl = 2500
Computing tensors: T
lmax_tensor = 600
max_eta_k = 6625.000
transfer kmax = 0.8000000
adding parameters for: lowlike_v222.clik
adding parameters for: WMAP
WMAP data param indices: 22
adding parameters for: BICEP2
adding parameters for: DR7
adding parameters for: 6DF
adding parameters for: DR9
adding parameters for: commander_v4.1_lm49.clik
adding parameters for: CAMspec_v6.2TN_2013_02_26_dist.clik
CAMspec_v6.2TN_2013_02_26_dist.clik data param indices: 23 24
25 26 27 28 29 30
31 32 33 34 35 36
Fast divided into 1 blocks
21 parameters
starting Monte-Carlo
1 Likelihood: 1.0000000E+30 Current Like: 1.0000000E+30
1 Likelihood: 1.0000000E+30 Current Like: 1.0000000E+30
Initialising BBN Helium data...
Done. Interpolation table is 48 by 13
Calling CAMB
CAMB done
CAMB done
CAMB done
CAMB done
CAMB done
CAMB done
CAMB done
CAMB done
lowlike_v222.clik lnlike = 1007.48585777694
Initializing WMAP likelihood, version v5
lowlike_v222.clik lnlike = 1007.15757264436
Initializing WMAP likelihood, version v5
lowlike_v222.clik lnlike = 1007.15542863624
Initializing WMAP likelihood, version v5
lowlike_v222.clik lnlike = 1007.65767679579
Initializing WMAP likelihood, version v5
lowlike_v222.clik lnlike = 1007.03265162590
Initializing WMAP likelihood, version v5
lowlike_v222.clik lnlike = 1007.89112738690
Initializing WMAP likelihood, version v5
lowlike_v222.clik lnlike = 1007.16065568822
Initializing WMAP likelihood, version v5
lowlike_v222.clik lnlike = 1007.77834242390
Initializing WMAP likelihood, version v5
Memory allocation error for ninvplninv2
MPI: MPI_COMM_WORLD rank 0 has terminated without calling MPI_Finalize()
MPI: aborting job
MPI: Received signal 9

I think the problem is in ninvplninv2. I have changed its definition from
real(8) to double precision but still same error is coming.
Can you help me.

Regards,
akhilesh

Jason Dossett
Posts: 97
Joined: March 19 2010
Affiliation: The University of Texas at Dallas
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Jason Dossett » March 31 2014

Hi Akhilesh,

Very curious. Just to make sure you recompiled WMAP on the cluster using the latest compilers? As an aside, I think it may be best to leave ninvplninv2 as a real(8) variable (even though they are equivalent) just for consistency. try running make check on your wmap likelihood on the cluster and see if that shows an issue. I can not think of a reason why that particular array cannot be allocated except for some issue with compilation or a compiler bug.

What is your exact ifort version?

Code: Select all

ifort --version 
Sorry I am not of more help.

-Jason

Akhilesh Nautiyal(akhi)
Posts: 72
Joined: June 13 2007
Affiliation: Malaviya National Institute of Technology Jaipur

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Akhilesh Nautiyal(akhi) » March 31 2014

Hi Jason,

I have ifort 14. After running the command it shows
ifort (IFORT) 14.0.2 20140120
Copyright (C) 1985-2014 Intel Corporation.

I am able to run make check even in the cluster. But when running cosmomc again the
same error is occurring.

Regards,
akhilesh

Jason Dossett
Posts: 97
Joined: March 19 2010
Affiliation: The University of Texas at Dallas
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Jason Dossett » April 01 2014

That is the same version I am using so I dont see what could be causing the issue. Did you run make clean before running make check on the cluster. That would ensure that your WMAP likelihood is compiled with the latest version of the compilers.

I am wondering if perhaps you have not requested enough memory for your runs from your cluster. Try running without the Planck Likelihoods and see if you have the same problem.

-Jason

Akhilesh Nautiyal(akhi)
Posts: 72
Joined: June 13 2007
Affiliation: Malaviya National Institute of Technology Jaipur

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Akhilesh Nautiyal(akhi) » April 01 2014

Hi, Jason,

yes I always do make clean before make all.
Now I am trying to run cosmomc only with WMAP by commenting PLANCk files in test.ini and it is running fine (writing .txt files in chains).

akhilesh

Jason Dossett
Posts: 97
Joined: March 19 2010
Affiliation: The University of Texas at Dallas
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Jason Dossett » April 01 2014

If it is running when you are not using Planck then that is more indicative that it may simply being an issue of not requesting enough memory when you are submitting to your cluster.

-Jason

Akhilesh Nautiyal(akhi)
Posts: 72
Joined: June 13 2007
Affiliation: Malaviya National Institute of Technology Jaipur

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Akhilesh Nautiyal(akhi) » April 01 2014

HI Jason,


Sorry to bother you again and again. I am adding the line
#PBS -l pmem=10gb to by pbs_script. Again same problem is happening.
Here is the output of top command while running cosmomc with PLANCK and WMAP (for nearly 50 seconds).




PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19776 akhilesh 20 0 1036m 684m 10m R 100 5.7 0:39.30 cosmomc
19777 akhilesh 20 0 1041m 687m 10m R 100 5.8 0:39.24 cosmomc
19778 akhilesh 20 0 1037m 683m 10m R 100 5.7 0:37.71 cosmomc
19780 akhilesh 20 0 1038m 685m 10m R 100 5.7 0:38.86 cosmomc
19781 akhilesh 20 0 1037m 684m 10m R 100 5.7 0:37.79 cosmomc
19782 akhilesh 20 0 1037m 684m 10m R 100 5.7 0:37.87 cosmomc
19775 akhilesh 20 0 1038m 685m 10m R 100 5.7 0:37.59 cosmomc
19779 akhilesh 20 0 1039m 686m 10m R 100 5.7 0:38.16 cosmomc
1 root 20 0 10376 740 604 S 0 0.0 0:04.20 init
2 root 20 0 0 0 0 S 0 0.0 0:00.01 kthreadd
3 root RT 0 0 0 0 S 0 0.0 0:00.21 migration/0
4 root 20 0 0 0 0 S 0 0.0 0:00.02 ksoftirqd/0
5 root RT 0 0 0 0 S 0 0.0 0:00.42 migration/1
6 root 20 0 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
7 root RT 0 0 0 0 S 0 0.0 0:00.15 migration/2
8 root 20 0 0 0 0 S 0 0.0 0:00.04 ksoftirqd/2
9 root RT 0 0 0 0 S 0 0.0 0:00.02 migration/3
10 root 20 0 0 0 0 S 0 0.0 0:00.00 ksoftirqd/3


The memory of my cluster I found with free -m command is
as follows.

total used free shared buffers cached
Mem: 11936 670 11266 0 0 161
-/+ buffers/cache: 509 11427
Swap: 0 0 0

Now can you please suggest me some thing.

Jason Dossett
Posts: 97
Joined: March 19 2010
Affiliation: The University of Texas at Dallas
Contact:

CosmoMC +WMAP +PLANCK : projected convergance file sizes?

Post by Jason Dossett » April 01 2014

Hi Akhilesh,

I am sorry, but I am a bit out of ideas at this point. I have been able to run the same configuration on multiple clusters on my end and have not run into the issue you are having so it is hard for me to say what to do next. I would suggest you contact your sys admin to see if they can help you troubleshoot the issue.

Sorry I couldn't help you fix it.

Best,
Jason

Post Reply