COSMOMC: Chains error while getdist and re-running with checkpoint = T

Use of Cobaya. camb, CLASS, cosmomc, compilers, etc.
Post Reply
Akhilesh Nautiyal(akhi)
Posts: 72
Joined: June 13 2007
Affiliation: Malaviya National Institute of Technology Jaipur

COSMOMC: Chains error while getdist and re-running with chec

Post by Akhilesh Nautiyal(akhi) » January 22 2016

HI,

I am running cosmomc in workstation using MPI with 4 processors. When my workstation is getting off due to power shut down, I am trying to re-run cosmomc as I have put

checkpoint = T

But, it is getting stopped with the following output.

Number of MPI processes: 4
file_root:test_apm
Random seeds: 18982, 29194 rand_inst: 4
Random seeds: 18883, 29194 rand_inst: 3
Random seeds: 18857, 29201 rand_inst: 2
Random seeds: 18841, 29209 rand_inst: 1
Using clik with likelihood file
./data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik
----
clik version 6dc2a8cf3965
smica
----
clik version 6dc2a8cf3965
smica
----
clik version 6dc2a8cf3965
smica
----
clik version 6dc2a8cf3965
smica
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik' on test data. got -1215.31 expected -1215.31 (diff -8.99
163e-08)
----
TT from l=0 to l= 2508
EE from l=0 to l= 2508
TE from l=0 to l= 2508
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik' on test data. got -1215.31 expected -1215.31 (diff -8.99
167e-08)
----
TT from l=0 to l= 2508
EE from l=0 to l= 2508
TE from l=0 to l= 2508
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik' on test data. got -1215.31 expected -1215.31 (diff -8.99
163e-08)
----
TT from l=0 to l= 2508
EE from l=0 to l= 2508
TE from l=0 to l= 2508
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik' on test data. got -1215.31 expected -1215.31 (diff -8.99
163e-08)
----
TT from l=0 to l= 2508
EE from l=0 to l= 2508
TE from l=0 to l= 2508
Clik will run with the following nuisance parameters:
A_cib_217
cib_index
xi_sz_cib
A_sz
ps_A_100_100
ps_A_143_143
ps_A_143_217
ps_A_217_217
ksz_norm
gal545_A_100
gal545_A_143
gal545_A_143_217
gal545_A_217
galf_EE_A_100
galf_EE_A_100_143
galf_EE_A_100_217
galf_EE_A_143
galf_EE_A_143_217
galf_EE_A_217
galf_EE_index
galf_TE_A_100
galf_TE_A_100_143
galf_TE_A_100_217
galf_TE_A_143
galf_TE_A_143_217
galf_TE_A_217
galf_TE_index
bleak_epsilon_0_0T_0E
bleak_epsilon_1_0T_0E
bleak_epsilon_2_0T_0E
bleak_epsilon_3_0T_0E
bleak_epsilon_4_0T_0E
bleak_epsilon_0_0T_1E
bleak_epsilon_1_0T_1E
bleak_epsilon_2_0T_1E
bleak_epsilon_3_0T_1E
bleak_epsilon_4_0T_1E
bleak_epsilon_0_0T_2E
bleak_epsilon_1_0T_2E
bleak_epsilon_2_0T_2E
bleak_epsilon_3_0T_2E
bleak_epsilon_4_0T_2E
bleak_epsilon_0_1T_1E
bleak_epsilon_1_1T_1E
bleak_epsilon_2_1T_1E
bleak_epsilon_3_1T_1E
bleak_epsilon_4_1T_1E
bleak_epsilon_0_1T_2E
bleak_epsilon_1_1T_2E
bleak_epsilon_2_1T_2E
bleak_epsilon_3_1T_2E
bleak_epsilon_4_1T_2E
bleak_epsilon_0_2T_2E
bleak_epsilon_1_2T_2E
bleak_epsilon_2_2T_2E
bleak_epsilon_3_2T_2E
bleak_epsilon_4_2T_2E
bleak_epsilon_0_0E_0E
bleak_epsilon_1_0E_0E
bleak_epsilon_2_0E_0E
bleak_epsilon_3_0E_0E
bleak_epsilon_4_0E_0E
bleak_epsilon_0_0E_1E
bleak_epsilon_1_0E_1E
bleak_epsilon_2_0E_1E
bleak_epsilon_3_0E_1E
bleak_epsilon_4_0E_1E
bleak_epsilon_0_0E_2E
bleak_epsilon_1_0E_2E
bleak_epsilon_2_0E_2E
bleak_epsilon_3_0E_2E
bleak_epsilon_4_0E_2E
bleak_epsilon_0_1E_1E
bleak_epsilon_1_1E_1E
bleak_epsilon_2_1E_1E
bleak_epsilon_3_1E_1E
bleak_epsilon_4_1E_1E
bleak_epsilon_0_1E_2E
bleak_epsilon_1_1E_2E
bleak_epsilon_2_1E_2E
bleak_epsilon_3_1E_2E
bleak_epsilon_4_1E_2E
bleak_epsilon_0_2E_2E
bleak_epsilon_1_2E_2E
bleak_epsilon_2_2E_2E
bleak_epsilon_3_2E_2E
bleak_epsilon_4_2E_2E
calib_100T
calib_217T
calib_100P
calib_143P
calib_217P
A_pol
A_planck
Using clik with likelihood file
./data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik
BFLike Ntemp = 2876
BFLike Nq = 1407
BFLike Nu = 1407
BFLike Nside = 16
BFLike Nwrite = 32393560
BFLike Ntemp = 2876
BFLike Nq = 1407
BFLike Nu = 1407
BFLike Nside = 16
BFLike Nwrite = 32393560
BFLike Ntemp = 2876
BFLike Nq = 1407
BFLike Nu = 1407
BFLike Nside = 16
BFLike Nwrite = 32393560
BFLike Ntemp = 2876
BFLike Nq = 1407
BFLike Nu = 1407
BFLike Nside = 16
BFLike Nwrite = 32393560
cls file appears to have 5+ columns
assuming it is a CAMB file with l, TT, EE, BB, TE
cls file appears to have 5+ columns
assuming it is a CAMB file with l, TT, EE, BB, TE
cls file appears to have 5+ columns
assuming it is a CAMB file with l, TT, EE, BB, TE
cls file appears to have 5+ columns
assuming it is a CAMB file with l, TT, EE, BB, TE
info = 0
info = 0
info = 0
info = 0
----
clik version 6dc2a8cf3965
bflike_smw
----
clik version 6dc2a8cf3965
bflike_smw
Checking likelihood './data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik' on test data. got -5247.87 expected -5247.87
(diff 3.85655e-07)
----
TT from l=0 to l= 29
EE from l=0 to l= 29
BB from l=0 to l= 29
TE from l=0 to l= 29
Clik will run with the following nuisance parameters:
A_planck
Doing non-linear Pk: F
Doing CMB lensing: T
Doing non-linear lensing: T
TT lmax = 2508
EE lmax = 2508
ET lmax = 2508
BB lmax = 2500
PP lmax = 2500
lmax_computed_cl = 2508
Computing tensors: F
max_eta_k = 14000.00
transfer kmax = 5.000000
----
clik version 6dc2a8cf3965
bflike_smw
----
clik version 6dc2a8cf3965
bflike_smw
adding parameters for: lowl_SMW_70_dx11d_2014_10_03_v5c_Ap
adding parameters for: smica_g30_ftl_full_pp
adding parameters for: BKPlanck_detset_comb_dust
adding parameters for: plik_dx11dr2_HM_v18_TTTEEE
Fast divided into 1 blocks
37 parameters (11 slow ( 0 semi-slow), 26 fast ( 0 semi-fast))
Checking likelihood './data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik' on test data. got -5247.87 expected -5247.87
(diff 3.85655e-07)
----
TT from l=0 to l= 29
EE from l=0 to l= 29
BB from l=0 to l= 29
TE from l=0 to l= 29
Checking likelihood './data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik' on test data. got -5247.87 expected -5247.87
(diff 3.85655e-07)
----
TT from l=0 to l= 29
EE from l=0 to l= 29
BB from l=0 to l= 29
TE from l=0 to l= 29
Checking likelihood './data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik' on test data. got -5247.87 expected -5247.87
(diff 3.85655e-07)
----
TT from l=0 to l= 29
EE from l=0 to l= 29
BB from l=0 to l= 29
TE from l=0 to l= 29
forrtl: severe (59): list-directed I/O syntax error, unit -5, file Internal List-Directed Read
Image PC Routine Line Source
cosmomc 00000000006734C7 Unknown Unknown Unknown
cosmomc 000000000069A02F Unknown Unknown Unknown
cosmomc 0000000000698ABE Unknown Unknown Unknown
cosmomc 0000000000492545 Unknown Unknown Unknown
cosmomc 00000000004EDF6F Unknown Unknown Unknown
cosmomc 00000000005A6CBA Unknown Unknown Unknown
cosmomc 000000000040F5DE Unknown Unknown Unknown
libc.so.6 00007FDEBDDF976D Unknown Unknown Unknown
cosmomc 000000000040F4C9 Unknown Unknown Unknown
forrtl: severe (59): list-directed I/O syntax error, unit -5, file Internal List-Directed Read
Image PC Routine Line Source
cosmomc 00000000006734C7 Unknown Unknown Unknown
cosmomc 000000000069A02F Unknown Unknown Unknown
cosmomc 0000000000698ABE Unknown Unknown Unknown
cosmomc 0000000000492545 Unknown Unknown Unknown
cosmomc 00000000004EDF6F Unknown Unknown Unknown
cosmomc 00000000005A6CBA Unknown Unknown Unknown
cosmomc 000000000040F5DE Unknown Unknown Unknown
libc.so.6 00007FE259B5676D Unknown Unknown Unknown
cosmomc 000000000040F4C9 Unknown Unknown Unknown
forrtl: severe (59): list-directed I/O syntax error, unit -5, file Internal List-Directed Read
Image PC Routine Line Source
cosmomc 00000000006734C7 Unknown Unknown Unknown
cosmomc 000000000069A02F Unknown Unknown Unknown
cosmomc 0000000000698ABE Unknown Unknown Unknown
cosmomc 0000000000492545 Unknown Unknown Unknown
cosmomc 00000000004EDF6F Unknown Unknown Unknown
cosmomc 00000000005A6CBA Unknown Unknown Unknown
cosmomc 000000000040F5DE Unknown Unknown Unknown
libc.so.6 00007FBA3DCA776D Unknown Unknown Unknown
cosmomc 000000000040F4C9 Unknown Unknown Unknown

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 3016 RUNNING AT ARPESW
= EXIT CODE: 59
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[1]+ Exit 59 nohup mpiexec -np 4 ./cosmomc test.ini > output_apm.txt


If I try to run getdist distparams.ini with those chains, I am getting the following output.
$ ./getdist distparams.ini
skipped unused params: omegak mnu nnu yhe Alens nrun r r02
reading chains/test_apm_1.txt
error reading line 12287 - skipping to next row
reading chains/test_apm_2.txt
error reading line 12164 - skipping to next row
reading chains/test_apm_3.txt
reading chains/test_apm_4.txt
error reading line 12167 - skipping to next row
Number of chains used = 4
var(mean)/mean(var), remaining chains, worst e-value: R-1 = 0.01643
RL: Thin for Markov: 64
RL: Thin for indep samples: 65
RL: Estimated burn in steps: 384 (140 rows)
mean input multiplicity = 2.73657094347355
Random seeds: 22786, 12957 rand_inst: 0
using 34161 rows, processing 85 parameters
Approx indep samples: 1438
Best fit sample -log(Like) = 6500.30200000000
mean(-Ln(like)) = 6511.79818770057
-Ln(mean like) = 6506.69936720748
Warning: sharp edge in parameter thetarseq - check limits[thetarseq] or limits7
4
doing 2D plots for most correlated variables
Producing 12 2D plots
producing 1 2D colored scatter plots

These chains are not working with GetdistGUI. As I can see from the output that there is an error reading the second last line of the chains. But when I open the chain files all lines seems fine to me.
I will be grateful if someone can help me in figuring out whtat going wrong.

Thanks
akhilesh

Antony Lewis
Posts: 1941
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: COSMOMC: Chains error while getdist and re-running with

Post by Antony Lewis » January 22 2016

Check your .inputparams have flush_write=T.

If you make and run cosmomc_debug you may get a more helpful stack trace (not too clear from this if the error is in clik or reading files).

Akhilesh Nautiyal(akhi)
Posts: 72
Joined: June 13 2007
Affiliation: Malaviya National Institute of Technology Jaipur

COSMOMC: Chains error while getdist and re-running with chec

Post by Akhilesh Nautiyal(akhi) » January 25 2016

thanks,

Yes my flush_write = T
When I tried to run with OUTPUT_DIR = Debug to enable debugging flags, I got the following output.

Number of MPI processes: 4
file_root:test_mdm
Random seeds: 4923, 23875 rand_inst: 1
Random seeds: 5017, 23874 rand_inst: 2
Random seeds: 5129, 23875 rand_inst: 3
Random seeds: 5248, 23877 rand_inst: 4
Using clik with likelihood file
./data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik
----
clik version 6dc2a8cf3965
smica
----
clik version 6dc2a8cf3965
smica
----
clik version 6dc2a8cf3965
smica
----
clik version 6dc2a8cf3965
smica
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik' on test data. got -1215.31 expected -1215.31 (diff -8.99
165e-08)
----
TT from l=0 to l= 2508
EE from l=0 to l= 2508
TE from l=0 to l= 2508
BFLike Ntemp = 2876
BFLike Nq = 1407
BFLike Nu = 1407
BFLike Nside = 16
BFLike Nwrite = 32393560
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik' on test data. got -1215.31 expected -1215.31 (diff -8.99
163e-08)
----
sukanta@ARPESW:~/cmbsofts/cosmomdm[tex] tail -n 2000 output_mdm.txt
Number of MPI processes: 4
file_root:test_mdm
Random seeds: 4923, 23875 rand_inst: 1
Random seeds: 5017, 23874 rand_inst: 2
Random seeds: 5129, 23875 rand_inst: 3
Random seeds: 5248, 23877 rand_inst: 4
Using clik with likelihood file
./data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik
----
clik version 6dc2a8cf3965
smica
----
clik version 6dc2a8cf3965
smica
----
clik version 6dc2a8cf3965
smica
----
clik version 6dc2a8cf3965
smica
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik' on test data. got -1215.31 expected -1215.31 (diff -8.99165e-08)
----
TT from l=0 to l= 2508
EE from l=0 to l= 2508
TE from l=0 to l= 2508
BFLike Ntemp = 2876
BFLike Nq = 1407
BFLike Nu = 1407
BFLike Nside = 16
BFLike Nwrite = 32393560
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik' on test data. got -1215.31 expected -1215.31 (diff -8.99163e-08)
----
TT from l=0 to l= 2508
EE from l=0 to l= 2508
TE from l=0 to l= 2508
Clik will run with the following nuisance parameters:
A_cib_217
cib_index
xi_sz_cib
A_sz
ps_A_100_100
ps_A_143_143
ps_A_143_217
ps_A_217_217
ksz_norm
gal545_A_100
gal545_A_143
gal545_A_143_217
gal545_A_217
galf_EE_A_100
galf_EE_A_100_143
galf_EE_A_100_217
galf_EE_A_143
galf_EE_A_143_217
galf_EE_A_217
galf_EE_index
galf_TE_A_100
galf_TE_A_100_143
galf_TE_A_100_217
galf_TE_A_143
galf_TE_A_143_217
galf_TE_A_217
galf_TE_index
bleak_epsilon_0_0T_0E
bleak_epsilon_1_0T_0E
bleak_epsilon_2_0T_0E
bleak_epsilon_3_0T_0E
bleak_epsilon_4_0T_0E
bleak_epsilon_0_0T_1E
bleak_epsilon_1_0T_1E
bleak_epsilon_2_0T_1E
bleak_epsilon_3_0T_1E
bleak_epsilon_4_0T_1E
bleak_epsilon_0_0T_2E
bleak_epsilon_1_0T_2E
bleak_epsilon_2_0T_2E
bleak_epsilon_3_0T_2E
bleak_epsilon_4_0T_2E
bleak_epsilon_0_1T_1E
bleak_epsilon_1_1T_1E
bleak_epsilon_2_1T_1E
bleak_epsilon_3_1T_1E
bleak_epsilon_4_1T_1E
bleak_epsilon_0_1T_2E
bleak_epsilon_1_1T_2E
bleak_epsilon_2_1T_2E
bleak_epsilon_3_1T_2E
bleak_epsilon_4_1T_2E
bleak_epsilon_0_2T_2E
bleak_epsilon_1_2T_2E
bleak_epsilon_2_2T_2E
bleak_epsilon_3_2T_2E
bleak_epsilon_4_2T_2E
bleak_epsilon_0_0E_0E
bleak_epsilon_1_0E_0E
bleak_epsilon_2_0E_0E
bleak_epsilon_3_0E_0E
bleak_epsilon_4_0E_0E
bleak_epsilon_0_0E_1E
bleak_epsilon_1_0E_1E
bleak_epsilon_2_0E_1E
bleak_epsilon_3_0E_1E
bleak_epsilon_4_0E_1E
bleak_epsilon_0_0E_2E
bleak_epsilon_1_0E_2E
bleak_epsilon_2_0E_2E
bleak_epsilon_3_0E_2E
bleak_epsilon_4_0E_2E
bleak_epsilon_0_1E_1E
bleak_epsilon_1_1E_1E
bleak_epsilon_2_1E_1E
bleak_epsilon_3_1E_1E
bleak_epsilon_4_1E_1E
bleak_epsilon_0_1E_2E
bleak_epsilon_1_1E_2E
bleak_epsilon_2_1E_2E
bleak_epsilon_3_1E_2E
bleak_epsilon_4_1E_2E
bleak_epsilon_0_2E_2E
bleak_epsilon_1_2E_2E
bleak_epsilon_2_2E_2E
bleak_epsilon_3_2E_2E
bleak_epsilon_4_2E_2E
calib_100T
calib_217T
calib_100P
calib_143P
calib_217P
A_pol
A_planck
Using clik with likelihood file
./data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik
BFLike Ntemp = 2876
BFLike Nq = 1407
BFLike Nu = 1407
BFLike Nside = 16
BFLike Nwrite = 32393560
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik' on test data. got -1215.31 expected -1215.31 (diff -8.99186e-08)
----
TT from l=0 to l= 2508
EE from l=0 to l= 2508
TE from l=0 to l= 2508
BFLike Ntemp = 2876
BFLike Nq = 1407
BFLike Nu = 1407
BFLike Nside = 16
BFLike Nwrite = 32393560
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik' on test data. got -1215.31 expected -1215.31 (diff -8.99163e-08)
----
TT from l=0 to l= 2508
EE from l=0 to l= 2508
TE from l=0 to l= 2508
BFLike Ntemp = 2876
BFLike Nq = 1407
BFLike Nu = 1407
BFLike Nside = 16
BFLike Nwrite = 32393560
cls file appears to have 5+ columns
assuming it is a CAMB file with l, TT, EE, BB, TE
cls file appears to have 5+ columns
assuming it is a CAMB file with l, TT, EE, BB, TE
cls file appears to have 5+ columns
assuming it is a CAMB file with l, TT, EE, BB, TE
cls file appears to have 5+ columns
assuming it is a CAMB file with l, TT, EE, BB, TE
info = 0
info = 0
info = 0
info = 0
----
clik version 6dc2a8cf3965
bflike_smw
Checking likelihood './data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik' on test data. got -5247.87 expected -5247.87 (diff 3.85655e-07)
----
TT from l=0 to l= 29
EE from l=0 to l= 29
BB from l=0 to l= 29
TE from l=0 to l= 29
----
clik version 6dc2a8cf3965
bflike_smw
----
clik version 6dc2a8cf3965
bflike_smw
2 Reading checkpoint from chains/test_mdm_2.chk
Checking likelihood './data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik' on test data. got -5247.87 expected -5247.87 (diff 3.85655e-07)
----
TT from l=0 to l= 29
EE from l=0 to l= 29
BB from l=0 to l= 29
TE from l=0 to l= 29
----
clik version 6dc2a8cf3965
bflike_smw
3 Reading checkpoint from chains/test_mdm_3.chk
forrtl: severe (193): Run-Time Check Failure. The variable 'recombination_mp_ion_[/tex]CHI_H' is being used without being defined
Image PC Routine Line Source
cosmomc 0000000000BB0B3F recombination_mp_ 1022 recfast.f90
cosmomc 00000000009CB303 dverk_ 946 subroutines.f90
cosmomc 0000000000BAB547 recombination_mp_ 688 recfast.f90
cosmomc 0000000000A3A6BF thermodata_mp_ini 2544 modules.f90
cosmomc 0000000000B198F9 cambmain_mp_initv 769 cmbmain.f90
cosmomc 0000000000B04D11 cambmain_mp_cmbma 157 cmbmain.f90
cosmomc 0000000000B924CD camb_mp_camb_getr 127 camb.f90
cosmomc 0000000000B91E14 camb_mp_camb_gett 41 camb.f90
cosmomc 0000000000734FD2 calculator_camb_m 215 Calculator_CAMB.f90
cosmomc 000000000072B918 calclike_cosmolog 77 CalcLike_Cosmology.f90
cosmomc 000000000095A117 calclike_mp_theor 307 calclike.f90
cosmomc 0000000000951C96 calclike_mp_getlo 145 calclike.f90
cosmomc 000000000069877D montecarlo_mp_tsa 94 MCMC.f90
cosmomc 00000000006A1429 montecarlo_mp_tfa 369 MCMC.f90
cosmomc 000000000069992E montecarlo_mp_tch 144 MCMC.f90
cosmomc 00000000006F5C23 generalsetup_mp_t 137 GeneralSetup.f90
cosmomc 0000000000971D0A MAIN__ 268 driver.F90
cosmomc 000000000040F56E Unknown Unknown Unknown
libc.so.6 00007FCB20DB076D Unknown Unknown Unknown
cosmomc 000000000040F459 Unknown Unknown Unknown

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 733 RUNNING AT ARPESW
= EXIT CODE: 193
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================

Without the debugging flags I am getting the following output.

Number of MPI processes: 4
file_root:test_edm
Random seeds: 21597, 7173 rand_inst: 1
Random seeds: 21877, 7181 rand_inst: 3
Random seeds: 21980, 7181 rand_inst: 4
Random seeds: 22137, 7217 rand_inst: 2
Using clik with likelihood file
./data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik
----
clik version 6dc2a8cf3965
smica
----
clik version 6dc2a8cf3965
smica
----
clik version 6dc2a8cf3965
smica
----
clik version 6dc2a8cf3965
smica
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik' on test data. got -1215.31 expected -1215.31 (diff -8.9917e-08)
----
TT from l=0 to l= 2508
EE from l=0 to l= 2508
TE from l=0 to l= 2508
BFLike Ntemp = 2876
BFLike Nq = 1407
BFLike Nu = 1407
BFLike Nside = 16
BFLike Nwrite = 32393560
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik' on test data. got -1215.31 expected -1215.31 (diff -8.99165e-08)
----
TT from l=0 to l= 2508
EE from l=0 to l= 2508
TE from l=0 to l= 2508
BFLike Ntemp = 2876
BFLike Nq = 1407
BFLike Nu = 1407
BFLike Nside = 16
BFLike Nwrite = 32393560
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik' on test data. got -1215.31 expected -1215.31 (diff -8.99186e-08)
----
TT from l=0 to l= 2508
EE from l=0 to l= 2508
TE from l=0 to l= 2508
Clik will run with the following nuisance parameters:
A_cib_217
cib_index
xi_sz_cib
A_sz
ps_A_100_100
ps_A_143_143
ps_A_143_217
ps_A_217_217
ksz_norm
gal545_A_100
gal545_A_143
gal545_A_143_217
gal545_A_217
galf_EE_A_100
galf_EE_A_100_143
galf_EE_A_100_217
galf_EE_A_143
galf_EE_A_143_217
galf_EE_A_217
galf_EE_index
galf_TE_A_100
galf_TE_A_100_143
galf_TE_A_100_217
galf_TE_A_143
galf_TE_A_143_217
galf_TE_A_217
galf_TE_index
bleak_epsilon_0_0T_0E
bleak_epsilon_1_0T_0E
bleak_epsilon_2_0T_0E
bleak_epsilon_3_0T_0E
bleak_epsilon_4_0T_0E
bleak_epsilon_0_0T_1E
bleak_epsilon_1_0T_1E
bleak_epsilon_2_0T_1E
bleak_epsilon_3_0T_1E
bleak_epsilon_4_0T_1E
bleak_epsilon_0_0T_2E
bleak_epsilon_1_0T_2E
bleak_epsilon_2_0T_2E
bleak_epsilon_3_0T_2E
bleak_epsilon_4_0T_2E
bleak_epsilon_0_1T_1E
bleak_epsilon_1_1T_1E
bleak_epsilon_2_1T_1E
bleak_epsilon_3_1T_1E
bleak_epsilon_4_1T_1E
bleak_epsilon_0_1T_2E
bleak_epsilon_1_1T_2E
bleak_epsilon_2_1T_2E
bleak_epsilon_3_1T_2E
bleak_epsilon_4_1T_2E
bleak_epsilon_0_2T_2E
bleak_epsilon_1_2T_2E
bleak_epsilon_2_2T_2E
bleak_epsilon_3_2T_2E
bleak_epsilon_4_2T_2E
bleak_epsilon_0_0E_0E
bleak_epsilon_1_0E_0E
bleak_epsilon_2_0E_0E
bleak_epsilon_3_0E_0E
bleak_epsilon_4_0E_0E
bleak_epsilon_0_0E_1E
bleak_epsilon_1_0E_1E
bleak_epsilon_2_0E_1E
bleak_epsilon_3_0E_1E
bleak_epsilon_4_0E_1E
bleak_epsilon_0_0E_2E
bleak_epsilon_1_0E_2E
bleak_epsilon_2_0E_2E
bleak_epsilon_3_0E_2E
bleak_epsilon_4_0E_2E
bleak_epsilon_0_1E_1E
bleak_epsilon_1_1E_1E
bleak_epsilon_2_1E_1E
bleak_epsilon_3_1E_1E
bleak_epsilon_4_1E_1E
bleak_epsilon_0_1E_2E
bleak_epsilon_1_1E_2E
bleak_epsilon_2_1E_2E
bleak_epsilon_3_1E_2E
bleak_epsilon_4_1E_2E
bleak_epsilon_0_2E_2E
bleak_epsilon_1_2E_2E
bleak_epsilon_2_2E_2E
bleak_epsilon_3_2E_2E
bleak_epsilon_4_2E_2E
calib_100T
calib_217T
calib_100P
calib_143P
calib_217P
A_pol
A_planck
Using clik with likelihood file
./data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik
BFLike Ntemp = 2876
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik' on test data. got -1215.31 expected -1215.31 (diff -8.99204e-08)
----
TT from l=0 to l= 2508
EE from l=0 to l= 2508
TE from l=0 to l= 2508
BFLike Ntemp = 2876
BFLike Nq = 1407
BFLike Nu = 1407
BFLike Nside = 16
BFLike Nwrite = 32393560
BFLike Nq = 1407
BFLike Nu = 1407
BFLike Nside = 16
BFLike Nwrite = 32393560
cls file appears to have 5+ columns
assuming it is a CAMB file with l, TT, EE, BB, TE
cls file appears to have 5+ columns
assuming it is a CAMB file with l, TT, EE, BB, TE
cls file appears to have 5+ columns
assuming it is a CAMB file with l, TT, EE, BB, TE
cls file appears to have 5+ columns
assuming it is a CAMB file with l, TT, EE, BB, TE
info = 0
info = 0
info = 0
info = 0
----
clik version 6dc2a8cf3965
bflike_smw
----
clik version 6dc2a8cf3965
bflike_smw
----
clik version 6dc2a8cf3965
bflike_smw
Checking likelihood './data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik' on test data. got -5247.87 expected -5247.87 (diff 3.85655e-07)
----
TT from l=0 to l= 29
EE from l=0 to l= 29
BB from l=0 to l= 29
TE from l=0 to l= 29
Checking likelihood './data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik' on test data. got -5247.87 expected -5247.87 (diff 3.85655e-07)
----
TT from l=0 to l= 29
EE from l=0 to l= 29
BB from l=0 to l= 29
TE from l=0 to l= 29
Clik will run with the following nuisance parameters:
A_planck
reading BAO data set: 6DF
reading BAO data set: MGS
reading BAO data set: DR11CMASS
Checking likelihood './data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik' on test data. got -5247.87 expected -5247.87 (diff 3.85655e-07)
----
TT from l=0 to l= 29
EE from l=0 to l= 29
BB from l=0 to l= 29
TE from l=0 to l= 29
reading BAO data set: DR11LOWZ
Doing non-linear Pk: F
Doing CMB lensing: T
Doing non-linear lensing: T
TT lmax = 2508
EE lmax = 2508
ET lmax = 2508
BB lmax = 2500
PP lmax = 2500
lmax_computed_cl = 2508
Computing tensors: F
max_eta_k = 14000.00
transfer kmax = 5.000000
adding parameters for: lowl_SMW_70_dx11d_2014_10_03_v5c_Ap
adding parameters for: smica_g30_ftl_full_pp
adding parameters for: BKPlanck_detset_comb_dust
adding parameters for: DR11CMASS
adding parameters for: DR11LOWZ
adding parameters for: MGS
adding parameters for: 6DF
adding parameters for: plik_dx11dr2_HM_v18_TTTEEE
Fast divided into 1 blocks
37 parameters (11 slow ( 0 semi-slow), 26 fast ( 0 semi-fast))
----
clik version 6dc2a8cf3965
bflike_smw
Checking likelihood './data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik' on test data. got -5247.87 expected -5247.87 (diff 3.85655e-07)
----
TT from l=0 to l= 29
EE from l=0 to l= 29
BB from l=0 to l= 29
TE from l=0 to l= 29
3 Reading checkpoint from chains/test_edm_3.chk
1 Reading checkpoint from chains/test_edm_1.chk
forrtl: severe (59): list-directed I/O syntax error, unit -5, file Internal List-Directed Read
Image PC Routine Line Source
cosmomc 00000000006734C7 Unknown Unknown Unknown
cosmomc 000000000069A02F Unknown Unknown Unknown
cosmomc 0000000000698ABE Unknown Unknown Unknown
cosmomc 0000000000492545 Unknown Unknown Unknown
cosmomc 00000000004EDF6F Unknown Unknown Unknown
cosmomc 00000000005A6CBA Unknown Unknown Unknown
cosmomc 000000000040F5DE Unknown Unknown Unknown
libc.so.6 00007F6BB4DD676D Unknown Unknown Unknown
cosmomc 000000000040F4C9 Unknown Unknown Unknown
starting Monte-Carlo

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 22372 RUNNING AT ARPESW
= EXIT CODE: 59
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================

And if I delete all files in Chains directory cosmomc is running perfectly fine until my workstation goes off due to power shut down.
The above issue is only occuring once I try to re-run it with checkpoint = T option.

Antony Lewis
Posts: 1941
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: COSMOMC: Chains error while getdist and re-running with

Post by Antony Lewis » January 25 2016

Do all of the .chk files actually exist when you do this, or just some of them?

Akhilesh Nautiyal(akhi)
Posts: 72
Joined: June 13 2007
Affiliation: Malaviya National Institute of Technology Jaipur

COSMOMC: Chains error while getdist and re-running with chec

Post by Akhilesh Nautiyal(akhi) » January 27 2016

yes all the .chk files exists. I checked the log file and it is showing

CAMB called 100 times; 0 errors
CAMB called 200 times; 0 errors
CAMB called 300 times; 0 errors
CAMB called 400 times; 0 errors
CAMB called 500 times; 0 errors
CAMB called 600 times; 0 errors
CAMB called 700 times; 0 errors
CAMB called 800 times; 0 errors
CAMB called 900 times; 0 errors
CAMB called 1000 times; 0 errors
CAMB called 1100 times; 0 errors
CAMB called 1200 times; 0 errors
CAMB called 1300 times; 0 errors
CAMB called 1400 times; 0 errors
Re-starting from checkpoint

and then getting stopped.

Antony Lewis
Posts: 1941
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: COSMOMC: Chains error while getdist and re-running with

Post by Antony Lewis » January 27 2016

Sorry, I don't know - I can't reproduce a general problem with checkpoint files in the latest version. If you can run on a cluster or the Amazon virtual machine (http://cosmologist.info/CosmoBox/) so it can run to completion.

Post Reply