Page 1 of 1

covariance matrix estimation in cosmomc

Posted: April 26 2019
by Abdolali Banihashemi
Hi everybody,

I have a model with 2 extra free parameters (with respect to LCDM) and I'm using CosmoMC(November 2016) to constrain them. in file batch2/common.ini I have changed "estimate_propose_matrix = F" to "estimate_propose_matrix = T". without this change the cosmomc didn't run because of this error:
"error reading proposal covariance"

Now the program runs, but gets stuck in a point and doesn't progress.
Do you have any idea?
I have copied the terminal here for you to see the situation.

Many thanks.



cosmonima@CosmoNima:~/work/CosmoMC-Nov2016$ mpirun -np 3 ./cosmomc test.ini
Number of MPI processes: 3
file_root:test
Random seeds: 16872, 8996 rand_inst: 1
Random seeds: 16972, 8996 rand_inst: 2
Random seeds: 17072, 8996 rand_inst: 3
Using clik with likelihood file
./data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TT.clik
----
clik version 723c1a4b0580
smica
----
clik version 723c1a4b0580
smica
----
clik version 723c1a4b0580
smica
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TT.clik' on test data. got -380.979 expected -380.979 (diff -8.68408e-09)
----
TT from l=0 to l= 2508
----
clik version 723c1a4b0580
gibbs_gauss 1478fb2d-28fa-49ac-a8ae-677dbdc3600a
Checking likelihood './data/clik/low_l/commander/commander_rc2_v1.1_l2_29_B.clik' on test data. got -7.32304 expected -7.32304 (diff -2.52096e-10)
----
TT from l=0 to l= 29
minmizing fast parameters
minimizing 15 with rhobeg, rhoend = 3.000000 5.0000001E-02
minimization points: 31 chi2 converge tol: 5.0000001E-02
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TT.clik' on test data. got -380.979 expected -380.979 (diff -8.68408e-09)
----
TT from l=0 to l= 2508
----
clik version 723c1a4b0580
gibbs_gauss 1478fb2d-28fa-49ac-a8ae-677dbdc3600a
Checking likelihood './data/clik/low_l/commander/commander_rc2_v1.1_l2_29_B.clik' on test data. got -7.32304 expected -7.32304 (diff -2.52096e-10)
----
TT from l=0 to l= 29
minmizing fast parameters
minimizing 15 with rhobeg, rhoend = 3.000000 5.0000001E-02
minimization points: 31 chi2 converge tol: 5.0000001E-02
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TT.clik' on test data. got -380.979 expected -380.979 (diff -8.68408e-09)
----
TT from l=0 to l= 2508
Clik will run with the following nuisance parameters:
A_cib_217
cib_index
xi_sz_cib
A_sz
ps_A_100_100
ps_A_143_143
ps_A_143_217
ps_A_217_217
ksz_norm
gal545_A_100
gal545_A_143
gal545_A_143_217
gal545_A_217
calib_100T
calib_217T
A_planck
Using clik with likelihood file
./data/clik/low_l/commander/commander_rc2_v1.1_l2_29_B.clik
----
clik version 723c1a4b0580
gibbs_gauss 1478fb2d-28fa-49ac-a8ae-677dbdc3600a
Checking likelihood './data/clik/low_l/commander/commander_rc2_v1.1_l2_29_B.clik' on test data. got -7.32304 expected -7.32304 (diff -2.52096e-10)
----
TT from l=0 to l= 29
Clik will run with the following nuisance parameters:
A_planck
Doing non-linear Pk: F
Doing CMB lensing: T
Doing non-linear lensing: T
TT lmax = 2508
EE lmax = 2500
ET lmax = 2500
BB lmax = 2500
PP lmax = 2500
lmax_computed_cl = 2508
Computing tensors: F
max_eta_k = 14000.00
transfer kmax = 5.000000
adding parameters for: commander_rc2_v1.1_l2_29_B
adding parameters for: plik_dx11dr2_HM_v18_TT
Fast divided into 2 blocks
Block breaks at: 10
23 parameters ( 8 slow ( 0 semi-slow), 15 fast ( 0 semi-fast))
finding best fit point...
minmizing fast parameters
minimizing 15 with rhobeg, rhoend = 3.000000 5.0000001E-02
minimization points: 31 chi2 converge tol: 5.0000001E-02

New RHO = 3.8730D-01 Number of function values = 32
Least value of F = 4.138676938103846D+02 The corresponding X is:
0.000000D+00 0.000000D+00 1.363673D+00 0.000000D+00 0.000000D+00
0.000000D+00 0.000000D+00 0.000000D+00 0.000000D+00 0.000000D+00
0.000000D+00 0.000000D+00 -3.000000D+00 0.000000D+00 0.000000D+00

New RHO = 3.8730D-01 Number of function values = 39
Least value of F = 4.529922581093904D+02 The corresponding X is:
1.951918D+01 -1.550208D-01 2.740475D+00 -3.901173D+00 1.826301D-01
-1.576633D+00 2.469929D-01 -3.305144D-01 -3.485548D-01 4.670107D-01
5.107603D-01 1.995741D-02 -1.829085D-01 -3.233496D-01 4.023923D-02

New RHO = 3.8730D-01 Number of function values = 34
Least value of F = 4.865448279390076D+02 The corresponding X is:
-2.688557D+00 1.961169D+00 3.180071D+00 2.725701D+00 5.375859D-01
2.633561D+00 -1.593661D+00 1.467363D+00 1.878208D+00 9.050364D-02
1.603624D+00 1.451045D-01 2.895633D+00 -3.612449D-01 -1.186926D+00

New RHO = 5.0000D-02 Number of function values = 122
Least value of F = 4.000551269404194D+02 The corresponding X is:
-1.177104D+01 -4.377954D+00 1.968141D+00 6.946420D-01 3.055868D+00
4.768319D+00 8.395688D-01 -2.090754D+00 3.416560D+00 -1.245799D+00
1.287520D+00 -5.503225D+00 4.539044D+00 -1.109706D+00 -4.812435D-01

New RHO = 5.0000D-02 Number of function values = 166
Least value of F = 3.939818972190382D+02 The corresponding X is:
-1.640617D+00 4.929464D+00 2.125707D+00 -4.044483D+00 2.826738D+00
2.148035D+00 -2.271755D+00 -5.395532D+00 -2.424052D-02 1.673895D+00
3.335108D+00 -4.667239D+00 -1.143961D+01 -4.602112D-01 -3.033445D-01

New RHO = 5.0000D-02 Number of function values = 225
Least value of F = 4.026535342930455D+02 The corresponding X is:
2.465559D+01 7.263408D+00 2.627249D+00 1.898005D+00 -8.255710D-01
1.108777D+00 1.570290D+01 1.057827D-01 -3.097329D+00 -2.837745D+00
2.176128D+00 -2.551400D+00 -1.823738D-01 -1.699466D+00 -6.267989D-02

At the return from BOBYQA Number of function values = 313
Least value of F = 3.933836536540179D+02 The corresponding X is:
-1.017097D+00 4.350002D+00 1.475150D+00 -4.335326D+00 3.067170D+00
1.745837D+00 -3.010359D+00 -4.564179D+00 5.656584D-01 1.014690D+00
2.705419D+00 -6.957225D+00 -1.424431D+01 -3.869394D-01 -1.188351D+00
initial fast parameter minimize logLike: 393.383653654018
minmizing all parameters
minimizing 23 with rhobeg, rhoend = 3.000000 5.0000001E-02
minimization points: 47 chi2 converge tol: 5.0000001E-02

At the return from BOBYQA Number of function values = 324
Least value of F = 4.024648492540684D+02 The corresponding X is:
2.449731D+01 7.297969D+00 2.493445D+00 5.484219D-01 -2.217181D-01
1.928191D+00 1.521311D+01 -1.534351D-02 -3.097329D+00 -2.562672D+00
2.256828D+00 -2.645764D+00 -8.276240D-02 -1.635673D+00 -1.506279D-01
initial fast parameter minimize logLike: 402.464849254068
minmizing all parameters
minimizing 23 with rhobeg, rhoend = 3.000000 5.0000001E-02
minimization points: 47 chi2 converge tol: 5.0000001E-02

At the return from BOBYQA Number of function values = 382
Least value of F = 3.981222613624745D+02 The corresponding X is:
-1.144216D+01 4.443162D+00 2.821176D+00 -1.823598D-01 3.974933D+00
5.732526D+00 7.469486D-01 -7.574549D+00 2.926129D+00 -8.383905D-01
8.604632D-01 -6.595217D+00 9.840131D-01 -1.034050D+00 2.784879D-01
initial fast parameter minimize logLike: 398.122261362475
minmizing all parameters
minimizing 23 with rhobeg, rhoend = 3.000000 5.0000001E-02
minimization points: 47 chi2 converge tol: 5.0000001E-02

New RHO = 3.8730D-01 Number of function values = 47
Least value of F = 3.933836536540179D+02 The corresponding X is:
0.000000D+00 0.000000D+00 0.000000D+00 0.000000D+00 0.000000D+00
0.000000D+00 0.000000D+00 0.000000D+00 -1.017097D+00 4.350002D+00
1.475150D+00 -4.335326D+00 3.067170D+00 1.745837D+00 -3.010359D+00
-4.564179D+00 5.656584D-01 1.014690D+00 2.705419D+00 -6.957225D+00
-1.424431D+01 -3.869394D-01 -1.188351D+00

New RHO = 3.8730D-01 Number of function values = 48
Least value of F = 4.020712733270267D+02 The corresponding X is:
0.000000D+00 3.000000D+00 0.000000D+00 0.000000D+00 0.000000D+00
0.000000D+00 0.000000D+00 0.000000D+00 2.449731D+01 7.297969D+00
2.737033D+00 5.484219D-01 -2.217181D-01 1.928191D+00 1.521311D+01
-1.534351D-02 -3.097329D+00 -2.562672D+00 2.256828D+00 -2.645764D+00
-8.276240D-02 -1.635673D+00 -1.506279D-01

New RHO = 3.8730D-01 Number of function values = 48
Least value of F = 3.975690001515837D+02 The corresponding X is:
0.000000D+00 0.000000D+00 0.000000D+00 0.000000D+00 0.000000D+00
0.000000D+00 0.000000D+00 3.000000D+00 -1.144216D+01 4.443162D+00
2.821176D+00 -1.823598D-01 3.974933D+00 5.732526D+00 7.469486D-01
-7.574549D+00 2.274813D+00 -8.383905D-01 8.604632D-01 -6.595217D+00
9.840131D-01 -1.034050D+00 2.784879D-01

New RHO = 5.0000D-02 Number of function values = 63
Least value of F = 3.920681845725917D+02 The corresponding X is:
-1.163877D+00 1.348522D+00 -5.437211D-01 2.331210D-02 0.000000D+00
0.000000D+00 2.799118D-02 1.164581D+00 -1.047997D+00 4.377152D+00
1.473470D+00 -4.524852D+00 3.102144D+00 1.741431D+00 -3.028644D+00
-4.559689D+00 5.566631D-01 1.216073D+00 2.604987D+00 -7.014896D+00
-1.422032D+01 -4.920345D-01 -1.186074D+00

New RHO = 5.0000D-02 Number of function values = 214
Least value of F = 3.910095925461011D+02 The corresponding X is:
1.013602D+00 -2.313775D+00 1.471387D+00 -2.409091D-01 5.258125D+00
1.024957D-01 3.824885D+00 5.612198D+00 -2.312456D+00 4.343996D+00
4.115344D+00 2.717664D-01 3.281052D+00 5.389826D+00 2.597655D+00
-6.359063D+00 -1.529719D-01 -5.976913D-01 3.908494D-01 -7.102397D+00
1.009078D+00 -1.315880D+00 4.546981D-01

New RHO = 5.0000D-02 Number of function values = 217
Least value of F = 3.956446283562717D+02 The corresponding X is:
1.903077D+00 4.628325D-01 1.705740D+00 4.558708D-01 2.235393D+00
2.051640D+00 -5.476512D+00 9.753020D-02 1.496470D+01 6.283319D+00
4.692244D-01 1.336794D+00 -1.749252D-01 4.099505D-01 1.131729D+01
-1.114288D+00 -3.097329D+00 -2.324885D+00 4.341557D+00 -9.316539D-01
1.111725D+00 -1.667912D+00 -6.017690D-01

At the return from BOBYQA Number of function values = 508
Least value of F = 3.901932004328125D+02 The corresponding X is:
7.651590D-01 1.462880D-02 -6.645235D-01 -1.239892D-01 3.334644D+00
9.552898D-01 1.299478D+00 3.629873D+00 2.379220D+00 3.247530D+00
1.132509D+00 -1.806114D+00 1.902281D+00 -8.498185D-02 -2.366493D+00
-3.178584D+00 -2.598503D+00 1.217023D+00 2.670571D+00 -7.038918D+00
-1.402127D+01 -6.421359D-01 -1.355754D+00
minmizing fast parameters again
minimizing 15 with rhobeg, rhoend = 3.000000 5.0000001E-02
minimization points: 31 chi2 converge tol: 5.0000001E-02

New RHO = 3.8730D-01 Number of function values = 31
Least value of F = 3.902944508641110D+02 The corresponding X is:
2.379220D+00 3.247530D+00 1.363673D+00 -1.806114D+00 1.902281D+00
-8.498185D-02 -2.366493D+00 -3.178584D+00 -2.598503D+00 1.217023D+00
2.670571D+00 -7.038918D+00 -1.402127D+01 -6.421359D-01 -1.355754D+00
FVAL_Converge_difference reached

At the return from BOBYQA Number of function values = 34
Least value of F = 3.902944508641110D+02 The corresponding X is:
2.379220D+00 3.247530D+00 1.363673D+00 -1.806114D+00 1.902281D+00
-8.498185D-02 -2.366493D+00 -3.178584D+00 -2.598503D+00 1.217023D+00
2.670571D+00 -7.038918D+00 -1.402127D+01 -6.421359D-01 -1.355754D+00
fast parameter minimize logLike: 390.294450864111
1 Refining minimimum using low temp MCMC
1 Current logLike: 390.294450864111
1 Minimize MCMC with temp 1.000000000000000E-002
1 Stopping as have 460 samples.
slow changes 641 power changes 4
1 MCMC MaxLike = 390.093164406981
1 check likes, best_like: 390.0932 390.0932
390.2945
1 Minimize MCMC with temp 2.000000000000000E-003

At the return from BOBYQA Number of function values = 747
Least value of F = 3.897168739124219D+02 The corresponding X is:
1.880958D+00 -2.477840D+00 1.501808D+00 -4.788308D-01 7.482439D+00
-1.532468D+00 7.638666D+00 6.368844D+00 3.302043D+00 2.704837D+00
2.048657D+00 4.613926D+00 2.615664D+00 4.825560D+00 4.611893D+00
-2.525575D+00 -4.146819D+00 -8.007041D-01 5.077594D-01 -6.063808D+00
2.196559D+00 -1.012519D+00 -5.195594D-03
minmizing fast parameters again
minimizing 15 with rhobeg, rhoend = 3.000000 5.0000001E-02
minimization points: 31 chi2 converge tol: 5.0000001E-02

New RHO = 3.8730D-01 Number of function values = 32
Least value of F = 3.899395514155668D+02 The corresponding X is:
3.302043D+00 2.704837D+00 2.065231D+00 4.613926D+00 2.615664D+00
4.825560D+00 4.611893D+00 -2.525575D+00 -4.725187D+00 -8.007041D-01
5.077594D-01 -6.063808D+00 2.196559D+00 -1.012519D+00 -5.195594D-03

New RHO = 5.0000D-02 Number of function values = 66
Least value of F = 3.897051558469465D+02 The corresponding X is:
3.036284D+00 2.556234D+00 1.991760D+00 4.745701D+00 2.433960D+00
4.913839D+00 4.678128D+00 -2.340746D+00 -4.470193D+00 -8.754311D-01
3.966259D-01 -6.058620D+00 2.254240D+00 -1.015846D+00 -5.812679D-02

At the return from BOBYQA Number of function values = 138
Least value of F = 3.896453871730209D+02 The corresponding X is:
3.213615D+00 2.679435D+00 1.712307D+00 5.193108D+00 2.477656D+00
4.924330D+00 4.941402D+00 -2.083346D+00 -4.725187D+00 -7.251360D-01
4.937820D-01 -6.057646D+00 2.210488D+00 -9.574848D-01 -1.189101D-01
fast parameter minimize logLike: 389.645387173021
0 Refining minimimum using low temp MCMC
0 Current logLike: 389.645387173021
0 Minimize MCMC with temp 1.000000000000000E-002
1 Stopping as have 460 samples.
slow changes 802 power changes 4
1 MCMC MaxLike = 389.988432009728
1 check likes, best_like: 389.9884 389.9884
390.0932
1 Minimize MCMC with temp 4.000000000000000E-004
0 Stopping as have 460 samples.
slow changes 880 power changes 4
0 MCMC MaxLike = 389.643554192544
0 check likes, best_like: 389.6436 389.6436
389.6454
0 Minimize MCMC with temp 2.000000000000000E-003

At the return from BOBYQA Number of function values = 934
Least value of F = 3.899105229776663D+02 The corresponding X is:
1.176237D-01 1.825092D+00 -7.821685D-01 1.664022D+00 1.508644D+00
4.846408D+00 -6.964342D+00 -4.088229D-02 1.601981D+00 8.326364D+00
8.306360D-01 1.471246D+00 -5.100895D-01 2.233164D-01 1.228702D+01
-2.561926D+00 -3.097329D+00 -2.214800D+00 2.586862D+00 -3.371221D+00
-2.203389D+00 -1.875702D+00 -3.452060D-01
minmizing fast parameters again
minimizing 15 with rhobeg, rhoend = 3.000000 5.0000001E-02
minimization points: 31 chi2 converge tol: 5.0000001E-02

New RHO = 3.8730D-01 Number of function values = 32
Least value of F = 3.908157546168721D+02 The corresponding X is:
1.601981D+00 8.326364D+00 2.737033D+00 1.471246D+00 -5.100895D-01
2.233164D-01 1.528702D+01 -2.561926D+00 -3.097329D+00 -2.214800D+00
2.586862D+00 -3.371221D+00 -2.203389D+00 -1.875702D+00 -3.452060D-01

New RHO = 5.0000D-02 Number of function values = 47
Least value of F = 3.900604892076216D+02 The corresponding X is:
1.374528D+00 8.402004D+00 1.955619D+00 1.617140D+00 -4.916145D-01
3.147720D-01 1.394583D+01 -2.535288D+00 -3.097329D+00 -2.170063D+00
2.823505D+00 -3.233539D+00 -2.351467D+00 -1.881401D+00 -3.023726D-01

At the return from BOBYQA Number of function values = 143
Least value of F = 3.898727037531327D+02 The corresponding X is:
1.461500D+00 8.252922D+00 1.601352D+00 1.175842D+00 -6.340778D-01
9.241989D-01 1.353146D+01 -2.421463D+00 -3.097329D+00 -2.436922D+00
2.581372D+00 -3.120298D+00 -1.983514D+00 -2.083365D+00 -2.619455D-01
fast parameter minimize logLike: 389.872703753133
2 Refining minimimum using low temp MCMC
2 Current logLike: 389.872703753133
2 Minimize MCMC with temp 1.000000000000000E-002
1 Stopping as have 460 samples.
slow changes 963 power changes 4
1 MCMC MaxLike = 389.925511459958
1 check likes, best_like: 389.9255 389.9255
389.9884
1 Minimize MCMC with temp 8.000000000000001E-005
0 Stopping as have 460 samples.
slow changes 1042 power changes 4
0 MCMC MaxLike = 389.623812086310
0 check likes, best_like: 389.6238 389.6238
389.6436
2 Stopping as have 460 samples.
slow changes 1067 power changes 4
2 MCMC MaxLike = 389.872703753133
2 Minimize MCMC with temp 2.000000000000000E-003
1 Stopping as have 460 samples.
slow changes 1125 power changes 4
1 MCMC MaxLike = 389.896377121281
1 check likes, best_like: 389.8964 389.8964
389.9255

Re: covariance matrix estimation in cosmomc

Posted: April 26 2019
by Antony Lewis
The 2016 version is old, but if you want to run chains set estimate_propose_matrix = F, and check the propose_matrix is set correctly to a valid .covmat file or is an empty string.

Re: covariance matrix estimation in cosmomc

Posted: April 26 2019
by Abdolali Banihashemi
Thank you very much Sir,
The propose_matrix was set to "planck_covmats/base_TT_lowTEB_plik.covmat" by default; which is a valid file. And with this setting the "error reading proposal covariance" occurred.
Should I change it?