Cobaya and CLASS: comparing to Planck 2018 tables

Use of Cobaya. camb, CLASS, cosmomc, compilers, etc.
Post Reply
Tanvi Karwal
Posts: 11
Joined: April 10 2020
Affiliation: University of Pennsylvania

Cobaya and CLASS: comparing to Planck 2018 tables

Post by Tanvi Karwal » April 10 2020

I am running cobaya with CLASS, trying to match parameters and output of 2.20 on pg 29 of the Planck 2018 baseline parameters tables here.
I used the cobaya-cosmo-generator to build a basic input .yaml file which I modified to run on H0, point to installations, use the evaluate sampler and pass nuisance parameter values for Planck.

Here are the input parameters I am passing through cobaya under the evaluate sampler using the override option:

logA = 3.0476
n_s = 0.96822
H0 = 67.742
omega_b = 0.022436
omega_cdm = 0.11914
tau_reio = 0.0566
A_planck = 1.00068
calib_100T = 0.99973
calib_217T = 0.99817
A_cib_217 = 46.3
xi_sz_cib = 0.595
A_sz = 7.15
ksz_norm = 0.01
gal545_A_100 = 8.83
gal545_A_143 = 11.02
gal545_A_143_217 = 20.07
gal545_A_217 = 95.3
ps_A_100_100 = 248.3
ps_A_143_143 = 49.4
ps_A_143_217 = 51.4
ps_A_217_217 = 121.1
galf_TE_A_100 = 0.1139
galf_TE_A_100_143 = 0.1339
galf_TE_A_100_217 = 0.479
galf_TE_A_143 = 0.224
galf_TE_A_143_217 = 0.666
galf_TE_A_217 = 2.08


Here’s the output I get:

2020-04-10 10:44:34,176 [evaluate] log-posterior = -1939.61
2020-04-10 10:44:34,177 [evaluate] log-prior = -21.7161
2020-04-10 10:44:34,177 [evaluate] logprior_0 = -19.4873
2020-04-10 10:44:34,177 [evaluate] logprior_SZ = -2.2288
2020-04-10 10:44:34,177 [evaluate] log-likelihood = -1917.89
2020-04-10 10:44:34,177 [evaluate] chi2_planck_2018_lowl.TT = 22.835
2020-04-10 10:44:34,177 [evaluate] chi2_planck_2018_lowl.EE = 396.477
2020-04-10 10:44:34,177 [evaluate] chi2_planck_2018_highl_plik.TTTEEE = 2344.93
2020-04-10 10:44:34,177 [evaluate] chi2_planck_2018_lensing.clik = 8.81844
2020-04-10 10:44:34,177 [evaluate] chi2_bao.sixdf_2011_bao = 0.0229754
2020-04-10 10:44:34,177 [evaluate] chi2_bao.sdss_dr7_mgs = 1.29716
2020-04-10 10:44:34,177 [evaluate] chi2_bao.sdss_dr12_consensus_bao = 4.26967
2020-04-10 10:44:34,177 [evaluate] chi2_sn.pantheon = 1034.98
2020-04-10 10:44:34,177 [evaluate] chi2_my_H0 = 22.1479
2020-04-10 10:44:34,177 [evaluate] Derived params:
2020-04-10 10:44:34,177 [evaluate] A_s = 2.10647e-09
2020-04-10 10:44:34,177 [evaluate] Omega_m = 0.309917
2020-04-10 10:44:34,177 [evaluate] omegamh2 = 0.14222
2020-04-10 10:44:34,177 [evaluate] Omega_Lambda = 0.690004
2020-04-10 10:44:34,177 [evaluate] YHe = 0.245421
2020-04-10 10:44:34,177 [evaluate] z_reio = 7.88554
2020-04-10 10:44:34,177 [evaluate] sigma8 = 0.809922
2020-04-10 10:44:34,177 [evaluate] s8h5 = 0.984043
2020-04-10 10:44:34,177 [evaluate] s8omegamp5 = 0.450885
2020-04-10 10:44:34,177 [evaluate] s8omegamp25 = 0.604302
2020-04-10 10:44:34,177 [evaluate] A = 2.10647
2020-04-10 10:44:34,177 [evaluate] clamp = 1.88102
2020-04-10 10:44:34,177 [evaluate] age = 13.7846
2020-04-10 10:44:34,177 [evaluate] rs_drag = 147.25
2020-04-10 10:44:34,177 [evaluate] chi2__CMB = 2773.07
2020-04-10 10:44:34,178 [evaluate] chi2__BAO = 5.58981

I have a few questions about this and would appreciate any help:
  1. the outputs for \sigma_8 and S8 don’t match the Planck tables, but everything else does. The Planck tables use CAMB, are CLASS and CAMB is disagreement about S8 values? I am using an older version of CLASS, and that might be it. I'll run tests myself to eliminate that possibility, but asking incase someone already had the answer!
  2. I am not getting the same value for the Pantheon likelihood as in the Planck tables (10362.26). Moreover, I don’t see in cobaya output any nuisance parameter for Pantheon - is cobaya varying Pantheon nuisance params?
  3. Lastly, the BAO likelihood code under _bao_prototype.py mentions that BAO likelihoods are not yet compatible with CLASS? Has this issue been resolved? As I’m seeing nearly identical BAO chi^2’s between the Planck tables and my cobaya output.
Thank you!

Jesus Torrado
Posts: 37
Joined: April 15 2013
Affiliation: RWTH Aachen
Contact:

Re: Cobaya and CLASS: comparing to Planck 2018 tables

Post by Jesus Torrado » April 15 2020

the outputs for \sigma_8 and S8 don’t match the Planck tables, but everything else does. The Planck tables use CAMB, are CLASS and CAMB is disagreement about S8 values? I am using an older version of CLASS, and that might be it. I'll run tests myself to eliminate that possibility, but asking incase someone already had the answer!
Please, do try the newer version of CLASS (2.9.2, preferably), and the extra arguments for CLASS suggested by the cobaya-cosmo-generator:

Code: Select all

theory:
  classy:
    extra_args:
      non linear: hmcode
      hmcode_min_k_max: 20
      N_ncdm: 1
      N_ur: 2.0328
Does that improve it?
I am not getting the same value for the Pantheon likelihood as in the Planck tables (10362.26). Moreover, I don’t see in cobaya output any nuisance parameter for Pantheon - is cobaya varying Pantheon nuisance params?
Neither the Planck official chains or Cobaya run the Pantheon likelihood with any nuisance paramenter. The difference in chi2 in your case (~1.30) is a little higher than in our tests. Maybe the suggestion above improves it?
Lastly, the BAO likelihood code under _bao_prototype.py mentions that BAO likelihoods are not yet compatible with CLASS? Has this issue been resolved? As I’m seeing nearly identical BAO chi^2’s between the Planck tables and my cobaya output.
It has been indeed resolved, sorry. Now the incompatibility only applies to fsigma9, which cannot be obtained in CLASS via the Python interface.

Tanvi Karwal
Posts: 11
Joined: April 10 2020
Affiliation: University of Pennsylvania

Re: Cobaya and CLASS: comparing to Planck 2018 tables

Post by Tanvi Karwal » April 16 2020

Thank you for the response, Jesus!!

I tried with the latest version of CLASS from its github (2.9 master branch). That solves the issue of [math] and S8. To do this, I had already used the extra arguments given by cobaya-cosmo-generator, but for me, these look like:

Code: Select all

    extra_args:
      non linear: halofit
      N_ncdm: 1
      N_ur: 2.0328
      
And in fact, CLASS doesn't recognise the parameter

Code: Select all

hmcode_min_k_max: 20
Running with the updated CLASS and the same input params for cosmological and nuisance parameters from pg. 29 here, I am still getting nearly the same [math]'s:

Code: Select all

 2020-04-15 14:23:02,492 [evaluate]    chi2_planck_2018_lowl.TT = 22.835
 2020-04-15 14:23:02,492 [evaluate]    chi2_planck_2018_lowl.EE = 396.477
 2020-04-15 14:23:02,492 [evaluate]    chi2_planck_2018_highl_plik.TTTEEE = 2344.85
 2020-04-15 14:23:02,492 [evaluate]    chi2_planck_2018_lensing.clik = 8.81993
 2020-04-15 14:23:02,492 [evaluate]    chi2_bao.sixdf_2011_bao = 0.0229754
 2020-04-15 14:23:02,492 [evaluate]    chi2_bao.sdss_dr7_mgs = 1.29716
 2020-04-15 14:23:02,492 [evaluate]    chi2_bao.sdss_dr12_consensus_bao = 4.26967
 2020-04-15 14:23:02,492 [evaluate]    chi2_sn.pantheon = 1034.98
 
You'll notice that the Pantheon [math] is still off by the same amount, which I guess makes sense as the background cosmology in the older 2.7 version of CLASS was already matching the Planck tables and that's all that the Pantheon dataset should be concerned about.

Moreover, we've run the same best-fit numbers with CAMB and are getting a similar low [math] for Pantheon:

Code: Select all

 2020-04-10 13:24:33,597 [evaluate]  chi2_planck_2018_lowl.TT = 22.7943
 2020-04-10 13:24:33,597 [evaluate]  chi2_planck_2018_lowl.EE = 396.478
 2020-04-10 13:24:33,597 [evaluate]  chi2_planck_2018_highl_plik.TTTEEE = 2345.45
 2020-04-10 13:24:33,597 [evaluate]  chi2_planck_2018_lensing.clik = 8.73936
 2020-04-10 13:24:33,597 [evaluate]  chi2_bao.sixdf_2011_bao = 0.0227728
 2020-04-10 13:24:33,597 [evaluate]  chi2_bao.sdss_dr7_mgs = 1.299
 2020-04-10 13:24:33,597 [evaluate]  chi2_bao.sdss_dr12_consensus_bao = 4.26478
 2020-04-10 13:24:33,597 [evaluate]  chi2_sn.pantheon = 1034.98
 
I'll be grateful for any insight you can offer on what's going on here. Thanks!

Antony Lewis
Posts: 1941
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: Cobaya and CLASS: comparing to Planck 2018 tables

Post by Antony Lewis » April 16 2020

Also check you are comparing with the latest v2 tables as linked from the current wiki, e.g.

https://wiki.cosmos.esa.int/planck-legacy-archive/images/4/43/Baseline_params_table_2018_68pc_v2.pdf

Graeme Addison
Posts: 34
Joined: July 17 2014
Affiliation: Johns Hopkins University

Re: Cobaya and CLASS: comparing to Planck 2018 tables

Post by Graeme Addison » April 16 2020

Antony - that link doesn't work for me.

How big are the differences between the original 2018 tables and these v2 ones? I tried looking in the explanatory supplement and on the PLA but didn't find any clear documentation.

Tanvi Karwal
Posts: 11
Joined: April 10 2020
Affiliation: University of Pennsylvania

Re: Cobaya and CLASS: comparing to Planck 2018 tables

Post by Tanvi Karwal » April 16 2020

Same, that link isn't working for me either. In fact, none of the links here are working for me.

Is there a .yaml input file that was used to test CLASS and Cobaya to match output with Planck and other data sets that I could peek at?

Thank you!

Jesus Torrado
Posts: 37
Joined: April 15 2013
Affiliation: RWTH Aachen
Contact:

Re: Cobaya and CLASS: comparing to Planck 2018 tables

Post by Jesus Torrado » April 17 2020

The pdf links in the Planck wiki appear to be working again.

There are no .yaml files for comparison, because test points are generated dynamically during the tests.

You should be able to reproduce the best fit of Planck lowT+lowE+highellTTTEEE(plik)+lensing withing reasonable tolerance with the settings from the GUI and the test points for camb and class reproduced below. You should be able to get similar chi2 for any other likelihoods too with there points.

Code: Select all

camb_params = {
    'As': 2.1005829616811546e-09, 'ns': 0.96605, 'cosmomc_theta': 0.010409090000000001,
    'omegabh2': 0.022383, 'omegach2': 0.12011, 'mnu': 0.06, 'tau': 0.0543,

classy_params = {
    'As': 2.1005829616811546e-09, 'ns': 0.96605, 'H0': 67.32,
    'omegabh2': 0.022383, 'omegach2': 0.12011, 'm_ncdm': 0.06, 'tau': 0.0543}

nuisance_plank = {
 'A_planck': 1.00044, 'calib_100T': 0.99974, 'calib_217T': 0.99819, 'A_pol': 1,
 'calib_100P': 1.021, 'calib_143P': 0.966, 'calib_217P': 1.04, 'cib_index': -1.3,
 'A_cib_217': 46.1, 'xi_sz_cib': 0.66, 'A_sz': 7.08, 'ksz_norm': 0.0, 'gal545_A_100': 8.8,
 'gal545_A_143': 11.01, 'gal545_A_143_217': 20.16, 'gal545_A_217': 95.5,
 'A_sbpx_100_100_TT': 1, 'A_sbpx_143_143_TT': 1, 'A_sbpx_143_217_TT': 1,
 'A_sbpx_217_217_TT': 1, 'ps_A_100_100': 248.2, 'ps_A_143_143': 50.7,
 'ps_A_143_217': 53.3, 'ps_A_217_217': 121.9, 'galf_TE_index': -2.4,
 'galf_TE_A_100': 0.1138, 'galf_TE_A_100_143': 0.1346, 'galf_TE_A_100_217': 0.479,
 'galf_TE_A_143': 0.225, 'galf_TE_A_143_217': 0.665, 'galf_TE_A_217': 2.082,
 'galf_EE_index': -2.4, 'galf_EE_A_100': 0.055, 'galf_EE_A_100_143': 0.04,
 'galf_EE_A_100_217': 0.094, 'galf_EE_A_143': 0.086, 'galf_EE_A_143_217': 0.21,
 'galf_EE_A_217': 0.7, 'A_cnoise_e2e_100_100_EE': 1, 'A_cnoise_e2e_143_143_EE': 1,
 'A_cnoise_e2e_217_217_EE': 1, 'A_sbpx_100_100_EE': 1, 'A_sbpx_100_143_EE': 1,
 'A_sbpx_100_217_EE': 1, 'A_sbpx_143_143_EE': 1, 'A_sbpx_143_217_EE': 1,
 'A_sbpx_217_217_EE': 1}

Antony Lewis
Posts: 1941
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: Cobaya and CLASS: comparing to Planck 2018 tables

Post by Antony Lewis » April 17 2020

I think the PLA temporarily had a technical issue and links should be working again

Post Reply