CosmoMC: compiling with gfortran

Use of Healpix, camb, CLASS, cosmomc, compilers, etc.
Post Reply
André A. Costa
Posts: 5
Joined: April 08 2013
Affiliation: IF-USP

CosmoMC: compiling with gfortran

Post by André A. Costa » January 19 2016

Hello all,

I am trying to run CosmoMC July 2015 with the gfortran that comes with gcc 6. However, although it compiles without any error, when I run a simple test like "test_planck.ini", the program stops with the error (just the last lines):

transfer kmax = 5.00000000
adding parameters for: lowl_SMW_70_dx11d_2014_10_03_v5c_Ap
adding parameters for: smica_g30_ftl_full_pp
adding parameters for: BKPlanck_detset_comb_dust
adding parameters for: plik_dx11dr2_HM_v18_TT
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 128.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
parameter ranges not found: omegabh2
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 42181 on
node alphacrucis exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).

When I run the debug, it shows:

transfer kmax = 5.00000000
adding parameters for: lowl_SMW_70_dx11d_2014_10_03_v5c_Ap
adding parameters for: smica_g30_ftl_full_pp
adding parameters for: BKPlanck_detset_comb_dust
adding parameters for: plik_dx11dr2_HM_v18_TT
Fast divided into 1 blocks
23 parameters ( 9 slow ( 0 semi-slow), 14 fast ( 0 semi-fast))
Time for theory: 3.15932
Time for lowl_SMW_70_dx11d_2014_10_03_v5c_Ap: 0.38054394721984863
Time for smica_g30_ftl_full_pp: 7.8392028808593750E-004
Time for BKPlanck_detset_comb_dust: 5.2280426025390625E-003
Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL
ERROR STOP ** Likelihoods do not match **

Error termination. Backtrace:
Time for plik_dx11dr2_HM_v18_TT: 2.3849964141845703E-002
loglike chi-sq
22.258 44.516 CMB: BKPLANCK = BKPlanck_detset_comb_dust
6.079 12.157 CMB: lensing = smica_g30_ftl_full_pp
18604.526 37209.051 CMB: plik = plik_dx11dr2_HM_v18_TT
5249.288 10498.575 CMB: lowTEB = lowl_SMW_70_dx11d_2014_10_03_v5c_Ap
Test likelihoods done, total logLike, chi-eq = 23882.275 47764.549
Expected likelihoods, total logLike, chi-eq = 5859.141 11718.282
#0 0x556a55 in __generalsetup_MOD_tsetup_dotests
at /sto/home/alencar/CMB/cosmomc/source/GeneralSetup.f90:174
#1 0x6572ae in cosmomc
at /sto/home/alencar/CMB/cosmomc/source/driver.F90:279
#2 0x65739b in main
at /sto/home/alencar/CMB/cosmomc/source/driver.F90:3
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 47140 on
node alphacrucis exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).

Does anyone know how to fix this?

I configured and installed Planck 2015 likelihood with INTEL MKL (BLAS, LAPACK) and gcc 6 without any errors. I am compiling CosmoMC with

F90C = gfortran
FFLAGS = -cpp -O3 -ffast-math -ffree-line-length-none -fopenmp -fmax-errors=4 -m64 -I${MKLROOT}/include
DEBUGFLAGS = -cpp -g -fbounds-check -fbacktrace -ffree-line-length-none -fopenmp -fmax-errors=4 -ffpe-trap=invalid,overflow,zero -m64 -
I${MKLROOT}/include
MODOUT = -J$(OUTPUT_DIR)
LAPACKL = -Wl,--no-as-needed -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_gf_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blacs_intelmpi_lp64 -lpthread -lm

I also tried some variations of those flags, but the error persists.

Thanks in advance!

Antony Lewis
Posts: 1379
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: CosmoMC: compiling with gfortran

Post by Antony Lewis » January 19 2016

Which version of gcc is that exactly? (and where did you get it from?)

André A. Costa
Posts: 5
Joined: April 08 2013
Affiliation: IF-USP

CosmoMC: compiling with gfortran

Post by André A. Costa » January 20 2016

Hi, Antony. Thank you for your reply!

The version I'm using is:

[alencar@alphacrucis ~]$ gfortran -v
Using built-in specs.
COLLECT_GCC=gfortran
COLLECT_LTO_WRAPPER=/sto/home/alencar/gcc-6-20151227/libexec/gcc/x86_64-pc-linux-gnu/6.0.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /sto/home/alencar/gcc-6-20151227/configure --prefix=/sto/home/alencar/gcc-6-20151227 --enable-languages=c,c++,fortran
Thread model: posix
gcc version 6.0.0 20151227 (experimental) (GCC)

And I dowloaded it from the link GCC FTP site in https://gcc.gnu.org/wiki/GFortranSource.

Antony Lewis
Posts: 1379
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: CosmoMC: compiling with gfortran

Post by Antony Lewis » January 20 2016

Thanks, looks like that's a new build. Maybe some more bugs have accumulated in gcc 6 since the good branch used to build CosmoBox (http://cosmologist.info/CosmoBox/, which I think should work OK).

Does test.ini work? If you can help track down exactly where in the code things are going wrong that would be very helpful (to submit new GCC bug reports if necessary).

André A. Costa
Posts: 5
Joined: April 08 2013
Affiliation: IF-USP

CosmoMC: compiling with gfortran

Post by André A. Costa » January 21 2016

No, test.ini produces the same errors. Actually, I just observe that there is another error before the program stops:

Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TT.clik' on test data. got -9143.49 expected -380.979 (diff 8762.51)

although

Checking likelihood './data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik' on test data. got -5247.87 expected -5247.87 (diff 3.49413e-07)

I tried an earlier version of gcc 6, gcc version 6.0.0 20150809. This time the program shows:

[alencar@alphacrucis cosmomc]$ mpirun -np 1 ./cosmomc test_planck.ini
Number of MPI processes: 1
file_root:test
Random seeds: 18452, 8999 rand_inst: 1
Using clik with likelihood file ./data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TT.clik
----
clik version 6dc2a8cf3965
smica
Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TT.clik' on test data. got -12191.3 expected -380.979 (diff 11810.3)
----
TT from l=0 to l= 2508
Clik will run with the following nuisance parameters:
A_cib_217
cib_index
xi_sz_cib
A_sz
ps_A_100_100
ps_A_143_143
ps_A_143_217
ps_A_217_217
ksz_norm
gal545_A_100
gal545_A_143
gal545_A_143_217
gal545_A_217
calib_100T
calib_217T
A_planck
Using clik with likelihood file ./data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik
BFLike Ntemp = 2876
BFLike Nq = 1407
BFLike Nu = 1407
BFLike Nside = 16
BFLike Nwrite = 32393560
cls file appears to have 5+ columns
assuming it is a CAMB file with l, TT, EE, BB, TE
info = 0
----
clik version 6dc2a8cf3965
bflike_smw
Checking likelihood './data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik' on test data. got -5247.87 expected -5247.87 (diff 3.49413e-07)
----
TT from l=0 to l= 29
EE from l=0 to l= 29
BB from l=0 to l= 29
TE from l=0 to l= 29
Clik will run with the following nuisance parameters:
A_planck
Doing non-linear Pk: F
Doing CMB lensing: T
Doing non-linear lensing: T
TT lmax = 2508
EE lmax = 2500
ET lmax = 2500
BB lmax = 2500
PP lmax = 2500
lmax_computed_cl = 2508
Computing tensors: F
max_eta_k = 14000.0000
transfer kmax = 5.00000000
adding parameters for: lowl_SMW_70_dx11d_2014_10_03_v5c_Ap
adding parameters for: smica_g30_ftl_full_pp
adding parameters for: BKPlanck_detset_comb_dust
adding parameters for: plik_dx11dr2_HM_v18_TT
Fast divided into 1 blocks
23 parameters ( 9 slow ( 0 semi-slow), 14 fast ( 0 semi-fast))
loglike chi-sq
0.000 0.000 CMB: BKPLANCK = BKPlanck_detset_comb_dust
0.000 0.000 CMB: lensing = smica_g30_ftl_full_pp
0.000 0.000 CMB: plik = plik_dx11dr2_HM_v18_TT
0.000 0.000 CMB: lowTEB = lowl_SMW_70_dx11d_2014_10_03_v5c_Ap
Test likelihoods done, parameter point rejected (logZero or outside prior)
Total time: 46 ( 0.01283 hours )

or using the debug option (the rest is the same):

23 parameters ( 9 slow ( 0 semi-slow), 14 fast ( 0 semi-fast))
loglike chi-sq
0.000 0.000 CMB: BKPLANCK = BKPlanck_detset_comb_dust
0.000 0.000 CMB: lensing = smica_g30_ftl_full_pp
0.000 0.000 CMB: plik = plik_dx11dr2_HM_v18_TT
0.000 0.000 CMB: lowTEB = lowl_SMW_70_dx11d_2014_10_03_v5c_Ap
Test likelihoods done, parameter point rejected (logZero or outside prior)
Total time: 47 ( 0.01301 hours )
Note: The following floating-point exceptions are signalling: IEEE_DENORMAL

Could you tell me how you got the gcc version in CosmoBOX?

Antony Lewis
Posts: 1379
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: CosmoMC: compiling with gfortran

Post by Antony Lewis » January 21 2016

The default test.ini should not be using any clik files?

You can see the build code for cosmobox from source here:

https://github.com/cmbant/docker-gcc-bu ... Dockerfile

Antony Lewis
Posts: 1379
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: CosmoMC: compiling with gfortran

Post by Antony Lewis » January 22 2016

There seem to be at least two bugs in the current gcc6 master branch, you can track status here

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=69422
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=69423

Antony Lewis
Posts: 1379
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: CosmoMC: compiling with gfortran

Post by Antony Lewis » February 22 2016

As of now these are fixed and the master gcc branch is working again.

André A. Costa
Posts: 5
Joined: April 08 2013
Affiliation: IF-USP

CosmoMC: compiling with gfortran

Post by André A. Costa » April 20 2016

Dear Prof. Lewis, Thank you for all your support.

I am trying to run CosmoMC with:

gcc version 6.0.0 20160410 (I also tried 20160221)
Intel MKL 10.3
OpenMPI
Python 2.7
Red Hat Enterprise Linux Server release 6.0 (Santiago)

Code: Select all

F90C = gfortran                                                                   
FFLAGS = -cpp -O3 -ffast-math -ffree-line-length-none -fmax-errors=4 -fopenmp -m64 -I[tex]{MKLROOT}/include
DEBUGFLAGS = -cpp -g -fbounds-check -fbacktrace -ffree-line-length-none -fopenmp -fmax-errors=4 -ffpe-trap=invalid,overflow,zero -m64 -I[/tex]{MKLROOT}/include       
LAPACKL = -Wl,--no-as-needed -L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64 -lmkl_gf_lp64 -lmkl_core -lmkl_gnu_thread -lmkl_blacs_intelmpi_lp64 -lpthread -lm
I have been able to configure and install Planck likelihood successfully. I also have been able to compile CosmoMC without error. However, when I run CosmoMC, although the program seems to be running fine, the Planck hi-l likelihood yields

Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TTTEEE.clik' on test data. got -29167.5 expected -1215.31 (diff 27952.2)

When I run test_planck.ini, although the program finishes saying ok, I get

Checking likelihood './data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TT.clik' on test data. got -12191.3 expected -380.979 (diff 11810.3)

and

Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG

When I compile with cosmomc_debug I get:

mpif90 -cpp -g -fbounds-check -fbacktrace -ffree-line-length-none -fopenmp -fmax-errors=4 -ffpe-trap=invalid,overflow,zero -m64 -I/sto/home/alencar/intel/composer_xe_2013.3.163/mkl/include -DMPI -DCLIK -I../camb/DebugMPI -I/sto/home/alencar/CMB/PLC/plc-2.0/include -JDebugMPI -IDebugMPI/ -c IniObjects.f90 -o DebugMPI/IniObjects.o
IniObjects.f90:346:0:

S = S(i+2:)

internal compiler error: in gfc_trans_block_construct, at fortran/trans-stmt.c:1788
0x753190 gfc_trans_block_construct(gfc_code*)
/sto/home/alencar/gcc-6-20160410/gcc/fortran/trans-stmt.c:1788
0x6d1b57 trans_code
/sto/home/alencar/gcc-6-20160410/gcc/fortran/trans.c:1785
0x74a423 gfc_trans_if_1
/sto/home/alencar/gcc-6-20160410/gcc/fortran/trans-stmt.c:1299
0x74a434 gfc_trans_if_1
/sto/home/alencar/gcc-6-20160410/gcc/fortran/trans-stmt.c:1303
0x751e2a gfc_trans_if(gfc_code*)
/sto/home/alencar/gcc-6-20160410/gcc/fortran/trans-stmt.c:1330
0x6d1bc7 trans_code
/sto/home/alencar/gcc-6-20160410/gcc/fortran/trans.c:1777
0x74a423 gfc_trans_if_1
/sto/home/alencar/gcc-6-20160410/gcc/fortran/trans-stmt.c:1299
0x751e2a gfc_trans_if(gfc_code*)
/sto/home/alencar/gcc-6-20160410/gcc/fortran/trans-stmt.c:1330
0x6d1bc7 trans_code
/sto/home/alencar/gcc-6-20160410/gcc/fortran/trans.c:1777
0x753f94 gfc_trans_do_while(gfc_code*)
/sto/home/alencar/gcc-6-20160410/gcc/fortran/trans-stmt.c:2294
0x6d1b77 trans_code
/sto/home/alencar/gcc-6-20160410/gcc/fortran/trans.c:1797
0x74a423 gfc_trans_if_1
/sto/home/alencar/gcc-6-20160410/gcc/fortran/trans-stmt.c:1299
0x751e2a gfc_trans_if(gfc_code*)
/sto/home/alencar/gcc-6-20160410/gcc/fortran/trans-stmt.c:1330
0x6d1bc7 trans_code
/sto/home/alencar/gcc-6-20160410/gcc/fortran/trans.c:1777
0x7005a8 gfc_generate_function_code(gfc_namespace*)
/sto/home/alencar/gcc-6-20160410/gcc/fortran/trans-decl.c:6154
0x6d61b9 gfc_generate_module_code(gfc_namespace*)
/sto/home/alencar/gcc-6-20160410/gcc/fortran/trans.c:2058
0x68c82d translate_all_program_units
/sto/home/alencar/gcc-6-20160410/gcc/fortran/parse.c:5600
0x68c82d gfc_parse_file()
/sto/home/alencar/gcc-6-20160410/gcc/fortran/parse.c:5819
0x6ce875 gfc_be_parse_file
/sto/home/alencar/gcc-6-20160410/gcc/fortran/f95-lang.c:201
Please submit a full bug report,
with preprocessed source if appropriate.
Please include the complete backtrace with any bug report.
See <http://gcc.gnu.org/bugs.html> for instructions.
make[1]: *** [DebugMPI/IniObjects.o] Error 1
make[1]: Leaving directory `/sto/home/alencar/CMB/cosmomc_jul15/source'
make: *** [cosmomc_debug] Error 2

As you have been able to run ComosMC, I am thinking if the problem is not with gcc itself, but with the interface with other library.

Have you any idea how to solve this?

Antony Lewis
Posts: 1379
Joined: September 23 2004
Affiliation: University of Sussex
Contact:

Re: CosmoMC: compiling with gfortran

Post by Antony Lewis » April 21 2016

The first one sounds as though it's nothing to do with CosmoMC - try running the Clik tests directly on the .clik file? But yuu may want to make sure Clik was compiled with the same version of gfortran as cosmomc.

I've submitted the second compiler bug at

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70748

André A. Costa
Posts: 5
Joined: April 08 2013
Affiliation: IF-USP

CosmoMC: compiling with gfortran

Post by André A. Costa » April 22 2016

Yes, you are right. The first error seems to be related with the Planck high-l likelihood itself:

[alencar@alphacrucis plc-2.0]$ clik_example_C ~/CMB/cosmomc_jul15/data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TT.clik/
----
clik version 6dc2a8cf3965
smica
Checking likelihood '/sto/home/alencar/CMB/cosmomc_jul15/data/clik/hi_l/plik/plik_dx11dr2_HM_v18_TT.clik/' on test data. got -12191.3 expected -380.979 (diff 11810.3)
----
Likelihood use Cl
TT from l=0 to l=2508 (incl.)
With 16 extra parameters
A_cib_217
cib_index
xi_sz_cib
A_sz
ps_A_100_100
ps_A_143_143
ps_A_143_217
ps_A_217_217
ksz_norm
gal545_A_100
gal545_A_143
gal545_A_143_217
gal545_A_217
calib_100T
calib_217T
A_planck
parameter vector has 2525 elements

Although the low-l seems to be ok:

[alencar@alphacrucis plc-2.0]$ clik_example_C ~/CMB/cosmomc_jul15/data/clik/low_l/commander/commander_rc2_v1.1_l2_29_B.clik/
----
clik version 6dc2a8cf3965
gibbs_gauss 1478fb2d-28fa-49ac-a8ae-677dbdc3600a
Checking likelihood '/sto/home/alencar/CMB/cosmomc_jul15/data/clik/low_l/commander/commander_rc2_v1.1_l2_29_B.clik/' on test data. got -7.32304 expected -7.32304 (diff -2.52096e-10)
----
Likelihood use Cl
TT from l=0 to l=29 (incl.)
With 1 extra parameters
A_planck
parameter vector has 31 elements
[alencar@alphacrucis plc-2.0]$ clik_example_C ~/CMB/cosmomc_jul15/data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik/
BFLike Ntemp = 2876
BFLike Nq = 1407
BFLike Nu = 1407
BFLike Nside = 16
BFLike Nwrite = 32393560
cls file appears to have 5+ columns
assuming it is a CAMB file with l, TT, EE, BB, TE
info = 0
----
clik version 6dc2a8cf3965
bflike_smw
Checking likelihood '/sto/home/alencar/CMB/cosmomc_jul15/data/clik/low_l/bflike/lowl_SMW_70_dx11d_2014_10_03_v5c_Ap.clik/' on test data. got -5247.87 expected -5247.87 (diff 3.49413e-07)
----
Likelihood use Cl
TT from l=0 to l=29 (incl.)
EE from l=0 to l=29 (incl.)
BB from l=0 to l=29 (incl.)
TE from l=0 to l=29 (incl.)
With 1 extra parameters
A_planck
parameter vector has 121 elements

Post Reply