Segmentation fault while using cosmomc

Use of Healpix, camb, CLASS, cosmomc, compilers, etc.
Post Reply
Yi Wang
Posts: 5
Joined: November 23 2008
Affiliation: Institute of Theoretical Physics, Chinese Acadmy of Sciences
Contact:

Segmentation fault while using cosmomc

Post by Yi Wang » November 23 2008

Hi,

I am learning to use cosmomc. I met "segmentation fault" problem when I use the WMAP data. When I edit the params.ini and do not use the WMAP data (use HST instead, for example), there is no problem. The detailed message is as follows.

Thank you very much for the help:)

(The segmentation fault happens at the same place, but the error message is random. I use Ubuntu 8.10, June 2008 version of cosmomc, and Intel ifort v11.0. I also run cosmomc on another computer with the same configuration, and the problem remains)

----------Error message that most commonly appeard------------
Random seeds: 30487, 4431 rand_inst: 0
Computing tensors: F
Doing CMB lensing: T
lmax = 2100
Number of C_ls = 3
Varying 7 parameters ( 0 fast)
reading: params_CMB.covmat
starting Monte-Carlo
reading WMAP5 data
Initializing WMAP likelihood, version v3
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
. B7FDF430 Unknown Unknown Unknown
libguide.so B73ECFC2 Unknown Unknown Unknown
libpthread.so.0 B736C50F Unknown Unknown Unknown
libc.so.6 B72457EE Unknown Unknown Unknown



----------One error message that has the most details------------

Random seeds: 13224, 26419 rand_inst: 0
Computing tensors: F
Doing CMB lensing: T
lmax = 2100
Number of C_ls = 3
Varying 7 parameters ( 0 fast)
reading: params_CMB.covmat
starting Monte-Carlo
reading WMAP5 data
Initializing WMAP likelihood, version v3
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
cosmomc 080594AB wmap_teeebb_lowl_ 452 WMAP_5yr_teeebb_pixlike.F90
cosmomc 08065DE6 wmap_likelihood_5 401 WMAP_5yr_likelihood.F90
cosmomc 080AECB3 Unknown Unknown Unknown
cosmomc 080D08AB Unknown Unknown Unknown
cosmomc 080D037A Unknown Unknown Unknown
cosmomc 080DAC40 Unknown Unknown Unknown
cosmomc 080E0C3F Unknown Unknown Unknown
cosmomc 0804D2B1 Unknown Unknown Unknown
libc.so.6 B71C1685 Unknown Unknown Unknown
cosmomc 0804D1C1 Unknown Unknown Unknown
*** glibc detected *** ./cosmomc: munmap_chunk(): invalid pointer: 0xb5b07ef0 ***
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6[0xb721a3f4]
./cosmomc[0x8284299]
./cosmomc[0x8271eee]
./cosmomc[0x825ff3f]
./cosmomc[0x825e552]
./cosmomc[0x826066c]
[0xb8026410]
/opt/intel/Compiler/11.0/069/lib/ia32/libguide.so[0xb7433fc2]
/lib/tls/i686/cmov/libpthread.so.0[0xb73b350f]
/lib/tls/i686/cmov/libc.so.6(clone+0x5e)[0xb728c7ee]
======= Memory map: ========
08048000-08360000 r-xp 00000000 08:01 245714 /home/wangyi/numerical/cosmomc/cosmomc
08360000-0836b000 rwxp 00318000 08:01 245714 /home/wangyi/numerical/cosmomc/cosmomc
0836b000-09ec4000 rwxp 0836b000 00:00 0
0a2a4000-0a2f6000 rwxp 0a2a4000 00:00 0 [heap]
a7651000-a7652000 r-xp 00000000 08:01 443902 /opt/intel/Compiler/11.0/069/lib/ia32/locale/en_US/irc_msg.cat
a7652000-a765a000 r-xp 00000000 08:01 443905 /opt/intel/Compiler/11.0/069/lib/ia32/locale/en_US/ifcore_msg.cat
a765a000-ad6f2000 rwxp a765a000 00:00 0
adff3000-b5b00000 rwxp adff3000 00:00 0
b5b00000-b5b7e000 rwxp b5b00000 00:00 0
b5b7e000-b5c00000 ---p b5b7e000 00:00 0
b5f04000-b5f05000 ---p b5f04000 00:00 0
b5f05000-b6105000 rwxp b5f05000 00:00 0
b6105000-b6106000 ---p b6105000 00:00 0
b6106000-b68a2000 rwxp b6106000 00:00 0
b68a2000-b7147000 r-xp 00000000 08:01 475492 /opt/intel/Compiler/11.0/069/mkl/lib/32/libmkl_p4m.so
b7147000-b7196000 rwxp 008a5000 08:01 475492 /opt/intel/Compiler/11.0/069/mkl/lib/32/libmkl_p4m.so
b7196000-b7198000 rwxp b7196000 00:00 0
b7198000-b719a000 r-xp 00000000 08:01 180537 /lib/tls/i686/cmov/libdl-2.8.90.so
b719a000-b719b000 r-xp 00001000 08:01 180537 /lib/tls/i686/cmov/libdl-2.8.90.so
b719b000-b719c000 rwxp 00002000 08:01 180537 /lib/tls/i686/cmov/libdl-2.8.90.so
b719c000-b71a9000 r-xp 00000000 08:01 163262 /lib/libgcc_s.so.1
b71a9000-b71aa000 r-xp 0000c000 08:01 163262 /lib/libgcc_s.so.1
b71aa000-b71ab000 rwxp 0000d000 08:01 163262 /lib/libgcc_s.so.1
b71ab000-b7303000 r-xp 00000000 08:01 180531 /lib/tls/i686/cmov/libc-2.8.90.so
b7303000-b7305000 r-xp 00158000 08:01 180531 /lib/tls/i686/cmov/libc-2.8.90.so
b7305000-b7306000 rwxp 0015a000 08:01 180531 /lib/tls/i686/cmov/libc-2.8.90.so
b7306000-b7309000 rwxp b7306000 00:00 0
b7309000-b737b000 r-xp 00000000 08:01 443877 /opt/intel/Compiler/11.0/069/lib/ia32/libiomp5.so
b737b000-b7380000 rwxp 00072000 08:01 443877 /opt/intel/Compiler/11.0/069/lib/ia32/libiomp5.so
b7380000-b7387000 rwxp b7380000 00:00 0
b7387000-b73ab000 r-xp 00000000 08:01 180539 /lib/tls/i686/cmov/libm-2.8.90.so
b73ab000-b73ac000 r-xp 00023000 08:01 180539 /lib/tls/i686/cmov/libm-2.8.90.so
b73ac000-b73ad000 rwxp 00024000 08:01 180539 /lib/tls/i686/cmov/libm-2.8.90.so
b73ad000-b73c2000 r-xp 00000000 08:01 180557 /lib/tls/i686/cmov/libpthread-2.8.90.so
b73c2000-b73c3000 r-xp 00014000 08:01 180557 /lib/tls/i686/cmov/libpthread-2.8.90.so
b73c3000-b73c4000 rwxp 00015000 08:01 180557 /lib/tls/i686/cmov/libpthread-2.8.90.so
b73c4000-b73c6000 rwxp b73c4000 00:00 0
b73d3000-b743e000 r-xp 00000000 08:01 443891 /opt/intel/Compiler/11.0/069/lib/ia32/libguide.so
b743e000-b7443000 rwxp 0006b000 08:01 443891 /opt/intel/Compiler/11.0/069/lib/ia32/libguide.so
b7443000-b7449000 rwxp b7443000 00:00 0
b7449000-b74aa000 r-xp 00000000 08:01 475484 /opt/intel/Compiler/11.0/069/mkl/lib/32/libmkl_core.so
b74aa000-b74ae000 rwxp 00060000 08:01 475484 /opt/intel/Compiler/11.0/069/mkl/lib/32/libmkl_core.so
b74ae000-b74bd000 rwxp b74ae000 00:00 0
b74bd000-b785f000 r-xp 00000000 08:01 475487 /opt/intel/Compiler/11.0/069/mkl/lib/32/libmkl_intel_thread.so
b785f000-b7906000 rwxp 003a1000 08:01 475487 /opt/intel/Compiler/11.0/069/mkl/lib/32/libmkl_intel_thread.so
b7906000-b790a000 rwxp b7906000 00:00 0
b790a000-b7a69000 r-xp 00000000 08:01 475517 /opt/intel/Compiler/11.0/069/mkl/lib/32/libmkl_intel.so
b7a69000-b7a6d000 rwxp 0015e000 08:01 475517 /opt/intel/Compiler/11.0/069/mkl/lib/32/libmkl_intel.so
b7a6d000-b7a74000 rwxp b7a6d000 00:00 0
b7a74000-b8006000 r-xp 00000000 08:01 475518 /opt/intel/Compiler/11.0/069/mkl/lib/32/libmkl_lapack.so
b8006000-b8009000 rwxp 00592000 08:01 475518 /opt/intel/Compiler/11.0/069/mkl/lib/32/libmkl_lapack.so
b8009000-b800c000 rwxp b8009000 00:00 0
b800c000-b8026000 r-xp 00000000 08:01 163219 /lib/ld-2.8.90.so
b8026000-b8027000 r-xp b8026000 00:00 0 [vdso]
b8027000-b8028000 r-xp 0001a000 08:01 163219 /lib/ld-2.8.90.so
b8028000-b8029000 rwxp 0001b000 08:01 163219 /lib/ld-2.8.90.so
bf9dc000-bfb28000 rwxp bfeb4000 00:00 0 [stack]
Aborted

Steven Pediani
Posts: 16
Joined: October 06 2008
Affiliation: University of Manchester

Segmentation fault while using cosmomc

Post by Steven Pediani » November 23 2008

I have just set up cosmomc on my home machine, also running Ubuntu 8.1, and have just encountered the same problem. If I sort it out I'll let you know what I did.

Steven Pediani
Posts: 16
Joined: October 06 2008
Affiliation: University of Manchester

Segmentation fault while using cosmomc

Post by Steven Pediani » November 23 2008

Ok, the problem occurs when likelihood code calls DGEMV for me. Sounds like a library issue. I'll play around some more and report back.

Steven Pediani
Posts: 16
Joined: October 06 2008
Affiliation: University of Manchester

Segmentation fault while using cosmomc

Post by Steven Pediani » November 23 2008

In your makefile, what have you got in the line, LAPACK? I changed some of the stuff in here and this seems to have sorted the problem.

Yi Wang
Posts: 5
Joined: November 23 2008
Affiliation: Institute of Theoretical Physics, Chinese Acadmy of Sciences
Contact:

Re: Segmentation fault while using cosmomc

Post by Yi Wang » November 24 2008

Steven Pediani wrote:In your makefile, what have you got in the line, LAPACK? I changed some of the stuff in here and this seems to have sorted the problem.
Thanks a lot for your reply. I use the LAPACK provided by ifort itself. Here are the details:

----------Here is part of the makefile for WMAP likelihood:

Code: Select all

LAPACK = -L/opt/intel/Compiler/11.0/069/mkl/lib/32  -lmkl_lapack -lmkl -lguide -lpthread
F90    = ifort
FFLAGS = -O2 $(WMAPFLAGS) -u -g -O2 -CB -traceback -warn all -warn noerrors \
-check all -check noarg_temp_created -fpe0 -zero -Vaxlib -fpp
INCS   = -I. -I$(CFITSIO)/include
LIBS   = -L. -L$(CFITSIO)/lib -lcfitsio $(LAPACK)
----------And here is part of the makefile for cosmomc

Code: Select all

F90C     = ifort
FFLAGS = -O2 -Vaxlib -ip -W0 -WB -openmp -fpp
LAPACKL = -L/opt/intel/Compiler/11.0/069/mkl/lib/32 -lmkl_lapack -lmkl -lguide -lpthread
----------In fact, for me it is not necessary to specify the LAPACK path, because I have set the paths in my .bashrc file:

Code: Select all

source /opt/intel/Compiler/11.0/069/bin/ifortvars.sh ia32
export LD_LIBRARY_PATH=/opt/intel/Compiler/11.0/069/lib/ia32:/opt/intel/Compiler/11.0/069/mkl/lib/32

Steven Pediani
Posts: 16
Joined: October 06 2008
Affiliation: University of Manchester

Segmentation fault while using cosmomc

Post by Steven Pediani » November 24 2008

In the line LAPACK, I added lmkl_core, and changed the other parts to,

Code: Select all

LAPACK = -L/opt/intel/mkl/10.1.0.015/lib/32 -lmkl_lapack -lmkl_ia32 -lmkl_core -lguide -lpthread 
Try this and see if it runs.

Yi Wang
Posts: 5
Joined: November 23 2008
Affiliation: Institute of Theoretical Physics, Chinese Acadmy of Sciences
Contact:

Segmentation fault while using cosmomc

Post by Yi Wang » November 24 2008

Thanks for your reply. However, my problem still exists:

Using your makefile directly, I can compile the program, but when I run the program (either the make check for WMAP likelihood or the cosmomc), the following error message appears and the program aborts:

MKL FATAL ERROR on load the function dll_cpu_version

When I change the option -lmkl_ia32 to -lmkl, this above problem is avoided, but the original segment fault problem in my first post still exists, even with -lmkl_core.

Thank you all the same:)

---------------Note added:

I also tried to compile LAPACK myself, instead of using the LAPACK from ifort: now I can use -lmkl_ia32. The options reads

-lmkl_ia32 -lmkl_core -lguide -lpthread

but the segment fault problem still exists.

Steven Pediani
Posts: 16
Joined: October 06 2008
Affiliation: University of Manchester

Segmentation fault while using cosmomc

Post by Steven Pediani » November 24 2008

Where you have listed your options, I'm guessing that you still have lmkl_lapack in there?

Having set this up on a few machines, the way the options are listed seems to be different for different machines, ie some need lmkl_intel_thread, and some work with lpthread. I don't quite understand why this is the case but the only other thing I can suggest is play around with the way the options are written (and make sure to always do the same to both cosmomc and wmap) and make clean, then make them. If you manage to sort this, let me know the solution for further reference.

Yi Wang
Posts: 5
Joined: November 23 2008
Affiliation: Institute of Theoretical Physics, Chinese Acadmy of Sciences
Contact:

Segmentation fault while using cosmomc

Post by Yi Wang » November 25 2008

Thanks for your reply. I have not solved the problem yet. The following make options does not work on two computers that I use:

Code: Select all

LAPACKL = -L/opt/intel/Compiler/11.0/069/mkl/lib/32 -lmkl_lapack -lmkl -lmkl_intel_thread -lmkl_core -lguide -lpthread
or

Code: Select all

LAPACKL = -L/opt/intel/Compiler/11.0/069/mkl/lib/32 -lmkl_lapack -lmkl -lmkl_intel_thread -lguide -lpthread
or

Code: Select all

LAPACKL = -L/opt/intel/Compiler/11.0/069/mkl/lib/32 -lmkl_lapack -lmkl -lmkl_core -lguide -lpthread
I will try some other possibilities. I will report the progress if I solved the problem.

(BTW: in the above post, I did not use -lmkl_lapack because I use the LAPACK compiled myself in the above example. Now I still use the LAPACK provided by intel ifort)

Post Reply