Cobaya error MPI disabled

[Karim M]

Hello,

I've been trying to run a simple example from Cobaya using MPI.
However, the MPI apparently is not working because if I run the example using 4 cores as:

Code: Select all

 mpirun -n 4 cobaya-run gaussian.yaml 

The following error is displayed (4 times appears in this case):

Code: Select all

-------------------------------------------------------------
[output] *ERROR* File %s is locked by another process, you are running with MPI disabled but may have more than one process. Note that --test should not be used with MPI.
[exception handler] ---------------------------------------

Traceback (most recent call last):
  File "/home/karimpsi/.local/bin/cobaya-run", line 8, in <module>
    sys.exit(run_script())
  File "/home/karimpsi/.local/lib/python3.9/site-packages/cobaya/run.py", line 203, in run_script
    run(info, **arguments.__dict__)
  File "/home/karimpsi/.local/lib/python3.9/site-packages/cobaya/run.py", line 130, in run
    out.check_and_dump_info(info, updated_info, check_compatible=False)
  File "/home/karimpsi/.local/lib/python3.9/site-packages/cobaya/mpi.py", line 274, in wrapper
    result = method(self, *args, **kwargs)
  File "/home/karimpsi/.local/lib/python3.9/site-packages/cobaya/output.py", line 312, in check_and_dump_info
    self.check_lock()
  File "/home/karimpsi/.local/lib/python3.9/site-packages/cobaya/mpi.py", line 230, in wrapper
    return func(*args, **kwargs)
  File "/home/karimpsi/.local/lib/python3.9/site-packages/cobaya/output.py", line 498, in check_lock
    self.lock.check_error()
  File "/home/karimpsi/.local/lib/python3.9/site-packages/cobaya/output.py", line 118, in check_error
    self.lock_error()
  File "/home/karimpsi/.local/lib/python3.9/site-packages/cobaya/output.py", line 93, in lock_error
    raise LoggedError(self.log,
cobaya.log.LoggedError: File %s is locked by another process, you are running with MPI disabled but may have more than one process. Note that --test should not be used with MPI.
-------------------------------------------------------------
 

After that, I am also deleting the output folder before I run the .yaml file again. Unfortunately, I face again the same error.

The machine has Ubuntu 20.04.4 LTS (with 24 cores) and these are the stuff that I am using:
- Cobaya was installed from github repo.
- It is running within a conda environment with python = 3.8
- I have OpenMpi 4.1.3
- I installed the python wrapper mpi4py 3.1.3 by using:

Code: Select all

conda install -c conda-forge mpi4py

(I also tried to install through pip but it keeps displaying the same error).

Moreover, when I run

Code: Select all

mpirun -n 4 hostname 

, the output is printed correctly. I also checked the version of mpi4py as Cobaya documentation suggests as

Code: Select all

mpirun -n 2 python -c "from mpi4py import MPI, __version__; print(__version__ if MPI.COMM_WORLD.Get_rank() else '')"

and it is 3.1.3 version. So, I am not sure if my mpirun is the problem, I am out of ideas.


Please, could anyone help me on how to solve that issue? I really need to use Cobaya with MPI.


Thanks in advance,
Regards,
Karim

[Antony Lewis]

Note sure. Check COBAYA_NOMPI env variable is not set. MPI being disabled should only be because either that is set or MPI cannot be imported from mpi4py.

[Juan Cruz]

I was also having similar issues. Make sure there is only one openmp and that it is properly linked. You probably have different installations, one from your linux package manager and one from anaconda. I would say option one, try to create a clean new environment in conda and at creation include openmp and mpi4py so that conda does the correct linking. Option 2 create a new invironment without openmp or mpi4py and use pip (of the new enviroment) afterwards to install mpi4py, this should link to the openmp from the linux package manager. Hopefully this solves the issue.

Obs: Also your logs seem to say python 3.9 not 3.8 and you seem to have installed cobaya using the "--user" which makes things go to .local, for easy access and cleanliness when using conda environments avoid using --user so that the environment remains self-contained. (my opinion)

[Hao-Hao Li]

I meet the same error when I use command

Code: Select all

 mpirun -np 4 cobaya-run test-mpi.yaml

the error is

Code: Select all

cobaya.log.LoggedError: File %s is locked by another process, you are running with MPI disabled but may have more than one process. Make sure that you have mpi4py installed and working.Note that --test should not be used with MPI.

My mistake is that I installed mpi4py by default channel, it should be conda-forge channel.
First, use command:

Code: Select all

conda list

to check the version and channel of your mpi4py. If the version is satisfied, but the channel is default, then you should uninstall your mpi4py

Code: Select all

conda uninstall mpi4py

Next, reinstall mpi4py by conda-forge channel

Code: Select all

conda install -c conda-forge mpi4py

I hope these operations will be helpful.