proper environment setup for MPI in jupyter notebook

Hello,

How to properly set up jupyter notebook so that I can use MPI there? I couldn't find a solution online.
For python, we simply preload the library using the ngspy hack: Can we do the same thing for jupyter notebook?

Best,
Guosheng

Are you really sure this is something you absolutely want to do?

As far as the preloads are concerned, If you set the LD_PRELOAD from ngspy as an environment variable it should work. But that will not yet allow you to run MPI parallel jupyter notebooks - the jupyter notebook spawns a python instance in the background, and you want THAT to be parallel, not the notebook itself.

I have managed to set something like this up in the past using jupyter clusters, but is is very finicky and a bit unstable.

You can look into jupyter clusters and have a look at the MPI jupyter notebooks from the 2nd user meeting if you want to give it a try. I can also give you the ipyparallel config files I used back then if you want.


Best,
Lukas

Hi Lukas,

Looks very complicated to do... ;<
I am doing jupyter only for education purpose. I just want to show the petsc amg preconditioner from the ngs-petsc library, which needs MPI to start with.
Currently, I simply go back to the plain python code to do the demo.
Can you walk me thought the setup? :>

Best,
Guosheng

The petsc-amg preconditioner needs an NGSolve version that is compiled with MPI, but it does not require you to actually run in parallel.

If you want to use this in a sequential run (but an MPI version), just setting the LD_LIBRARY_PATH should do it. It only gets complicated when you want multiple MPI ranks.

If this does not work for some reason, a workaround would be to manually load the libraries from the ngspy script at the top of your jupyter notebook. Have a look at CDLL from the ctypes python module.

Best,
Lukas

I just want to use the preconditioner in serial.
Setting LD_LIBRARY_PATH in my bashrc file does not work.
Can you give me a demo on manual load via ngspy?
CDLL and ctypes in python are foreign language to me. I am afraid dig into that.

Put something like this at the top of the jupyter notebook
Generally, I have found that you really only need to preload the libmkl_rt library when you do it this way.

Best,
Lukas