I have not found a solution I am really satisfied with myself, but I can tell you what I usually do.
Just keep in mind that this is all just cobbled together.
For small jobs, you can run each MPI process in a seperate xterm-window and attach gdb to each of them.
Code:
mpirun -np 1 gdb -ex run --args python3 PYTHON_SCRIPT : \
-np X xterm -hold -e gdb -ex run --args python3 PYTHON_SCRIPT
You can also pipe the output of each process to a seperate file, for example, with OpenMPI you could write
a small script like this:
Code:
#!/bin/sh
$@ 1>out_p$OMPI_COMM_WORLD_RANK 2>err_p$OMPI_COMM_WORLD_RANK
Then you can use gdb on each process, and tell it to print the back-trace when it encounters a problem
Code:
mpirun -np X bash WRAP_SCRIPT gdb -batch -ex "run" -ex bt --args python3 PYTHON_SCRIPT
This way you can do bigger jobs, but you cannot use gdb interactively.
This also works with valgrind instead of gdb.