Numerical artifacts when using MPI parallelisation

2 weeks 1 hour ago #2908 by philipp
Dear NGSolve community,

recently I have been experimenting with NGSolve and MPI.
For a start, I used a transport problem whose velocity is given by a Stokes
equation. With shared-memory parallelisation this works just fine.
When I switch to MPI (basically the same code but with mesh distribution),
I observe first of all that the velocity somehow increases and also
its vector field is being distorted. Depending on the number of ranks,
their seem also to be artifacts near the boundary, where the mesh is divided.
For matrix inversion I tried pardiso and bddc + GMRes which both produce
the same result.

I have no idea what kind of problem is causing this and any hints or suggestions
are very much appreciated.

All the best,
Philipp

Please Log in or Create an account to join the conversation.

1 week 6 days ago #2910 by lkogler
Could you upload your code? I will take a look at it.

Best,
Lukas

Please Log in or Create an account to join the conversation.

1 week 6 days ago #2911 by philipp
Hallo Lukas,

thank you very much for your interest.
I attached a mwe (just forced instationary Stokes flow) that reproduces the
issue on my system. The problem is particularly evident with three ranks.

All the best,
Philipp
Attachments:

Please Log in or Create an account to join the conversation.

1 week 4 days ago - 1 week 4 days ago #2917 by lkogler
The problem is that the NumberFESpace does not work with MPI. Fixing this should not be a problem, but it requires a couple tweaks in numerous places - expect it in the master sometime this week, I will post here when it is in the master branch.

However, even once it works, be aware that the NumberFESpace is quite a bottleneck with MPI - it consists of one DOF that is shared by all MPI ranks, so whenever a vector is cumulated (so when it's values are made consistent) EVERY rank (except rank 0) sends a message to, and receives a message from EVERY OTHER rank (except rank 0).

For this reason, in practice it might be a good idea to add a small regularization in the pressure-pressure block for Stokes problems with pure dirichlet conditions.

Best,
Lukas

Please Log in or Create an account to join the conversation.

1 week 3 days ago #2920 by philipp
Thank you very much for the clarification and the advice. I will go with the way you suggested for the sake of performance.

All the best,
Philipp

Please Log in or Create an account to join the conversation.

1 week 2 days ago #2926 by lkogler
The NumberFESpace should work with MPI with the coming nightly release.

Best, Lukas
The following user(s) said Thank You: philipp

Please Log in or Create an account to join the conversation.

© 2019 Netgen/NGSolve