install NGsolve without root access

About taking the right MPI-version:

Basically, cmake is looking for a library called libmpi (and a few others, I think) and where it looks is determined (among others) by your LD_LIBRARY_PATH , which is I think searched through front to back, so make sure that the mpi-library you want to use shows up first. You can also use the cmake-variable CMAKE_PREFIX_PATH to give cmake additional hints where to look for libraries, however I am not sure in which order LD_LIBRARY_PATH and the prefix_paths are looked through.

In any case, cmake should give you shell output which exact MPI-library it will be using, and at a later date you can look up which library was used in "build_folder/CMakeCache.txt", where there should be a couple of lines like these:

//MPI CXX libraries to link against
MPI_CXX_LIBRARIES:STRING=/home/lukas/local/openmpi-2.1-gcc-6.2/lib/libmpi.so

//MPI CXX linking flags
MPI_CXX_LINK_FLAGS:STRING=-Wl,-rpath -Wl,/home/lukas/local/openmpi-2.1-gcc-6.2/lib -Wl,--enable-new-dtags


Environment modules can be used to properly seperate different versions of MPI if available.




The last problem is definitely a bug/missing feature, I am working on it. Thank you for bringing this to our attention.

Temporary workaround:

Change
a = BilinearForm (V, symmetric=False)
to
a = BilinearForm (V, symmetric=True)

However, keep in mind that you have to put it back to symmetric=False if you want to use HYPRE.

You can also tell bddc to use MUMPS inverse (once you get that running), this should generally work.


If you are interested in what exactly the problem is, the story goes like this:

Symmetric storage means that only the lower triangular part (+the diagonal) of the matrix are stored, which saves a bit of memory.

Masterinverse collects the entire matrix to be inverted on the master proc, which then inverts it - in combination with bddc, this should be ok for smaller problems, because bddc already decreases the size of the problem that has to be solved considerably.

This collecting of the matrix is currently only properly implemented for symmetrically stored matrices, and unfortunately it does not "terminate gracefully" if called with a fully stored one. This should only require a small fix.

I created an issue on gitlab for this:
gitlab.asc.tuwien.ac.at/jschoeberl/ngsolve/issues/42

Hi Matthias,

Thanks for the detailed response.
Yeah, umfpack is compiled in a lower version than ngsolve, similar things happens for mumps.
I will use your setup to see what happens then.

Even without the linear solver, I am pretty happy with the current MPI version.

Best,
Guosheng

Now mumpus is installed with the same compiler with ngsolve, but I still get the following error at the final linking stage:

CMakeFiles/ngs.dir/ngs.cpp.o: In function `_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPKcEEvT_S8_St20forward_iterator_tag.isra.32':
/panfs/roc/msisoft/gcc/5.1.0/include/c++/5.1.0/bits/basic_string.tcc:223: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long)'
libsolve.so: undefined reference to `std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream(std::_Ios_Openmode)'
../comp/libngcomp.so: undefined reference to `VTT for std::__cxx11::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >'
libsolve.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::push_back(char)'
../comp/libngcomp.so: undefined reference to `std::basic_istream<char, std::char_traits<char> >& std::operator>><char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
libsolve.so: undefined reference to `std::runtime_error::runtime_error(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
../comp/libngcomp.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::resize(unsigned long, char)'
../linalg/libngla.so: undefined reference to `blacs_gridinfo_'
libsolve.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_erase(unsigned long, unsigned long)'
../comp/libngcomp.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::rfind(char, unsigned long) const'
libsolve.so: undefined reference to `std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_stringstream()'
libsolve.so: undefined reference to `VTT for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >'
libsolve.so: undefined reference to `std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::str() const'
libsolve.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find(char const*, unsigned long, unsigned long) const'
libsolve.so: undefined reference to `VTT for std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >'
libsolve.so: undefined reference to `operator delete[](void*, unsigned long)'
../comp/libngcomp.so: undefined reference to `std::__cxx11::collate<char> const& std::use_facet<std::__cxx11::collate<char> >(std::locale const&)'
libsolve.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::append(char const*)'
libsolve.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::assign(char const*)'
libsolve.so: undefined reference to `std::runtime_error::runtime_error(char const*)'
libsolve.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::rfind(char const*, unsigned long, unsigned long) const'
libsolve.so: undefined reference to `std::runtime_error::runtime_error(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
libsolve.so: undefined reference to `vtable for std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >'
../linalg/libngla.so: undefined reference to `pzgetrs_'
libsolve.so: undefined reference to `vtable for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >'
libsolve.so: undefined reference to `operator delete(void*, unsigned long)'
libsolve.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_replace(unsigned long, unsigned long, char const*, unsigned long)'
../linalg/libngla.so: undefined reference to `blacs_gridinit_'
../comp/libngcomp.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()'
/home/cockburn/fug/netgen/inst/lib/libinterface.so: undefined reference to `std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::_Ios_Openmode)'
../fem/libngfem.so: undefined reference to `std::runtime_error::runtime_error(std::runtime_error const&)'
../linalg/libngla.so: undefined reference to `pzpotrf_'
../linalg/libngla.so: undefined reference to `descinit_'
libsolve.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long)'
libsolve.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::swap(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
../comp/libngcomp.so: undefined reference to `std::basic_ostream<char, std::char_traits<char> >& std::operator<< <char, std::char_traits<char>, std::allocator<char> >(std::basic_ostream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
../comp/libngcomp.so: undefined reference to `vtable for std::__cxx11::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >'
libsolve.so: undefined reference to `std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_ostringstream()'
../linalg/libngla.so: undefined reference to `pzgetrf_'
../fem/libngfem.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::replace(unsigned long, unsigned long, char const*, unsigned long)'
../linalg/libngla.so: undefined reference to `pdpotrf_'
libsolve.so: undefined reference to `vtable for std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >'
../linalg/libngla.so: undefined reference to `pdgetrs_'
../ngstd/libngstd.so: undefined reference to `std::basic_istream<char, std::char_traits<char> >& std::getline<char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char)'
libsolve.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_append(char const*, unsigned long)'
libsolve.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(char const*) const'
libsolve.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve(unsigned long)'
../comp/libngcomp.so: undefined reference to `std::__cxx11::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_istringstream()'
../comp/libngcomp.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const'
../linalg/libngla.so: undefined reference to `blacs_gridexit_'
../linalg/libngla.so: undefined reference to `pdpotrs_'
../comp/libngcomp.so: undefined reference to `std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::_M_sync(char*, unsigned long, unsigned long)'
../linalg/libngla.so: undefined reference to `pzpotrs_'
../comp/libngcomp.so: undefined reference to `std::basic_ofstream<char, std::char_traits<char> >::basic_ofstream(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::_Ios_Openmode)'
libsolve.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::find(char, unsigned long) const'
libsolve.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
../comp/libngcomp.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::erase(unsigned long, unsigned long)'
../ngstd/libngstd.so: undefined reference to `std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>, void (*)())'
libsolve.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::append(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long)'
../comp/libngcomp.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct(unsigned long, char)'
libsolve.so: undefined reference to `std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream(std::_Ios_Openmode)'
../linalg/libngla.so: undefined reference to `numroc_'
libsolve.so: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_replace_aux(unsigned long, unsigned long, unsigned long, char)'
libsolve.so: undefined reference to `std::runtime_error::runtime_error(char const*)'
../linalg/libngla.so: undefined reference to `pdgetrf_'
collect2: error: ld returned 1 exit status
make[5]: *** [solve/ngs] Error 1
make[4]: *** [solve/CMakeFiles/ngs.dir/all] Error 2
make[3]: *** [all] Error 2
make[2]: *** [dependencies/Stamp/ngsolve/ngsolve-build] Error 2
make[1]: *** [CMakeFiles/ngsolve.dir/all] Error 2
make: *** [all] Error 2

Just to add an additional comment.

Now, I finally got consistent result on the installation error. My previously working installation without mumps crashes with the same message :<

It fails to generate the dynamic library libsolve.so at the final stage, and the following error message appears no matter whether MPI is turned on/off or mumps is turned on/off:
CMakeFiles/ngs.dir/ngs.cpp.o: In function `main':
ngs.cpp:(.text.startup+0xda): undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long)'
libsolve.so: undefined reference to `std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::basic_stringstream(std::_Ios_Openmode)'


I tried to use different python3 libraries, or different gcc compilers, but no good luck.

Continue with a positive update.

I realized the issue is indeed incompatible of gcc compiler version as Matthias pointed out.
I used an mpi module mpich which seems to be only compatible with gcc 4.9.2, because when I call module avail mpich, it gives me
mpich/3.1.4/gnu-4.9.2

After changing my gcc compiler to version 4.9.2, I am able to install and run ngsolve with MPI, still without direct solver installed.
I changed the compiler to version 5.1.0 or 6.1.0, then the same ...std_cxx11... error appears.

Now, coming back to mumps issue. After turning on mumps, I have parmetis and mumps installed with the same gcc 4.9.2 compiler, but it gives me the following error when building the libsolve.so library at the final step

../linalg/libngla.so: undefined reference to `blacs_gridinfo_'
../linalg/libngla.so: undefined reference to `pzgetrs_'
../linalg/libngla.so: undefined reference to `blacs_gridinit_'
../linalg/libngla.so: undefined reference to `pzpotrf_'
../linalg/libngla.so: undefined reference to `descinit_'
../linalg/libngla.so: undefined reference to `pzgetrf_'
../linalg/libngla.so: undefined reference to `pdpotrf_'
../linalg/libngla.so: undefined reference to `pdgetrs_'
../linalg/libngla.so: undefined reference to `blacs_gridexit_'
../linalg/libngla.so: undefined reference to `pdpotrs_'
../linalg/libngla.so: undefined reference to `pzpotrs_'
../linalg/libngla.so: undefined reference to `numroc_'
../linalg/libngla.so: undefined reference to `pdgetrf_'


I typed
nm ngsolve/linalg/libngla.so -a | grep blacs
and get
U blacs_gridexit_
U blacs_gridinfo_
U blacs_gridinit_
Similar thing for the other reference names, all start with a captal U
What does this mean?

We are getting closer to a running version. The problem now is that libscalapack is not linked to libngla. In case you are linking with MKL this should be handled automatically. So I assume you disabled USE_MKL, right?

Without MKL you have to set scalapack manually, e.g. by configuring with
-DSCALAPACK_LIBRARY=/usr/lib/libscalapack.so

Regards,
Matthias