Segmentation fault error while running FSI example

[MaganS]

Hi,

I installed SU2-7.5.1 on our institute's cluster, which uses Linux OS based on CentOS 7.x distribution. I configured it with the following options.
-Dcustom-mpi=true -Denable-mkl=true -Dmkl_root=/home-ext/apps/spack/opt/spack/linux-centos7-cascadelake/intel-19.0.5.281/intel-mkl2020.4.304-fet6h2j2qeq5alsxjiw7fzjkweqorbjf/mkl/ -Denable-pywrapper=true

I installed SU2-7.5.1 with the following required packages.
GCC v12.2.0
py-mpi4py v3.1.4
openmpi v4.1.5
python v3.10.10
swig v4.1.1
intel-mkl v2020.4.304

Now, I am checking the code by running the following FSI example on the cluster.
https://github.com/su2code/Tutorials/tree/master/multiphysics/unsteady_fsi_python/Ma01

I am getting the following error while running the example.
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
mpirun noticed that process rank 116 with PID 0 on node cn215 exited on signal 11 (Segmentation fault).

The following is my job script for running on cluster.
#!/bin/bash
#SBATCH -N 3
#SBATCH --ntasks-per-node=48
#SBATCH --time=00:04:00
#SBATCH --job-name=ddd
#SBATCH --error=job.%J.err
#SBATCH --output=job.%J.out
#SBATCH --exclude=cn001
#SBATCH --partition=debug
echo "Number of Nodes Allocated = $SLURM_JOB_NUM_NODES"
echo "Number of Tasks Allocated = $SLURM_NTASKS"
echo "Number of Cores/Task Allocated = $SLURM_CPUS_PER_TASK"
cd $SLURM_SUBMIT_DIR
. /opt/ohpc/admin/lmod/lmod/init/sh
module load spack/0.17
. /home-ext/apps/spack/share/spack/setup-env.sh
spack unload
spack load [email protected]%gcc@=11.2.0 /5o2xfb3
spack load [email protected]%gcc@=12.2.0 /z5t7bsx
spack load [email protected]%gcc@=12.2.0 /6waw53s
spack load [email protected]%gcc@=12.2.0 /6m7g42w
spack load [email protected]%[email protected] /fet6h2j
mpirun -n $SLURM_NTASKS --oversubscribe python3 -m mpi4py /scratch/asemagan/exctbls/su2/SU2-7.3.1/bin/fsi_computation.py --parallel -f fsi.cfg

But the same example runs well on our lab PC based on Manjaro Linux. I installed SU2-7.5.1 on the cluster with the same versions of packages as in our lab PC; those are mentioned above.

Please suggest. Any help will be appreciated.

Thank you in advance

[MaganS]

Installing with following particular versions solved the problem.
gcc v12.2.0
mpi4py v3.1.4
python v3.10.8
openmpi v4.1.5
swig v4.1.1
cmake v3.25.1
numpy v1.24.0
scipy v1.5.4
rtree v1.0.1
petsc4py v3.16.0

Best,
Magan

[aozuzun]

Hey MaganS,

I have a similar error #2070 , so I will try to reinstall, as you did. May I ask how to reinstall using specified MPI, compilers, and libraries during the installation process? I will use the meson.py and ninja commands again as described in the below, right? I mean there is no fundamental change in installation procedure?

https://su2code.github.io/docs_v7/Build-SU2-Linux-MacOS/

[MaganS]

Hello,
My installation procedure is the same as given on link (you mentioned) for FSI problems. You can try these versions by loading them into your environment and exporting CC and CXX and adding their libraries to PATH.

We are using spack package manager. This is what I have done on cluster and followed the same procedure as given on link for installation.
spack unload
spack load [email protected]%gcc@=11.2.0 /5o2xfb3
spack load [email protected]%gcc@=12.2.0 /xewtquj
spack load [email protected]%gcc@=12.2.0 /ow3ikza
spack load [email protected]%gcc@=12.2.0 /6waw53s
spack load [email protected]%gcc@=12.2.0 /azublrf
spack load [email protected]%gcc@=12.2.0 /mroegiy
spack load [email protected]%gcc@=12.2.0 /y4ztfyq
spack load [email protected]%gcc@=12.2.0 /fd47w2h
spack load [email protected]%gcc@=12.2.0 /kzrwewu
spack load py-petsc4py

export MPICC="/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-12.2.0/openmpi-4.1.5-6waw53sisgrm5ennlo67m6bubosrxwrn/bin/mpicc"
export MPICXX="/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-12.2.0/openmpi-4.1.5-6waw53sisgrm5ennlo67m6bubosrxwrn/bin/mpic++"
export CC=$MPICC
export CXX=$MPICXX
export PATH=$PATH:/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-12.2.0/openmpi-4.1.5-6waw53sisgrm5ennlo67m6bubosrxwrn/lib
export PATH=$PATH:/home/ext/apps/spack/opt/spack/linux-centos7-cascadelake/gcc-12.2.0/openmpi-4.1.5-6waw53sisgrm5ennlo67m6bubosrxwrn/include

[aozuzun]

Thank you for your response. I followed your method and reinstalled SU2, all libraries, and compilers, but unfortunately, it did not resolve the error in my situation.

However, I managed to solve the error on my own. The problem was resolved when I made the following changes to the MPI parameters.

mpirun --mca btl_openib_cpc_exclude rdmacm --mca mpi_leave_pinned 0 --mca btl_openib_allow_ib 1 --mca btl openib,self,vader -n $NSLOTS