dwifslpreproc: Cannot run eddy_quad subcommand via MRtrix3 Docker image

[jhuguetn]

Describe the bug

Running eddy_quad via the MRtrix3 Docker image throws an exception. The objective was running the command via dwifslpreproc internally but it crashed.

To Reproduce

docker run -it --rm -v `pwd`/data:/data mrtrix3/mrtrix3
root@17ffaea246c9:/# /opt/fsl/fslpython/envs/fslpython/bin/python3.7 /opt/fsl/fslpython/envs/fslpython/bin/eddy_quad subject -idx ./eddy_idx.txt -par ./eddy_cfg.txt -m /data/mask.nii.gz -b /data/DWI_MB2.bval -g /data/DWI_MB2.bvec -o /data/ -v
Error processing line 1 of /opt/fsl/fslpython/envs/fslpython/lib/python3.7/site-packages/sphinxcontrib_applehelp-1.0.2-py3.8-nspkg.pth:

  Traceback (most recent call last):
    File "/opt/fsl/fslpython/envs/fslpython/lib/python3.7/site.py", line 168, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "<frozen importlib._bootstrap>", line 580, in module_from_spec
  AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored
Error processing line 1 of /opt/fsl/fslpython/envs/fslpython/lib/python3.7/site-packages/sphinxcontrib_devhelp-1.0.2-py3.8-nspkg.pth:

  Traceback (most recent call last):
    File "/opt/fsl/fslpython/envs/fslpython/lib/python3.7/site.py", line 168, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "<frozen importlib._bootstrap>", line 580, in module_from_spec
  AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored
Error processing line 1 of /opt/fsl/fslpython/envs/fslpython/lib/python3.7/site-packages/sphinxcontrib_htmlhelp-2.0.0-py3.9-nspkg.pth:

  Traceback (most recent call last):
    File "/opt/fsl/fslpython/envs/fslpython/lib/python3.7/site.py", line 168, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "<frozen importlib._bootstrap>", line 580, in module_from_spec
  AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored
Error processing line 1 of /opt/fsl/fslpython/envs/fslpython/lib/python3.7/site-packages/sphinxcontrib_jsmath-1.0.1-py3.7-nspkg.pth:

  Traceback (most recent call last):
    File "/opt/fsl/fslpython/envs/fslpython/lib/python3.7/site.py", line 168, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "<frozen importlib._bootstrap>", line 580, in module_from_spec
  AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored
Error processing line 1 of /opt/fsl/fslpython/envs/fslpython/lib/python3.7/site-packages/sphinxcontrib_qthelp-1.0.3-py3.8-nspkg.pth:

  Traceback (most recent call last):
    File "/opt/fsl/fslpython/envs/fslpython/lib/python3.7/site.py", line 168, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "<frozen importlib._bootstrap>", line 580, in module_from_spec
  AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored
Error processing line 1 of /opt/fsl/fslpython/envs/fslpython/lib/python3.7/site-packages/sphinxcontrib_serializinghtml-1.1.5-py3.9-nspkg.pth:

  Traceback (most recent call last):
    File "/opt/fsl/fslpython/envs/fslpython/lib/python3.7/site.py", line 168, in addpackage
      exec(line)
    File "<string>", line 1, in <module>
    File "<frozen importlib._bootstrap>", line 580, in module_from_spec
  AttributeError: 'NoneType' object has no attribute 'loader'

Remainder of file ignored
Traceback (most recent call last):
  File "/opt/fsl/fslpython/envs/fslpython/lib/python3.7/site-packages/numpy/core/__init__.py", line 24, in <module>
    from . import multiarray
  File "/opt/fsl/fslpython/envs/fslpython/lib/python3.7/site-packages/numpy/core/multiarray.py", line 14, in <module>
    from . import overrides
  File "/opt/fsl/fslpython/envs/fslpython/lib/python3.7/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: libcblas.so.3: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/fsl/fslpython/envs/fslpython/bin/eddy_quad", line 53, in <module>
    from eddy_qc.QUAD.quad import main
  File "/opt/fsl/fslpython/envs/fslpython/lib/python3.7/site-packages/eddy_qc/QUAD/__init__.py", line 1, in <module>
    from . import quad
  File "/opt/fsl/fslpython/envs/fslpython/lib/python3.7/site-packages/eddy_qc/QUAD/quad.py", line 8, in <module>
    import numpy as np
  File "/opt/fsl/fslpython/envs/fslpython/lib/python3.7/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/opt/fsl/fslpython/envs/fslpython/lib/python3.7/site-packages/numpy/core/__init__.py", line 50, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.7 from "/opt/fsl/fslpython/envs/fslpython/bin/python3.7"
  * The NumPy version is: "1.18.5"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: libcblas.so.3: cannot open shared object file: No such file or directory

Platform/Environment/Version

Please provide the following information:

• OS: Docker image mrtrix3/mrtrix3:3.0.4 (d4034fa0d666)
• MRtrix3 version: 3.0.4 (Docker image)

[jhuguetn]

Issue might be related with #2165, hence we used /opt/fsl/fslpython/envs/fslpython/bin/python3.7 (and not the system default Python binary) to run eddy_quad.

[Lestropie]

Thanks for the comprehensive report. This may relate to FSL installation issues more generally, which has been a bit of an issue with #2689 and since. I've not 100% followed the full conversation, but it's something like: including ${FSLDIR}/bin in ``PATHis problematic because that includes a bunch of things that you don't want corrupting yourPATH` (like compilers); `${FSLDIR}/share/fsl/bin` should be used instead. I don't know if that change alone would rectify the problem here though.

On my native system eddy_quad has a shebang pointing straight to `${FSLDIR}/bin/python.

I just downloaded the 3.0.4 container, and it seems to me that while eddy_quad is present in fslpython/envs/fslpython/bin, it is actually missing from the FSL bin/ directory in the minified download, contrary to the intention of the minification process, so dwifslpreproc doesn't see it. Have you performed any other manipulation of the container prior to getting to the current sticking point? It might be that your report is a secondary problem, and it's actually the eddy_quad executable missing from the minified container that's the core problem.

[jhuguetn]

including ${FSLDIR}/bin in PATH is problematic because that includes a bunch of things that you don't want corrupting your PATH (like compilers); ${FSLDIR}/share/fsl/bin should be used instead. I don't know if that change alone would rectify the problem here though.

Such ${FSLDIR}/share/fsl/bin directory does not exists in the Docker container setting (there's a fslpython virtual environment supposedly though), see below:

jordi@pc:~/$ docker run -it --rm -v `pwd`/data:/data mrtrix3/mrtrix3
root@74f3f873f23f:/# find $FSLDIR -type d -name bin
/opt/fsl/bin
/opt/fsl/fslpython/envs/fslpython/bin

On my native system eddy_quad has a shebang pointing straight to `${FSLDIR}/bin/python.

On mine as well, but in the MRtrix3 Docker container it is pointing at ${FSLDIR}/fslpython/envs/fslpython/bin/python (which does not exist btw but can be softlinked to python3.7), see below:

root@74f3f873f23f:/# cat /opt/fsl/fslpython/envs/fslpython/bin/eddy_quad
#!/opt/fsl/fslpython/envs/fslpython/bin/python
import argparse
import sys
...

root@74f3f873f23f:/# ls -lr ${FSLDIR}/fslpython/envs/fslpython/bin/py*
-rwxrwxr-x 1 root root 12806296 Sep 28  2021 /opt/fsl/fslpython/envs/fslpython/bin/python3.7
-rwxrwxr-x 1 root root      107 Sep 28  2021 /opt/fsl/fslpython/envs/fslpython/bin/pydoc3.7
-rwxrwxr-x 1 root root      276 Sep 28  2021 /opt/fsl/fslpython/envs/fslpython/bin/pydicom

Have you performed any other manipulation of the container prior to getting to the current sticking point?

Nope, it's a clean MRTrix3 Image I am using, pulled from dockerhub repo. Anyone can easily reproduce it I believe. I also think that the main issue relates to eddy_quad executable missing in the minified FSL container and/or is misconfigured as is in the fslpython virtual environment.

So, I added the following fixes to make eddy_quad work, but this is just patching (see below from within a pristine MRtrix3 container). I think we would need to come up with a proper installation/configuration of all the FSL tools required by MRtrix3 in the Docker image. Let me know if we can help you somehow.

# (optional) add eddy_quad to PATH indirectly by adding it into the ${FSLDIR}/bin location
cd /opt/fsl/bin/
ln -s ../fslpython/envs/fslpython/bin/eddy_quad eddy_quad

# repair the eddy_quad shebang directive to use the virtual environment Python3 binary
cd /opt/fsl/fslpython/envs/fslpython/bin/
ln -s python3.7 python

# get rid of *.pth files frin Sphinx that seem to throw errors
rm -rf -v /opt/fsl/fslpython/envs/fslpython/lib/python3.7/site-packages/sphinxcontrib_*.pth

# install missing libcblas library 
find / -name libcblas.so.*
apt-get install libatlas-base-dev

# fix path to existing libraries location 
export LD_LIBRARY_PATH=/opt/fsl/fslpython/envs/fslpython/lib:$LD_LIBRARY_PATH

# add version-related softlinks to existing libraries 
cd /opt/fsl/fslpython/envs/fslpython/lib
ln -s libffi.so.6.0.4 libffi.so.6
ln -s libjpeg.so.9.4.0 libjpeg.so.9
ln -s libgfortran.so.4.0.0 libgfortran.so.4
ln -s libgfortran.so.4.0.0 libgfortran.so.4

[Lestropie]

Such ${FSLDIR}/share/fsl/bin directory does not exists in the Docker container setting

Correct: because my process of generating that dependency explicitly deletes them. But potentially, if my process of generating that dependency were to flag them as requiring preservation, and the container were to then invoke commands from that location via PATH, the issue might self-resolve. I'm attempting that now, it's just taking some time due to storage space issues.

[Lestropie]

It's possible that there have been upstream changes in the Python ecosystem that have resulted in the container's FSL configuration failing, and therefore the absence of the EddyQC command. /opt/fsl/etc/fslconf/fslpython_install.sh gives:

ERROR: Package 'requests' requires a different Python: 3.7.6 not in '>=3.8'

That Python version is specifically that of the FSL conda environment, not that of the MRtrix minification container. So it might not be possible to build the 3.0.x container without either hacking the FSL conda environment (might be feasible for yourself short term), or by incrementing the FSL version (what I'm more likely to do for a 3.0.5 fix).

[anaharrismatnez]

So it might not be possible to build the 3.0.x container without either hacking the FSL conda environment

What dou you mean by hacking the FSL conda environment? Could you explain what your idea is, or would it be something similar to what @jhuguetn reported above?

# (optional) add eddy_quad to PATH indirectly by adding it into the ${FSLDIR}/bin location
cd /opt/fsl/bin/
ln -s ../fslpython/envs/fslpython/bin/eddy_quad eddy_quad

# repair the eddy_quad shebang directive to use the virtual environment Python3 binary
cd /opt/fsl/fslpython/envs/fslpython/bin/
ln -s python3.7 python

# get rid of *.pth files frin Sphinx that seem to throw errors
rm -rf -v /opt/fsl/fslpython/envs/fslpython/lib/python3.7/site-packages/sphinxcontrib_*.pth

# install missing libcblas library 
find / -name libcblas.so.*
apt-get install libatlas-base-dev

# fix path to existing libraries location 
export LD_LIBRARY_PATH=/opt/fsl/fslpython/envs/fslpython/lib:$LD_LIBRARY_PATH

# add version-related softlinks to existing libraries 
cd /opt/fsl/fslpython/envs/fslpython/lib
ln -s libffi.so.6.0.4 libffi.so.6
ln -s libjpeg.so.9.4.0 libjpeg.so.9
ln -s libgfortran.so.4.0.0 libgfortran.so.4
ln -s libgfortran.so.4.0.0 libgfortran.so.4

Regarding the new version, when do you expect this version (3.0.5) to be ready?

[Lestropie]

What do you mean by hacking the FSL conda environment?

Either something like what you have there, or maybe incrementing the Python version to 3.8 and seeing whether it breaks anything.

Regarding the new version, when do you expect this version (3.0.5) to be ready?

The tag is likely still some time away. But if I can successfully regenerate the minified dependency, push changes to master, and push a new mrtrix3:latest, a container would then be available that works out of the box.

Slightly nervous about it though given that I don't seem to be able to regenerate the minified FSL dependency using the same version of FSL. This means that the changes to the container won't just be "fixing" the dependency, there's a risk that FSL behaviour could have changed, and that ideally shouldn't happen within a patch update of our container.