-
Notifications
You must be signed in to change notification settings - Fork 133
Troubleshooting
We strongly encourage using conda to maintain the Python libraries on which RAVEN depends. Note that conda can be installed locally without administrative privileges on most systems. However, if conda is not available, the necessary libraries and versions can be obtained by the following call:
cd raven
python scripts/TestHarness/testers/RavenUtils.py --conda-installThis will print to terminal a command that includes a listing of all the libraries, some with specific tagged versions. These libraries may be installed through pip or other Python or OS package managers, after which RAVEN can be run.
If library versions other than the specific tagged versions are installed, RAVEN will exit when run and complain of improper libraries for the version of RAVEN you are trying to run.
Note that the RAVEN scripts are tested in bash, so if you are using zsh or tsch shells you will probably get strange errors.
If there are missing libraries, (for example if you get errors like ImportError: No module named sklearn) then the Installing RAVEN Libraries page should be used to try and fix the problem.
In the raven directory, the command:
./run_tests --library_report
can be used to see which libraries are being used and which are missing.
Older versions of the MOOSE redistributable package (From http://mooseframework.org/getting-started/osx/ ) contains everything needed to run RAVEN except the scikit-learn package for Python. If this package is missing, running the tests looks like this...
[RAVENUser][~/projects/raven]> ./run_tests
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named sklearn
No raven_libs found and missing modules sklearn
core_example.PWR_CoreChannel_pre_dist.................................... skipped (Broken for unknown reason)
core_example.PWR_CoreChannel_controlled_var_csv......... skipped (Broken, and Josh is not sure what it te...)
core_example.TypPWR_Mult_CoreChannels_control_csv................................. skipped (Relap issue #111)
core_example.TypPWR_Mult_CoreChannels_control..................................... skipped (Relap issue #111)
check_errors.missing_python_file.......................................................................... OK
core_example.simple_branch................................................................................ OK
core_example.pump......................................................................................... OK
DtControlAndProximity.Dt_controlling_test................................................................. OK
DtControlAndProximity.TimeController_test................................................................. OK
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named sklearn
framework.simple_framework skipped (Missing python modules: sklearn PYTHONPATH=/Users/bobk/raven_libs/pylibs/lib/python2.7/site-packages/:/opt/moose/distcc-3.2rc1/lib/python2.6/site-packages)
Fortunately, the redistributable includes the Miniconda package manager which can be used to install it.
[RAVENUser][~]> sudo su
Password:
sh-3.2# conda install scikit-learn
Fetching package metadata: ....
Solving package specifications: .
Package plan for installation in environment /opt/moose/miniconda:
The following packages will be downloaded:
package | build
---------------------------|-----------------
sqlite-3.8.4.1 | 1 824 KB
tk-8.5.18 | 0 1.9 MB
conda-env-2.1.4 | py27_0 15 KB
nose-1.3.4 | py27_1 189 KB
numpy-1.9.2 | py27_0 2.9 MB
requests-2.6.0 | py27_0 594 KB
setuptools-15.0 | py27_0 436 KB
conda-3.10.1 | py27_0 164 KB
pip-6.1.1 | py27_0 1.4 MB
scikit-learn-0.16.0 | np19py27_0 3.3 MB
------------------------------------------------------------
Total: 11.7 MB
The following NEW packages will be INSTALLED:
nose: 1.3.4-py27_1
pip: 6.1.1-py27_0
scikit-learn: 0.16.0-np19py27_0
The following packages will be UPDATED:
conda: 3.10.0-py27_0 --> 3.10.1-py27_0
conda-env: 2.1.3-py27_0 --> 2.1.4-py27_0
numpy: 1.9.1-py27_0 --> 1.9.2-py27_0
requests: 2.5.3-py27_0 --> 2.6.0-py27_0
setuptools: 12.2-py27_0 --> 15.0-py27_0
sqlite: 3.8.4.1-0 --> 3.8.4.1-1
tk: 8.5.15-0 --> 8.5.18-0
Proceed ([y]/n)? y
Fetching packages ...
sqlite-3.8.4.1 100% |################################| Time: 0:00:01 650.17 kB/s
tk-8.5.18-0.ta 100% |################################| Time: 0:00:02 717.50 kB/s
conda-env-2.1. 100% |################################| Time: 0:00:00 203.41 kB/s
nose-1.3.4-py2 100% |################################| Time: 0:00:00 359.41 kB/s
numpy-1.9.2-py 100% |################################| Time: 0:00:05 554.83 kB/s
requests-2.6.0 100% |################################| Time: 0:00:00 639.57 kB/s
setuptools-15. 100% |################################| Time: 0:00:00 597.81 kB/s
conda-3.10.1-p 100% |################################| Time: 0:00:00 466.21 kB/s
pip-6.1.1-py27 100% |################################| Time: 0:00:01 771.29 kB/s
scikit-learn-0 100% |################################| Time: 0:00:04 865.69 kB/s
Extracting packages ...
[ COMPLETE ] |##################################################| 100%
Unlinking packages ...
[ COMPLETE ] |##################################################| 100%
Linking packages ...
[ COMPLETE ] |##################################################| 100%
sh-3.2#
If there are problems with distribution1D or interpolationND these are problems with the crow libraries. These can be built with ./build_raven in the raven directory.
Also, sometimes if the build version of python is different from the run version, the libraries cannot be found. This can generally be fixed by:
./clean_raven
./build_raven
to rebuild the libraries.
If some of the data can not be retrieved from the internet (or git clone or update commands fail) possibly the proxy is not being used. See INL proxy
INL cluster compute nodes cannot access the internet, so ssh to the login node and run them from the login node.
On some systems (for example Ubuntu 16 and Windows 7) the default backend for matplotlib is set to Qt5Agg, which may not be properly set up after installing libraries through conda. A simple fix is to change the default backend to something more generic, for example TkAgg.
The matplotlib defaults file for the raven_libraries conda environment is usually located in ~/miniconda2/envs/raven_libraries/lib/python2.7/site-packages/matplotlib/mpl-data/matplotlibrc. If not, the location can be discovered via the following python commands (from here):
>>> import matplotlib
>>> matplotlib.matplotlib_fname()
'/home/foo/.config/matplotlib/matplotlibrc'Other common paths include ~/miniconda2/envs/raven_libraries/libs/python2.7/site-packages/matplotlib/mpl-data/matplotlibrc.
The keyword to change is "backend".
The problems using QT5Agg seem not to be present if matplotlib is updated to version 2.1.1 or later. RAVEN will likely move to this library version in a future update.
If RAVEN runs in general are failing with errors regarding Qt and Matplotlib, this change often resolves the problem.
Depending on security settings, the python binary libraries might need to be signed on MacOS. If this is the problem, the .so files are created, but they cannot be loaded. A self signed certificate can be created by following instructions in https://developer.apple.com/library/archive/documentation/Security/Conceptual/CodeSigningGuide/Procedures/Procedures.html and then the name of it can be added to the .ravenrc file (for example: RAVEN_SIGNATURE = John Doe ) (The .ravenrc file is created in the raven directory when scripts/establish_conda_env.sh --install is run) Then when build_raven is run it will sign the compiled binary files.
Currently (2022-Nov-21) our regression machines are running conda versions 4.8 to 4.12, other versions are not being tested with.
If there is an error like:
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /home/fred/miniconda3/envs/raven_libraries/lib/python3.10/site-packages/pandas/_libs/window/aggregations.cpython-310-x86_64-linux-gnu.so)
this is caused by libraries installed by conda not finding the correct conda installed libraries. In Linux this can be fixed by setting the LD_LIBRARY_PATH variable. Example (adjust the directory to match your system):
export LD_LIBRARY_PATH=/home/fred/miniconda3/envs/raven_libraries/lib/
or if the LD_LIBRARY_PATH is already defined:
LD_LIBRARY_PATH="/home/fred/miniconda3/envs/raven_libraries/lib/:$LD_LIBRARY_PATH"
Example of this error:
git clone https://github.com/idaholab/raven.git
Cloning into 'raven'...
remote: Enumerating objects: 113201, done.
remote: Counting objects: 100% (173/173), done.
remote: Compressing objects: 100% (101/101), done.
error: 6041 bytes of body are still expected88 MiB | 12.58 MiB/s
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
This is a network problem, and your network provider might be able to help get this fixed.
A work around is to do a shallow clone to decrease the amount of data transferred:
git clone https://github.com/idaholab/raven.git --depth 1
Later other commits can be added with (the below command grabs 100 more commits):
git fetch --deepen=100
or the complete history can be gotten (after the network issues are fixed):
git fetch --unshallow
If you get a network error like:
$ conda install numpy
Collecting package metadata (current_repodata.json): failed
CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://repo.anaconda.com/pkgs/main/osx-64/current_repodata.json>
Elapsed: -
An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
If your current network has https://www.anaconda.com blocked, please file
a support request with your network engineering team.
'https://repo.anaconda.com/pkgs/main/osx-64'
try running it with -v:
$ conda install -v numpy
Collecting package metadata (current_repodata.json): ...working... Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)'))': /pkgs/r/osx-64/current_repodata.json
(and the output will go on for a while).
Or if you get an error like this when running git:
$ git clone https://github.com/idaholab/raven.git
Cloning into 'raven'...
fatal: unable to access 'https://github.com/idaholab/raven.git/': SSL certificate problem: self signed certificate in certificate chain
This can be caused by a corporate network or a virus scanner imposing it self between your computer and github or conda servers. This can also be caused by someone trying to hack your computer, so make sure you figure out which it is.
The certificate used can be found by using a command like:
openssl s_client -connect github.com:443
or
openssl s_client -connect repo.anaconda.com:443
In order to remove these errors, the first thing you need to do is get the certificate to use. The corporate IT department or the virus scanner vendor should provide this to you.
If it is provided as a .crt file (for example CARoot.crt), it can be converted to a .pem file with the command like:
openssl x509 -inform der -in CARoot.crt -out CARoot.pem
Alternatively, the openssl s_client -connect command can be used to get the root certificate, but if someone is trying to hack your computer, this will work as well.
For conda, export the ```REQUESTS_CA_BUNDLE`` pointing to the pem file (example):
export REQUESTS_CA_BUNDLE=/home/fred/CARoot.pem
For git, set the sslCAInfo variable (example):
git config --global http.sslCAInfo /home/fred/CARoot.pem
(If you ever need to remove this, this will work:
git config --global --unset http.sslcainfo
)
This removes SSL checking which will stop ssl checking errors from happening.
These commands make it easy for anyone that can control the network you connect to to hack your computer, by disabling the ssl checking:
conda config --set ssl_verify false
git config --global http.sslVerify false
To reenable ssl checking:
conda config --set ssl_verify true
git config --global --unset http.sslVerify
To temporarily disable ssl checking for pip, use:
pip install --trusted-host pypi.org
followed by the rest of the pip install command.
If you encounter SSL certificate issues at INL, see https://servicenow.inl.gov/ess?id=kb_article_view&sys_kb_id=382baf5593de8a10c36eb7986cba1080