Bug: Runtime error pybind11::handle::inc_ref() PyGILState_Check() failure.

[Kiliankleemann]

Error message below (bottom):
INFO @ 21 Oct 2024 16:38:48: [107 MB]

Command line: hmmratac -i BED_files_sorted/9.bedpe -f BEDPE -n macs3_output/9

Random seed selected as: 10151

Use --hmm-type to select a Gaussian ('gaussian') or Poisson ('poisson') model for the hidden markov model in HMMRATAC. Default: 'gaussian'.

INFO @ 21 Oct 2024 16:38:48: [107 MB] #1 Read fragments from BEDPE file...
INFO @ 21 Oct 2024 16:38:49: [197 MB] 1000000 fragments parsed
INFO @ 21 Oct 2024 16:38:50: [213 MB] 2000000 fragments parsed
INFO @ 21 Oct 2024 16:38:51: [222 MB] 3000000 fragments parsed
INFO @ 21 Oct 2024 16:38:52: [226 MB] 4000000 fragments parsed
INFO @ 21 Oct 2024 16:38:54: [226 MB] 5000000 fragments parsed
INFO @ 21 Oct 2024 16:38:55: [227 MB] 6000000 fragments parsed
INFO @ 21 Oct 2024 16:38:56: [228 MB] 7000000 fragments parsed
INFO @ 21 Oct 2024 16:38:57: [229 MB] 8000000 fragments parsed
INFO @ 21 Oct 2024 16:38:58: [239 MB] 9000000 fragments parsed
INFO @ 21 Oct 2024 16:39:00: [245 MB] 10000000 fragments parsed
INFO @ 21 Oct 2024 16:39:01: [259 MB] 11000000 fragments parsed
INFO @ 21 Oct 2024 16:39:02: [263 MB] 12000000 fragments parsed
INFO @ 21 Oct 2024 16:39:03: [263 MB] 13000000 fragments parsed
INFO @ 21 Oct 2024 16:39:04: [273 MB] 14000000 fragments parsed
INFO @ 21 Oct 2024 16:39:06: [273 MB] 15000000 fragments parsed
INFO @ 21 Oct 2024 16:39:07: [279 MB] 16000000 fragments parsed
INFO @ 21 Oct 2024 16:39:08: [279 MB] 17000000 fragments parsed
INFO @ 21 Oct 2024 16:39:09: [289 MB] 18000000 fragments parsed
INFO @ 21 Oct 2024 16:39:10: [305 MB] 19000000 fragments parsed
INFO @ 21 Oct 2024 16:39:24: [358 MB] #2 Use EM algorithm to estimate means and stddevs of fragment lengths
INFO @ 21 Oct 2024 16:39:24: [358 MB] # for mono-, di-, and tri-nucleosomal signals...
INFO @ 21 Oct 2024 16:39:24: [359 MB] # A random seed 10151 has been used in the sampling function
INFO @ 21 Oct 2024 16:39:26: [428 MB] # Downsampled 1904083 fragments will be used for EM training...
INFO @ 21 Oct 2024 16:45:35: [428 MB] # Reached convergence after 14 iterations
INFO @ 21 Oct 2024 16:45:35: [428 MB] # The means and stddevs after EM:
INFO @ 21 Oct 2024 16:45:35: [428 MB] # short mono di tri
INFO @ 21 Oct 2024 16:45:35: [428 MB] # means: 50 210.1 382.4 611.7
INFO @ 21 Oct 2024 16:45:35: [428 MB] # stddevs: 20 45.8 45.7 101.8
INFO @ 21 Oct 2024 16:45:35: [428 MB] # Compute the weights for each fragment length for each of the four signal types
INFO @ 21 Oct 2024 16:45:42: [428 MB] # Generate short, mono-, di-, and tri-nucleosomal signals
INFO @ 21 Oct 2024 16:56:22: [928 MB] # Pile up all fragments
INFO @ 21 Oct 2024 16:56:43: [928 MB] # Convert pileup to fold-change over average signal
INFO @ 21 Oct 2024 16:56:49: [928 MB] #3 Look for training set from 19146841 fragments
INFO @ 21 Oct 2024 16:56:49: [928 MB] # Call peak above within fold-change range of 10 and 20.
INFO @ 21 Oct 2024 16:56:49: [928 MB] # The minimum length of the region is set as the average template/fragment length in the dataset: 592
INFO @ 21 Oct 2024 16:56:49: [928 MB] # The maximum gap to merge nearby significant regions is set as the flanking size to extend training regions: 1000
INFO @ 21 Oct 2024 16:56:53: [928 MB] # Total training regions called after applying the lower cutoff 10: 12930
INFO @ 21 Oct 2024 16:56:53: [928 MB] # Total training regions after filtering with upper cutoff 20: 3820
INFO @ 21 Oct 2024 16:56:53: [928 MB] # IMPORTANT
INFO @ 21 Oct 2024 16:56:53: [928 MB] # Please review the cutoff analysis result in macs3_output/9_cutoff_analysis.tsv to verify
INFO @ 21 Oct 2024 16:56:53: [928 MB] # if the choices of lower, upper and prescanning cutoff are appropriate.
INFO @ 21 Oct 2024 16:56:53: [928 MB] # Please read the message in the section 'Choices of cutoff values' by running
INFO @ 21 Oct 2024 16:56:53: [928 MB] # macs3 hmmratac -h for detail.
INFO @ 21 Oct 2024 16:56:53: [928 MB] # ****
INFO @ 21 Oct 2024 16:57:39: [928 MB] # We randomly pick 1000 regions for training
INFO @ 21 Oct 2024 16:57:39: [928 MB] # We expand the training regions with 1000 basepairs and merge overlap
INFO @ 21 Oct 2024 16:57:39: [928 MB] #4 Train Hidden Markov Model with Multivariate Gaussian Emission
INFO @ 21 Oct 2024 16:57:39: [928 MB] # Extract signals in training regions with bin size of 10
INFO @ 21 Oct 2024 16:57:50: [1116 MB] # Use Baum-Welch algorithm to train the HMM
pybind11::handle::inc_ref() is being called while the GIL is either not held or invalid. Please see https://pybind11.readthedocs.io/en/stable/advanced/misc.html#common-sources-of-global-interpreter-lock-errors for debugging advice.
If you are convinced there is no bug in your code, you can #define PYBIND11_NO_ASSERT_GIL_HELD_INCREF_DECREF to disable this check. In that case you have to ensure this #define is consistently used for all translation units linked into a given pybind11 extension, otherwise there will be ODR violations. The failing pybind11::handle::inc_ref() call was triggered on a numpy.ndarray object.
Traceback (most recent call last):
File "/Users/kiliankleemann/anaconda3/envs/macs3/bin/macs3", line 983, in
main()
File "/Users/kiliankleemann/anaconda3/envs/macs3/bin/macs3", line 97, in main
run( args )
File "/Users/kiliankleemann/anaconda3/envs/macs3/lib/python3.12/site-packages/MACS3/Commands/hmmratac_cmd.py", line 305, in run
hmm_model = hmm_training( training_data, training_data_lengths, random_seed = options.hmm_randomSeed, hmm_type = options.hmm_type )
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "MACS3/Signal/HMMR_HMM.pyx", line 60, in MACS3.Signal.HMMR_HMM.hmm_training
File "MACS3/Signal/HMMR_HMM.pyx", line 70, in MACS3.Signal.HMMR_HMM.hmm_training
File "/Users/kiliankleemann/anaconda3/envs/macs3/lib/python3.12/site-packages/hmmlearn/base.py", line 485, in fit
stats, curr_logprob = self._do_estep(X, lengths)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/kiliankleemann/anaconda3/envs/macs3/lib/python3.12/site-packages/hmmlearn/base.py", line 764, in _do_estep
lattice, logprob, posteriors, fwdlattice, bwdlattice = impl(sub_X)
^^^^^^^^^^^
File "/Users/kiliankleemann/anaconda3/envs/macs3/lib/python3.12/site-packages/hmmlearn/base.py", line 884, in _fit_log
log_prob, fwdlattice = _hmmc.forward_log(
^^^^^^^^^^^^^^^^^^
RuntimeError: pybind11::handle::inc_ref() PyGILState_Check() failure.

System (please complete the following information):

• OS: Mac OSX
• Python version: Python 3.11.4
• MACS Version macs3 3.0.1

[taoliu]

@Kiliankleemann It seems to be caused by dependencies of numpy which has a major version update to version 2. Could you try to upgrade macs3 to 3.0.2 and try again? You need to use pip (https://pypi.org/project/MACS3/) to upgrade it. Let me know if the problem still exists. The reason may be that in 3.0.1 version, we only set the version of numpy to be >= 1.24, and the conda system may install the newest numpy 2 which seems to not work well with another dependency -- hmmlearn.