Importing torch crashed when metacall is imported first.

[iripoll]

🐛 Bug Report

Importing torch crashed when metacall is imported first.

Expected Behavior

Load torch silently.

Current Behavior

ipython and juputer-notebook crashes when importing torch.

Steps to Reproduce

$ ipython3 
Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from metacall import metacall

In [2]: import torch
Segmentation fault (core dumped)

Versions:

• torch: 2.0.1+cu117
• MetaCall: 0.7.4

Context (Environment)

Detailed Description

Possible Implementation

[viferga]

@iripoll Thank you so much for your report.

This is a limitation related to metacall that cannot run outside of metacall. This problem basically happens because once it's run outside metacall, it initializes two instances of python. The solution of this has been investigated and the proposal is to detect if it's being run under an existing runtime and then relink the loaders on the fly to the process instead of loading the loader with the libpython dependency.

Here is the issue described in more detail: #231

The same problem happens here with NodeJS: #31

I have investigated this in python and it happens also due to openssl but in this case is due to different versions of crypto library which have a dependency to openssl.

Our distributable mitigates the issue but it's not ideal, eventually we will solve this completely for all runtimes. We only need a bit of magic.

I am gonna keep the issue open so we can have more reference and cases where this happens for the future.

Meanwhile we also have an integration with Jupyter, anybody needs to run metacall on top of it: https://github.com/metacall/jupyter-kernel

[ChinmayK0607]

This happened with me as well when importing metacall on a colab notebook.