Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CUDA error while running "model.fit" #66

Open
NekoClaviS opened this issue May 9, 2024 · 0 comments
Open

CUDA error while running "model.fit" #66

NekoClaviS opened this issue May 9, 2024 · 0 comments

Comments

@NekoClaviS
Copy link

I am facing some cupy cuda error while running the following code:

CUPY_DUMP_CUDA_SOURCE_ON_ERROR =1
CUPY_DISABLE_JITIFY_CACHE = 1
model.fit(min_iter=30, max_iter=35)
model.plot_convergence()

The above code runs for 1 iteration and then gives the following error:

6 errors detected in the compilation of "cupy_jitify_exercise".

RuntimeError                              Traceback (most recent call last)
Cell In[80], line 3
      1 CUPY_DUMP_CUDA_SOURCE_ON_ERROR =1
      2 CUPY_DISABLE_JITIFY_CACHE = 1
----> 3 model.fit(min_iter=30, max_iter=35)
      4 model.plot_convergence()

File ~/anaconda3/envs/scenvi/lib/python3.9/site-packages/SEACells/core.py:587, in SEACells.fit(self, max_iter, min_iter, initial_archetypes)
    585 if max_iter < min_iter:
    586     raise ValueError("The maximum number of iterations specified is lower than the minimum number of iterations specified.")
--> 587 self._fit(max_iter=max_iter, min_iter=min_iter, initial_archetypes=initial_archetypes, initial_assignments=None)

File ~/anaconda3/envs/scenvi/lib/python3.9/site-packages/SEACells/core.py:544, in SEACells._fit(self, max_iter, min_iter, initial_archetypes, initial_assignments)
    532 def _fit(self, max_iter: int = 50, min_iter:int=10, initial_archetypes=None, initial_assignments=None):
    533     """
    534     Compute archetypes and loadings given kernel matrix K. Iteratively updates A and B matrices until maximum
    535     number of iterations or convergence has been achieved.
   (...)
    542 
    543     """
--> 544     self.initialize(initial_archetypes=initial_archetypes, initial_assignments=initial_assignments)
    546     converged = False
    547     n_iter = 0
...
File cupy/cuda/jitify.pyx:192, in cupy.cuda.jitify._init_cupy_headers_from_scratch()

File cupy/cuda/jitify.pyx:264, in cupy.cuda.jitify.jitify()

RuntimeError: Runtime compilation failed

I have tried multiple things and have even tried rebuilding my anaconda venv, but unable to fix this, it would be really helpful if someone could help with the same.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant