Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(nvidia): persistence mode check based on NVML, do not rely on "nvidia-persistenced" binary #137

Merged
merged 1 commit into from
Oct 27, 2024

Conversation

gyuho
Copy link
Contributor

@gyuho gyuho commented Oct 26, 2024

Even if "nvidia-persistenced" is not running, GPU can have persistence mode enabled via NVML, thus we should not rely on nvidia-persistenced daemon running to decide whether the persistence mode is enabled or not.

Fix

Screenshot 2024-10-26 at 6 17 16 PM

Tested with 4090 machine

##################
NVML GPU-eeea5b9b-3ae5-3be8-ac2c-d26ab02b7899

✔ NVML persistence mode is enabled (nvidia-persistenced running false)
✔ NVML found no hw slowdown error

…idia-persistenced" binary

Even if "nvidia-persistenced" is not running, GPU can have persistence
mode enabled via NVML, thus we should not rely on nvidia-persistenced
daemon running to decide whether the persistence mode is enabled or not.

Signed-off-by: Gyuho Lee <[email protected]>
@gyuho gyuho self-assigned this Oct 26, 2024
@gyuho gyuho added this to the v0.0.6 milestone Oct 26, 2024
@gyuho gyuho merged commit a9d8b90 into main Oct 27, 2024
5 checks passed
@gyuho gyuho deleted the fix-persistence-mode branch October 27, 2024 05:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants