Added GPU support for Google Cloud #2
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- check_machine_type_availability() verifies the machine is available in the zone - check_gpu_model_support() verifies that tha machine and gpu model are compatible - check_gpu_enabled() verifies that GPU_ENABLED=true if GPU_MODEL is populated
- Created src/infractl/deploy/gcp/main.py - Moved check_* functions to src/infractl/deploy/gcp/main.py - Made icl/jupyterhub module support 'intel' or 'nvidia'
- Check/validate fucntions from gke.sh moved to src/infractl/deploy/gcp/main.py - Built new mutli-stage image for GPU profile image which uses nvidia/cuda:12.2.2-base-ubuntu22.04 as the base and adds pbchekin/icl-jupyterhub:0.0.21 changes - Changed GKE_GPU_DRIVER_VERSION environment variable default to "LATEST" - Added outputs to terraform modules for visibility and ease of future debugging - Modified terraform/icl module to dynamically set selected GPU image with jupyterhub_gpu_profile_image
- Added shared_gpu variable added to terraform/gcp and terraform/gcp/icl-cluster - Created new conditional module in terraform/gcp/icl-cluster dependent on shared_gpu variable value - Modified pool names to reflect exclusive vs shared GPU modes - Added node_count and gpu_count variables to easily allow future addition of multi-node deployments - Changed var.jupyterhub_extra_resource_limits from map(string) to string - Removed default value for var.jupyterhub_extra_resource_limits
- Added default value for jupyterhub_extra_resource_limits - Fixed subprocess.run calls using shell - Removed unused $GPU_ENABLED parameter from gke.sh call to infractl.deploy.gcp.main - isort, black, and pyline changes
…to enhance flexibility in specifying the type of GPU
- Trailing lines added - Uneeded whitepspace trimmed - subprocess import in main.py changed and reordered - Unintentional ray downgrade reverted - Duplicate variable declaration removed from gke.sh - Added GKE_GPU_DRIVER_VERSION description to gke.sh help output - Removed print lines from subprocess.CalledProcessError exceptions
Co-authored-by: Pavel Chekin <[email protected]>
Co-authored-by: Pavel Chekin <[email protected]>
Co-authored-by: Vadim Musin <[email protected]>
|
@kwasd please take a look before merging. |
Closed
Merged
- Added firewall-rule-bastion-ports module - Added additional variables to /terraform/gcp/variables.tf for bastion-host - Added generate_bastion_key function to create public SSH key when CREATE_BASTION="true" - Added two new environment variables related to bastion creation - Added function check BASTION_SOURCE_RANGES exists and -neq "" if CREATE_BASTION=true
…nce installtion methods will vary across environments.
kwasd
reviewed
Jun 12, 2024
scripts/deploy/aws.sh
Outdated
| @@ -91,6 +91,7 @@ function x1_terraform_args() { | |||
| -var jupyterhub_extra_resource_limits="${JUPYTERHUB_EXTRA_RESOURCE_LIMITS}" | |||
| -var gpu_enabled="${GPU_ENABLED}" | |||
| -var gpu_type="${GPU_TYPE}" | |||
| -var deployment_type="aws" | |||
There was a problem hiding this comment.
I think it may be better to introduce something like nvidia_gpu_operator_enabled instead, if we need provider-specific behavior. This could be passed from each provider's top-level script. So it will keep modules provider-agnostic.
There was a problem hiding this comment.
Agreed, this variable has been changed to enable_nvidia_operator. Let me know if AWS deployments with and without GPU work with the new change.
…nvidia_operator" - Changed some ENV default values from string to boolean to better align with what TF is expecting - Fixed bastion_name variable - Updated terraform/icl/main.tf to use updated "enable_nvidia_operator" variable
|
I've checked the AWS part, LGTM! |
kwasd
approved these changes
Jul 9, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.