diff --git a/notebooks/community/model_garden/model_garden_pytorch_bart_large_cnn.ipynb b/notebooks/community/model_garden/model_garden_pytorch_bart_large_cnn.ipynb index 95010642b2..0790827ccd 100644 --- a/notebooks/community/model_garden/model_garden_pytorch_bart_large_cnn.ipynb +++ b/notebooks/community/model_garden/model_garden_pytorch_bart_large_cnn.ipynb @@ -4,11 +4,12 @@ "cell_type": "code", "execution_count": null, "metadata": { + "cellView": "form", "id": "7d9bbf86da5e" }, "outputs": [], "source": [ - "# Copyright 2023 Google LLC\n", + "# Copyright 2024 Google LLC\n", "#\n", "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", @@ -26,37 +27,29 @@ { "cell_type": "markdown", "metadata": { - "id": "2bd716bf3e39" + "id": "99c1c3fc2ca5" }, "source": [ "# Vertex AI Model Garden - Bart Large CNN\n", "\n", - "\n", - "
\n", - " \n", - " \"Colab Run in Colab\n", + "\n", + " \n", - " \n", - " \n", - "
\n", + " \n", + " \"Google
Run in Colab Enterprise\n", " \n", " 		\n", + " \n", " \n", - " \"GitHub\n", - " View on GitHub\n", + " \"GitHub
View on GitHub\n", " \n", " 		\n", - " \n", - " \"Vertex\n", - "Open in Vertex AI Workbench\n", - " \n", - " (a Python-3 CPU notebook is recommended)\n", - "
" + "
" ] }, { "cell_type": "markdown", "metadata": { - "id": "d8cd12648da4" + "id": "3de7470326a2" }, "source": [ "## Overview\n", @@ -76,7 +69,7 @@ "* Vertex AI\n", "* Cloud Storage\n", "\n", - "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage." + "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing), [Cloud Storage pricing](https://cloud.google.com/storage/pricing), and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage." ] }, { @@ -85,275 +78,229 @@ "id": "264c07757582" }, "source": [ - "## Setup environment\n", - "\n", - "**NOTE**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "d73ffa0c0b83" - }, - "source": [ - "### Colab only" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "2707b02ef5df" - }, - "outputs": [], - "source": [ - "!pip3 install --upgrade google-cloud-aiplatform" + "## Run the notebook" ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "b60a4d7100bf" + "cellView": "form", + "id": "ioensNKM8ned" }, "outputs": [], "source": [ - "from google.colab import auth as google_auth\n", + "# @title Setup Google Cloud project\n", "\n", - "google_auth.authenticate_user()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "0f826ff482a2" - }, - "source": [ - "### Setup Google Cloud project\n", + "# @markdown 1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n", "\n", - "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n", + "# @markdown 2. **[Optional]** [Create a Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for storing experiment outputs. Set the BUCKET_URI for the experiment environment. The specified Cloud Storage bucket (`BUCKET_URI`) should be located in the same region as where the notebook was launched. Note that a multi-region bucket (eg. \"us\") is not considered a match for a single region covered by the multi-region range (eg. \"us-central1\"). If not set, a unique GCS bucket will be created instead.\n", "\n", - "1. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n", + "BUCKET_URI = \"gs://\" # @param {type:\"string\"}\n", "\n", - "1. [Enable the Vertex AI API and Compute Engine API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com,compute_component).\n", + "# @markdown 3. **[Optional]** Set region. If not set, the region will be set automatically according to Colab Enterprise environment.\n", "\n", - "1. [Create a Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for storing experiment outputs." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "8958ebc71868" - }, - "source": [ - "Fill following variables for experiments environment:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "9db30f827a65" - }, - "outputs": [], - "source": [ - "# Cloud project id.\n", - "PROJECT_ID = \"\" # @param {type:\"string\"}\n", + "REGION = \"\" # @param {type:\"string\"}\n", "\n", - "# The region you want to launch jobs in.\n", - "REGION = \"us-central1\" # @param {type:\"string\"}\n", + "import datetime\n", + "import os\n", + "import uuid\n", "\n", - "# The Cloud Storage bucket for storing experiments output. Fill it without the 'gs://' prefix.\n", - "GCS_BUCKET = \"\" # @param {type:\"string\"}" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "92f16e22c20b" - }, - "source": [ - "Initialize Vertex AI API:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "1680c257acfb" - }, - "outputs": [], - "source": [ "from google.cloud import aiplatform\n", "\n", - "aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=GCS_BUCKET)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "6ca48b699d17" - }, - "source": [ - "### Define constants" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "de9882ea89ea" - }, - "outputs": [], - "source": [ - "# The pre-built serving docker image.\n", - "# The model artifacts are embedded within the container, except for model weights which will be downloaded during deployment.\n", - "SERVE_DOCKER_URI = \"us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-transformers-serve\"" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "10188266a5cd" - }, - "source": [ - "### Define common functions" + "# Get the default cloud project id.\n", + "PROJECT_ID = os.environ[\"GOOGLE_CLOUD_PROJECT\"]\n", + "\n", + "# Get the default region for launching jobs.\n", + "if not REGION:\n", + " REGION = os.environ[\"GOOGLE_CLOUD_REGION\"]\n", + "\n", + "# Enable the Vertex AI API and Compute Engine API, if not already.\n", + "print(\"Enabling Vertex AI API and Compute Engine API.\")\n", + "! gcloud services enable aiplatform.googleapis.com compute.googleapis.com\n", + "\n", + "# Cloud Storage bucket for storing the experiment artifacts.\n", + "# A unique GCS bucket will be created for the purpose of this notebook. If you\n", + "# prefer using your own GCS bucket, change the value yourself below.\n", + "now = datetime.datetime.now().strftime(\"%Y%m%d%H%M%S\")\n", + "BUCKET_NAME = \"/\".join(BUCKET_URI.split(\"/\")[:3])\n", + "\n", + "if BUCKET_URI is None or BUCKET_URI.strip() == \"\" or BUCKET_URI == \"gs://\":\n", + " BUCKET_URI = f\"gs://{PROJECT_ID}-tmp-{now}-{str(uuid.uuid4())[:4]}\"\n", + " BUCKET_NAME = \"/\".join(BUCKET_URI.split(\"/\")[:3])\n", + " ! gsutil mb -l {REGION} {BUCKET_URI}\n", + "else:\n", + " assert BUCKET_URI.startswith(\"gs://\"), \"BUCKET_URI must start with `gs://`.\"\n", + " shell_output = ! gsutil ls -Lb {BUCKET_NAME} | grep \"Location constraint:\" | sed \"s/Location constraint://\"\n", + " bucket_region = shell_output[0].strip().lower()\n", + " if bucket_region != REGION:\n", + " raise ValueError(\n", + " \"Bucket region %s is different from notebook region %s\"\n", + " % (bucket_region, REGION)\n", + " )\n", + "print(f\"Using this GCS Bucket: {BUCKET_URI}\")\n", + "\n", + "STAGING_BUCKET = os.path.join(BUCKET_URI, \"temporal\")\n", + "MODEL_BUCKET = os.path.join(BUCKET_URI, \"bart-large-cnn\")\n", + "\n", + "\n", + "# Initialize Vertex AI API.\n", + "print(\"Initializing Vertex AI API.\")\n", + "aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=STAGING_BUCKET)\n", + "\n", + "# Gets the default SERVICE_ACCOUNT.\n", + "shell_output = ! gcloud projects describe $PROJECT_ID\n", + "project_number = shell_output[-1].split(\":\")[1].strip().replace(\"'\", \"\")\n", + "SERVICE_ACCOUNT = f\"{project_number}-compute@developer.gserviceaccount.com\"\n", + "print(\"Using this default Service Account:\", SERVICE_ACCOUNT)\n", + "\n", + "\n", + "# Provision permissions to the SERVICE_ACCOUNT with the GCS bucket\n", + "! gsutil iam ch serviceAccount:{SERVICE_ACCOUNT}:roles/storage.admin $BUCKET_NAME\n", + "\n", + "! gcloud config set project $PROJECT_ID\n", + "! gcloud projects add-iam-policy-binding --no-user-output-enabled {PROJECT_ID} --member=serviceAccount:{SERVICE_ACCOUNT} --role=\"roles/storage.admin\"\n", + "! gcloud projects add-iam-policy-binding --no-user-output-enabled {PROJECT_ID} --member=serviceAccount:{SERVICE_ACCOUNT} --role=\"roles/aiplatform.user\"\n", + "\n", + "models, endpoints = {}, {}" ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "cac4478ae098" + "cellView": "form", + "id": "2707b02ef5df" }, "outputs": [], "source": [ - "def deploy_model(model_id, task):\n", - " \"\"\"Uploads and deploys the model to Vertex AI endpoint for prediction.\"\"\"\n", - " model_name = \"bart_large_cnn\"\n", + "# @title Deploy the model to Vertex for online predictions\n", + "\n", + "# @markdown This section uploads the model to Model Registry and deploys it on the Endpoint. It takes ~15 minutes to finish.\n", + "\n", + "MODEL_ID = \"facebook/bart-large-cnn\"\n", + "task = \"summarization\"\n", + "\n", + "# The pre-built serving docker image. It contains serving scripts and models.\n", + "SERVE_DOCKER_URI = \"us-docker.pkg.dev/deeplearning-platform-release/gcr.io/huggingface-pytorch-inference-cu121.2-2.transformers.4-41.ubuntu2204.py311\"\n", + "\n", + "\n", + "def deploy_model(\n", + " model_id, task, machine_type=\"g2-standard-8\", accelerator_type=\"NVIDIA_L4\"\n", + "):\n", + " \"\"\"Create a Vertex AI Endpoint and deploy the specified model to the endpoint.\"\"\"\n", + " model_name = model_id\n", + "\n", " endpoint = aiplatform.Endpoint.create(display_name=f\"{model_name}-endpoint\")\n", " serving_env = {\n", - " \"MODEL_ID\": model_id,\n", - " \"TASK\": task,\n", + " \"HF_MODEL_ID\": model_id,\n", + " \"HF_TASK\": task,\n", " \"DEPLOY_SOURCE\": \"notebook\",\n", " }\n", - " # If the model_id is a GCS path, use artifact_uri to pass it to serving docker.\n", - " artifact_uri = model_id if model_id.startswith(\"gs://\") else None\n", + "\n", " model = aiplatform.Model.upload(\n", " display_name=model_name,\n", " serving_container_image_uri=SERVE_DOCKER_URI,\n", " serving_container_ports=[7080],\n", - " serving_container_predict_route=\"/predictions/transformers_serving\",\n", + " serving_container_predict_route=\"/pred\",\n", " serving_container_health_route=\"/ping\",\n", " serving_container_environment_variables=serving_env,\n", - " artifact_uri=artifact_uri,\n", " )\n", + "\n", " model.deploy(\n", " endpoint=endpoint,\n", - " machine_type=\"n1-standard-8\",\n", - " accelerator_type=\"NVIDIA_TESLA_T4\",\n", + " machine_type=machine_type,\n", + " accelerator_type=accelerator_type,\n", " accelerator_count=1,\n", " deploy_request_timeout=1800,\n", + " service_account=SERVICE_ACCOUNT,\n", " )\n", - " return model, endpoint" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "d2d72ecdb8c9" - }, - "source": [ - "## Upload and deploy models" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "9448c5f545fa" - }, - "source": [ - "This section uploads the pre-trained model to Model Registry and deploys it on the Endpoint with 1 T4 GPU.\n", + " return model, endpoint\n", "\n", - "The model deployment step will take ~15 minutes to complete." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "b4b46c28d8b1" - }, - "outputs": [], - "source": [ - "model, endpoint = deploy_model(model_id=\"facebook/bart-large-cnn\", task=\"summarization\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "nTs3JBT3W_lJ" - }, - "source": [ - "## Send prediction request" + "\n", + "models[\"model\"], endpoints[\"endpoint\"] = deploy_model(model_id=MODEL_ID, task=task)\n", + "\n", + "print(\"endpoint_name:\", endpoints[\"endpoint\"].name)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "6be655247cb1" + "cellView": "form", + "id": "bb7adab99e41" }, "outputs": [], "source": [ - "ARTICLE = \"\"\" New York (CNN)When Liana Barrientos was 23 years old, she got married in Westchester County, New York.\n", - "A year later, she got married again in Westchester County, but to a different man and without divorcing her first husband.\n", - "Only 18 days after that marriage, she got hitched yet again. Then, Barrientos declared \"I do\" five more times, sometimes only within two weeks of each other.\n", - "In 2010, she married once more, this time in the Bronx. In an application for a marriage license, she stated it was her \"first and only\" marriage.\n", - "Barrientos, now 39, is facing two criminal counts of \"offering a false instrument for filing in the first degree,\" referring to her false statements on the\n", - "2010 marriage license application, according to court documents.\n", - "Prosecutors said the marriages were part of an immigration scam.\n", - "On Friday, she pleaded not guilty at State Supreme Court in the Bronx, according to her attorney, Christopher Wright, who declined to comment further.\n", - "After leaving court, Barrientos was arrested and charged with theft of service and criminal trespass for allegedly sneaking into the New York subway through an emergency exit, said Detective\n", - "Annette Markowski, a police spokeswoman. In total, Barrientos has been married 10 times, with nine of her marriages occurring between 1999 and 2002.\n", - "All occurred either in Westchester County, Long Island, New Jersey or the Bronx. She is believed to still be married to four men, and at one time, she was married to eight men at once, prosecutors say.\n", - "Prosecutors said the immigration scam involved some of her husbands, who filed for permanent residence status shortly after the marriages.\n", - "Any divorces happened only after such filings were approved. It was unclear whether any of the men will be prosecuted.\n", - "The case was referred to the Bronx District Attorney\\'s Office by Immigration and Customs Enforcement and the Department of Homeland Security\\'s\n", - "Investigation Division. Seven of the men are from so-called \"red-flagged\" countries, including Egypt, Turkey, Georgia, Pakistan and Mali.\n", - "Her eighth husband, Rashid Rajput, was deported in 2006 to his native Pakistan after an investigation by the Joint Terrorism Task Force.\n", - "If convicted, Barrientos faces up to four years in prison. Her next court appearance is scheduled for May 18.\n", - "\"\"\"\n", - "\n", - "instances = [\n", - " {\"text\": ARTICLE},\n", - "]\n", - "preds = endpoint.predict(instances=instances).predictions\n", + "# @title Predict\n", + "# @markdown Once deployment succeeds, you can send requests to the endpoint with input text.\n", + "\n", + "# @markdown This example uses the following input:\n", + "\n", + "# @markdown > Machine learning (ML) is a field of study in artificial\n", + "# @markdown > intelligence concerned with the development and study of\n", + "# @markdown > statistical algorithms that can learn from data and generalize to\n", + "# @markdown > unseen data and thus perform tasks without explicit instructions.\n", + "# @markdown > Recently, artificial neural networks have been able to surpass\n", + "# @markdown > many previous approaches in performance. ML finds application in\n", + "# @markdown > many fields, including natural language processing, computer\n", + "# @markdown > vision, speech recognition, email filtering, agriculture, and\n", + "# @markdown > medicine. When applied to business problems, it is known under the\n", + "# @markdown > name predictive analytics. Although not all machine learning is\n", + "# @markdown > statistically based, computational statistics is an important\n", + "# @markdown > source of the field's methods. The mathematical foundations of ML\n", + "# @markdown > are provided by mathematical optimization (mathematical\n", + "# @markdown > programming) methods. Data mining is a related (parallel) field of\n", + "# @markdown > study, focusing on exploratory data analysis (EDA) through\n", + "# @markdown > unsupervised learning. From a theoretical viewpoint, probably\n", + "# @markdown > approximately correct (PAC) learning provides a framework for\n", + "# @markdown > describing machine learning.\n", + "\n", + "# Loads an existing endpoint instance using the endpoint name:\n", + "# - Using `endpoint_name = endpoint.name` allows us to get the endpoint name of\n", + "# the endpoint `endpoint` created in the cell above.\n", + "# - Alternatively, you can set `endpoint_name = \"1234567890123456789\"` to load\n", + "# an existing endpoint with the ID 1234567890123456789.\n", + "\n", + "# You may uncomment the code below to load an existing endpoint.\n", + "# endpoint_name = \"\" # @param {type:\"string\"}\n", + "# aip_endpoint_name = (\n", + "# f\"projects/{PROJECT_ID}/locations/{REGION}/endpoints/{endpoint_name}\"\n", + "# )\n", + "# endpoint = aiplatform.Endpoint(aip_endpoint_name)\n", + "# print(\"Using this existing endpoint from a different session: {aip_endpoint_name}\")\n", + "\n", + "input = \"Machine learning (ML) is a field of study in artificial intelligence concerned with the development and study of statistical algorithms that can learn from data and generalize to unseen data and thus perform tasks without explicit instructions. Recently, artificial neural networks have been able to surpass many previous approaches in performance. ML finds application in many fields, including natural language processing, computer vision, speech recognition, email filtering, agriculture, and medicine. When applied to business problems, it is known under the name predictive analytics. Although not all machine learning is statistically based, computational statistics is an important source of the field's methods. The mathematical foundations of ML are provided by mathematical optimization (mathematical programming) methods. Data mining is a related (parallel) field of study, focusing on exploratory data analysis (EDA) through unsupervised learning. From a theoretical viewpoint, probably approximately correct (PAC) learning provides a framework for describing machine learning.\" # @param {type:\"string\"}\n", + "\n", + "instances = [input]\n", + "preds = endpoints[\"endpoint\"].predict(instances=instances).predictions\n", "print(preds)" ] }, - { - "cell_type": "markdown", - "metadata": { - "id": "xKj4wBQNWuHT" - }, - "source": [ - "## Clean up" - ] - }, { "cell_type": "code", "execution_count": null, "metadata": { - "id": "2ccf3714dbe9" + "cellView": "form", + "id": "6c460088b873" }, "outputs": [], "source": [ + "# @title Clean up resources\n", + "# @markdown Delete the experiment models and endpoints to recycle the resources\n", + "# @markdown and avoid unnecessary continuous charges that may incur.\n", + "\n", "# Undeploy model and delete endpoint.\n", - "endpoint.delete(force=True)\n", + "for endpoint in endpoints.values():\n", + " endpoint.delete(force=True)\n", "\n", "# Delete models.\n", - "model.delete()" + "for model in models.values():\n", + " model.delete()\n", + "\n", + "delete_bucket = False # @param {type:\"boolean\"}\n", + "if delete_bucket:\n", + " ! gsutil -m rm -r $BUCKET_NAME" ] } ],