deploy-llama-2-7b-vllm-on-amazon-sagemaker.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "05dbf1e1-03b7-4766-a2b1-632c4d0c7ccf",
   "metadata": {},
   "source": [
    "# Increase throughput for Llama-2-7b using Batching techniques on SageMaker LMI v5\n",
    "\n",
    "---\n",
    "이 노트북은 SageMaker LMI v5 컨테이너를 사용하여 Llama-2-7b-fp16-7b 대규모 언어 모델의 처리량을 증가시키기 위한 다양한 배치 기술을 탐구합니다. 이 예제에서는 LMI 컨테이너에 번들로 제공되는 DJLServing을 모델 서빙 솔루션으로 사용합니다. DJLServing은 프로그래밍 언어에 구애받지 않는 고성능 범용 모델 서빙 솔루션이며, Deep Java Library (DJL)에 의해 구동됩니다. DJL과 DJLServing에 대해 자세히 알아보려면 이 링크를 참조하세요. (https://docs.djl.ai/docs/serving/index.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6e4d479-b7c8-4fc6-86fc-151d1c2b2d40",
   "metadata": {},
   "source": [
    "이 노트북에서는 연속 배치를 위한 롤링 배치 기능과 페이지 주의를 제공하는 SageMaker LMI v5 컨테이너를 사용합니다. 여기서는 ml.g5.12xlarge 인스턴스에서 GPU에 걸쳐 TheBloke/Llama-2-7b-fp16-7b-dpo-v6 모델을 배포합니다."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73f91c7b-906c-4696-9c50-1eaa861550ff",
   "metadata": {},
   "source": [
    "# 1. 필요한 라이브러리 가져오기 및 SageMaker SDK 사용하여 세션 설정하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ada2cef-9b6d-4874-9af7-3b8a7747b942",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install sagemaker boto3 huggingface_hub --upgrade --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5a858f0-4af7-4c45-acaa-bab30ddf6e82",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "sess = sagemaker.Session()\n",
    "# sagemaker session bucket -> used for uploading data, models and logs\n",
    "# sagemaker will automatically create this bucket if it not exists\n",
    "sagemaker_session_bucket=None\n",
    "if sagemaker_session_bucket is None and sess is not None:\n",
    "    # set to default bucket if a bucket name is not given\n",
    "    sagemaker_session_bucket = sess.default_bucket()\n",
    "\n",
    "try:\n",
    "    role = sagemaker.get_execution_role()\n",
    "except ValueError:\n",
    "    iam = boto3.client('iam')\n",
    "    role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']\n",
    "\n",
    "sess = sagemaker.Session(default_bucket=sagemaker_session_bucket)\n",
    "\n",
    "print(f\"sagemaker role arn: {role}\")\n",
    "print(f\"sagemaker bucket: {sess.default_bucket()}\")\n",
    "print(f\"sagemaker session region: {sess.boto_region_name}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad1a8656-99b0-418a-837e-3d709b558877",
   "metadata": {},
   "source": [
    "# 2. Hugging Face LLM DLC 검색"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ebed6cd-d367-47d2-8974-5a0a4f9eb514",
   "metadata": {},
   "source": [
    "일반적인 Hugging Face 모델을 배포하는 것과 비교할 때, 우선 컨테이너 URI를 검색하고 이를 HuggingFaceModel 모델 클래스에 image_uri를 지정하여 제공해야 합니다. Amazon SageMaker에서 새로운 Hugging Face LLM DLC를 검색하려면, SageMaker SDK에서 제공하는 get_huggingface_llm_image_uri 메소드를 사용할 수 있습니다. 이 메소드를 통해 지정된 백엔드, 세션, 지역, 버전에 기반한 원하는 Hugging Face LLM DLC의 URI를 검색할 수 있습니다. 사용 가능한 버전은 여기에서 확인할 수 있습니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de4d0793-ddf6-4695-95f3-8a5d4877b9a8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.huggingface import get_huggingface_llm_image_uri\n",
    "from sagemaker import image_uris\n",
    "\n",
    "# retrieve the llm image uri\n",
    "llm_image = image_uris.retrieve(\n",
    "    framework=\"djl-deepspeed\", region=sess.boto_region_name, version=\"0.25.0\"\n",
    ")\n",
    "\n",
    "# print ecr image uri\n",
    "print(f\"llm image uri: {llm_image}\")\n",
    "\n",
    "# You can retrieve the image in this way:\n",
    "# llm_image = get_huggingface_llm_image_uri(\n",
    "#   backend=\"lmi\", # or huggingface\n",
    "#   region=sess.boto_region_name\n",
    "# )\n",
    "\n",
    "# # print ecr image uri\n",
    "# print(f\"llm image uri: {llm_image}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c4c34f1-9d1b-40ae-b473-05d554d877b9",
   "metadata": {},
   "source": [
    "### 허깅페이스에서 모델을 다운로드하고 Amazon S3에 모델 아티팩트를 업로드합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7f08068-c058-4ce8-a61c-da59cbdfa64b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from huggingface_hub import snapshot_download\n",
    "from pathlib import Path\n",
    "import os\n",
    "\n",
    "# - This will download the model into the current directory where ever the jupyter notebook is running\n",
    "local_model_path = Path(\".\")\n",
    "local_model_path.mkdir(exist_ok=True)\n",
    "model_name = \"TheBloke/Llama-2-7b-fp16\"\n",
    "# Only download pytorch checkpoint files\n",
    "allow_patterns = [\"*.json\", \"*.txt\", \"*.model\", \"*.safetensors\", \"*.bin\", \"*.chk\", \"*.pth\"]\n",
    "\n",
    "# - Leverage the snapshot library to donload the model since the model is stored in repository using LFS\n",
    "model_download_path = snapshot_download(\n",
    "    repo_id=model_name, cache_dir=local_model_path, allow_patterns=allow_patterns\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e7a48fc-e9f7-418a-9866-3cc88069cf5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# upload files from local to S3 location\n",
    "s3_model_prefix = \"hf-large-model-djl/Llama-2-7b-fp16/model\"  # folder within bucket where code artifact will go\n",
    "pretrained_model_location = sess.upload_data(path=model_download_path, key_prefix=s3_model_prefix)\n",
    "print(f\"Model uploaded to --- > {pretrained_model_location}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "052947d3-6188-4bf7-b3a8-fd685dc95aff",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 3. Amazon SageMaker에 Llama-2-7b-fp16 배포하기\n",
    "\n",
    "Amazon SageMaker에 파인튜닝한 모델을 배포하기 위해 `HuggingFaceModel` 모델 클래스를 생성하고, `hf_model_id`, `instance_type` 등을 포함한 엔드포인트 구성을 정의합니다. 여기서는 4개의 NVIDIA A10G GPU와 96GB의 GPU 메모리를 갖춘 `g5.12xlarge` 인스턴스 타입을 사용할 것입니다.\n",
    "\n",
    "먼저 serving.properties for Paged Attention Batching 파일을 생성합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b515fb3-1369-4f05-96fa-38d7b41a2008",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!rm -rf code_Llama-2-7b-fp16\n",
    "!mkdir -p code_Llama-2-7b-fp16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0ba1538-1ddf-4f1f-8d64-59fe5fcf371f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%writefile code_Llama-2-7b-fp16/serving.properties\n",
    "engine = MPI\n",
    "option.tensor_parallel_degree = 1\n",
    "option.rolling_batch = auto\n",
    "option.max_rolling_batch_size = 8\n",
    "option.model_loading_timeout = 3600\n",
    "option.model_id = {{model_id}}\n",
    "option.paged_attention = true\n",
    "option.trust_remote_code = true\n",
    "option.dtype = fp16\n",
    "option.enable_streaming=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac216196-e008-4e7b-b2ca-0fdac00252ad",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import jinja2\n",
    "from pathlib import Path\n",
    "\n",
    "jinja_env = jinja2.Environment()\n",
    "\n",
    "# we plug in the appropriate model location into our `serving.properties`\n",
    "template = jinja_env.from_string(Path(\"code_Llama-2-7b-fp16/serving.properties\").open().read())\n",
    "Path(\"code_Llama-2-7b-fp16/serving.properties\").open(\"w\").write(\n",
    "    template.render(model_id=pretrained_model_location)\n",
    ")\n",
    "!pygmentize code_Llama-2-7b-fp16/serving.properties | cat -n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d60076b-5b14-49a1-88cd-238a890c8480",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!rm model.tar.gz\n",
    "!tar czvf model.tar.gz code_Llama-2-7b-fp16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c6055d8-7db0-4fc7-a2e8-2e98ab6cfc7b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "s3_code_prefix = \"hf-large-model-djl/Llama-2-7b-fp16/code\"  # folder within bucket where code artifact will go\n",
    "s3_code_artifact = sess.upload_data(\"model.tar.gz\", sess.default_bucket(), s3_code_prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9be8527-8aba-4d8f-b13e-ff00c94d1348",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sm_client = boto3.client(\"sagemaker\")\n",
    "\n",
    "from sagemaker.utils import name_from_base\n",
    "\n",
    "model_name = name_from_base(f\"Llama-2-7b-fp16-mpi\")\n",
    "print(model_name)\n",
    "\n",
    "create_model_response = sm_client.create_model(\n",
    "    ModelName=model_name,\n",
    "    ExecutionRoleArn=role,\n",
    "    PrimaryContainer={\"Image\": llm_image, \"ModelDataUrl\": s3_code_artifact},\n",
    ")\n",
    "model_arn = create_model_response[\"ModelArn\"]\n",
    "\n",
    "print(f\"Created Model: {model_arn}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0692ff5a-0ac3-45cf-9564-e5ae3085e7a7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "endpoint_config_name = f\"{model_name}-config\"\n",
    "endpoint_name = f\"{model_name}-endpoint\"\n",
    "\n",
    "endpoint_config_response = sm_client.create_endpoint_config(\n",
    "    EndpointConfigName=endpoint_config_name,\n",
    "    ProductionVariants=[\n",
    "        {\n",
    "            \"VariantName\": \"variant1\",\n",
    "            \"ModelName\": model_name,\n",
    "            \"InstanceType\": \"ml.g5.2xlarge\",\n",
    "            \"InitialInstanceCount\": 1,\n",
    "            \"ModelDataDownloadTimeoutInSeconds\": 900,\n",
    "            \"ContainerStartupHealthCheckTimeoutInSeconds\": 900,\n",
    "        },\n",
    "    ],\n",
    ")\n",
    "endpoint_config_response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4841444-2e09-4357-bf73-009ae1bf8fd4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "create_endpoint_response = sm_client.create_endpoint(\n",
    "    EndpointName=f\"{endpoint_name}\", EndpointConfigName=endpoint_config_name\n",
    ")\n",
    "print(f\"Created Endpoint: {create_endpoint_response['EndpointArn']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca6d8b3c-e25e-44a8-b0d9-fff3cd175053",
   "metadata": {},
   "source": [
    "### 시간이 걸릴 수 있으니 잠시 기다립니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c820c65-93e5-402a-a912-e26819bc45f0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "resp = sm_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "status = resp[\"EndpointStatus\"]\n",
    "print(\"Status: \" + status)\n",
    "\n",
    "while status == \"Creating\":\n",
    "    time.sleep(60)\n",
    "    resp = sm_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "    status = resp[\"EndpointStatus\"]\n",
    "    print(\"Status: \" + status)\n",
    "\n",
    "print(\"Arn: \" + resp[\"EndpointArn\"])\n",
    "print(\"Status: \" + status)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38e9b22d-e32a-4548-8a1a-d9e238a9fc16",
   "metadata": {},
   "source": [
    "## 4. 배포된 모델에서 추론 요청하기\n",
    "\n",
    "이 코드를 사용하여 자신의 입력으로 모델을 테스트할 수도 있습니다.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19625a1b-91bd-4a78-8b78-d577c7c595aa",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker import serializers, deserializers\n",
    "\n",
    "endpoint_name=\"Llama-2-7b-fp16-mpi-XXX-endpoint\"\n",
    "\n",
    "llm = sagemaker.Predictor(\n",
    "    endpoint_name=endpoint_name,\n",
    "    sagemaker_session=sess,\n",
    "    serializer=serializers.JSONSerializer(),\n",
    "    deserializer=deserializers.JSONDeserializer(),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cd3a6fe-d044-4513-bbdc-918a002841d8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "llm.predict(\n",
    "    {\n",
    "        \"inputs\": [\"Amazon.com is the best \", \"Large model inference is\"],\n",
    "        \"parameters\": {\"min_new_tokens\": 20, \"max_new_tokens\": 200, \"temperature\": 0.8},\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce11b7b8-918c-4a2d-b67e-d8fe38e6422f",
   "metadata": {},
   "source": [
    "## 5. Benchmarking throughput for Paged Attention Batching"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7baaa18f-f4a5-41ef-9a39-83117bfa5277",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://github.com/frankfliu/junkyard/releases/download/v0.3.1/awscurl\n",
    "!chmod +x awscurl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4558b970-4ef1-48e8-96e2-68504e81afb0",
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf dyn_prompts\n",
    "!mkdir dyn_prompts\n",
    "!echo '{\"inputs\":\"The diamondback terrapin or simply terrapin is a species of turtle native to the brackish coastal tidal marshes of the\",\"parameters\":{\"min_new_tokens\":128, \"max_new_tokens\":128, \"do_sample\":true}}' > dyn_prompts/dyn_prompt1.txt\n",
    "!echo '{\"inputs\":\"Write a program to add two numbers in python\",\"parameters\":{\"min_new_tokens\":128, \"max_new_tokens\":128, \"do_sample\":true}}' > dyn_prompts/dyn_prompt2.txt\n",
    "!echo '{\"inputs\":\"Generate a list of ten titles for my book. The book is about my travels around the world, experiencing different cultures and cuisines, meeting many different personalities and finally settling in a remote village in the himalayas\",\"parameters\":{\"min_new_tokens\":128, \"max_new_tokens\":128, \"do_sample\":true}}' > dyn_prompts/dyn_prompt3.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "174e8f27-c088-426b-b05c-d64299bfdc8f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!apt -y update\n",
    "!apt -y -qq install default-jre wget "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b4ff504-6254-43ee-a941-c3d2a6024072",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!TOKENIZER=TheBloke/Llama-2-7b-fp16 AWS_REGION=us-east-1 ./awscurl --region us-east-1 -c 10 -N 10 -X POST \"https://runtime.sagemaker.us-east-1.amazonaws.com/endpoints/Llama-2-7b-fp16-mpi-XXX/invocations\" --connect-timeout 60   -H \"Content-Type: application/json\" --dataset dyn_prompts -t -n sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5c7a0fd-e440-4657-91df-0c3d29fdcc90",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.c5.2xlarge",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}