Llama2/Llama2-7b/LMI/llama2-7b-multi-lora-adapters-sagemaker.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a729a7fc-a1b6-4832-9155-d240ccd8ecc0",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Serving LoRA-based Llama 2 adapters with high performance on SageMaker \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2958bcf-767f-4cd7-826e-963f91565876",
   "metadata": {},
   "source": [
    "This notebook will demonstrate how you can deploy multiple fine-tuned LoRA adapters with a single base model copy on SageMaker using the DJL Serving Large Model Inference DLC. LoRA (Low Rank Adapters) is a powerful technique for fine-tuning large language models. This technique significantly reduces the number of trainable parameters compared to traditional fine-tuning while achieving comparable or superior performance. You can learn more about the LoRA technique in this paper.\n",
    "\n",
    "A major benefit of LoRA is that the fine-tuned adapters can easily be added to and removed from the base model, which makes switching adapters pretty cheap and viable at runtime. In this notebook we will show how you can deploy a SageMaker endpoint with a single base model and multiple LoRA adapters, and change adapters for different requests.\n",
    "\n",
    "Since LoRA adapters are much smaller than the size of a base model (can realistically be 100x-1000x smaller), we can deploy an endpoint with a single base model and multiple LoRA adapters using much less hardware than deploying an equivalent number of fully fine-tuned models.\n",
    "\n",
    "In this notebook, we deploy the [llama2-7B](https://huggingface.co/TheBloke/Llama-2-7b-fp16) as the base model and 3 LoRA adapters fine tuned for a specific language on SageMaker as shown below by leveraging the [SageMaker Large Model Inference Container](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#large-model-inference-containers).\n",
    "\n",
    "![Multiple adapters for Llama 2 as a base model](Usecase-adapter-basemodel.png)\n",
    "\n",
    "The LMI container offers the out-of-box integration with SageMaker for hosting multiple LoRA adapters with higher performance (low latency and high throughput) using the [vLLM](https://docs.vllm.ai/en/latest/models/lora.html) library that uses [S-LORA](https://github.com/S-LoRA/S-LoRA) and [Punica](https://arxiv.org/pdf/2310.18547.pdf). S-LoRA stores all adapters in the main memory and fetches the adapters used by the currently running queries to the GPU memory. To efficiently use the GPU memory and reduce fragmentation, S-LoRA proposes Unified Paging. Unified Paging uses a unified memory pool to manage dynamic adapter weights with different ranks and KV cache tensors with varying sequence lengths. Additionally, S-LoRA employs a novel tensor parallelism strategy and highly optimized custom CUDA kernels for heterogeneous batching of LoRA computation. Collectively, these features enable S-LoRA to serve thousands of LoRA adapters on a single GPU or across multiple GPUs with a small overhead.\n",
    "\n",
    "Below diagram shows the Multi LoRA-Adapter serving stack of LMI container on SageMaker\n",
    "![Multi LoRA-Adapter serving stack of LMI container on SageMaker](LoRA-LMI-SageMaker.png)\n",
    "\n",
    "#### Can we teach a new language to the Large Model ?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37ffcfa8-02c5-4492-af5c-fdcf60f369db",
   "metadata": {},
   "source": [
    "# License agreement\n",
    " - View license information https://huggingface.co/meta-llama before using the model.\n",
    " - This notebook is a sample notebook and not intended for production use. Please refer to the licence at https://github.com/aws/mit-0. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3a133db-f99f-4e46-8fe1-1a5af6a846a1",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Install, import the required libraries; set some variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8dd6a36f-ee95-453d-a127-c8a7de6a026d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install sagemaker boto3 huggingface_hub awscli --upgrade --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf0c89f4-679c-4557-b95d-1d954c15a020",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import jinja2\n",
    "from sagemaker import image_uris\n",
    "import boto3\n",
    "import os\n",
    "import time\n",
    "import json\n",
    "from pathlib import Path\n",
    "from huggingface_hub import snapshot_download"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "561b79ba-7354-4d40-87ce-dc5813092576",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "role = sagemaker.get_execution_role()  # execution role for the endpoint\n",
    "sess = sagemaker.session.Session()  # sagemaker session for interacting with different AWS APIs\n",
    "bucket = sess.default_bucket()  # bucket to house artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f68b5181-d018-4564-9762-fa8770a9672f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_bucket = sess.default_bucket()  # bucket to house model artifacts\n",
    "s3_code_prefix = \"hf-large-model-djl/multi-lora/Llama-2-7b-fp16/code\"  # folder within bucket where model/code artifacts will go\n",
    "region = sess._region_name\n",
    "account_id = sess.account_id()\n",
    "\n",
    "s3_client = boto3.client(\"s3\")\n",
    "sm_client = boto3.client(\"sagemaker\")\n",
    "smr_client = boto3.client(\"sagemaker-runtime\")\n",
    "\n",
    "jinja_env = jinja2.Environment()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1c28a75-9e5c-4be7-8ac2-f3c9f69cbb10",
   "metadata": {},
   "source": [
    "We will be deploying an endpoint with 3 LoRA adapters. These are the models we will be using:\n",
    "\n",
    "Base Model: https://huggingface.co/TheBloke/Llama-2-7B-Chat-fp16\n",
    "LoRA Fine Tuned Adapter 1: https://huggingface.co/UnderstandLing/llama-2-7b-chat-ru\n",
    "LoRA Fine Tuned Adapter 2: https://huggingface.co/UnderstandLing/llama-2-7b-chat-es\n",
    "LoRA Fine Tuned Adapter 3: https://huggingface.co/UnderstandLing/llama-2-7b-chat-fr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "558d3f19-173e-460c-9be3-e54a7dcf0b0c",
   "metadata": {
    "tags": []
   },
   "source": [
    "The core structure to cover here is the model directory. We include both the base model and LoRA adapters in the model directory like this:\n",
    "\n",
    "```\n",
    "|- model_dir\n",
    "    |- adapters/\n",
    "        |--- <adapter_1>/\n",
    "        |--- <adapter_2>/\n",
    "        |--- ...\n",
    "        |--- <adapter_n>/\n",
    "    |- serving.properties\n",
    "    |- model.py (optional)\n",
    "\n",
    "```\n",
    "\n",
    "It is also possible to have model files located in a separate s3 bucket by specifying that location using an s3 `option.model_id` in the serving.properties. In this case, the adapters directory can be located either alongside the `serving.properties` or alongside the model files in s3.\n",
    "\n",
    "Each of the adapters in the `adapters` directory contains the LoRA adapter artifacts. Typically there are two files: `adapter_model.bin` and `adapter_config.json` which are the adapter weights and adapter configuration respectively. These are typically obtained from the Peft library via the `PeftModel.save_pretrained()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a44a1b1c-d3eb-426a-95ce-c74783e7ddc8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!rm -rf lora-multi-adapter\n",
    "!mkdir -p lora-multi-adapter/adapters\n",
    "!echo \"Lora Multi Adapter Model\" > lora-multi-adapter/README.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9d8c8fc-1640-4f07-899a-e71667a7173f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "snapshot_download(\"UnderstandLing/llama-2-7b-chat-ru\", local_dir=\"lora-multi-adapter/adapters/ru\", local_dir_use_symlinks=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fec6d2fa-d5be-41d9-ae0e-2432e24f8ea7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "snapshot_download(\"UnderstandLing/llama-2-7b-chat-es\", local_dir=\"lora-multi-adapter/adapters/es\", local_dir_use_symlinks=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8d540e8-1873-485d-a46a-ac64778aeea8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "snapshot_download(\"UnderstandLing/llama-2-7b-chat-fr\", local_dir=\"lora-multi-adapter/adapters/fr\", local_dir_use_symlinks=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1e803e2-40af-4c49-b65c-ff3bc244d621",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!rm -f model.tar.gz\n",
    "!rm -rf lora-multi-adapter/.ipynb_checkpoints\n",
    "!tar czvf model.tar.gz -C lora-multi-adapter ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f13c1d4a-29b3-4f87-848f-a3854cee227d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "s3_code_artifact_accelerate = sess.upload_data(\"model.tar.gz\", model_bucket, s3_code_prefix)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "346f8270-69e1-47a0-ad6e-076da0f09e28",
   "metadata": {},
   "source": [
    "### Select the appropriate configuration parameters and container¶"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d559783-8b78-4c24-8220-82e75c323d51",
   "metadata": {},
   "source": [
    "To optimize the deployment of Large Language Models (LLMs); one needs to choose the appropriate model partitioning framework, optimal batching technique, batching size, tensor parallelism degree, etc. The choice of a particular configuration depends on the usecase.\n",
    "\n",
    "Hence, based on the usecase, you need to:\n",
    "1. set the configuration parameters for the container.\n",
    "2. select the appropriate container image to be used for inference."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e11eddcc-d301-4acc-86b5-1612501aaa39",
   "metadata": {},
   "source": [
    "### Set the configuration parameters using environment variables\n",
    "1. `SERVING_LOAD_MODELS` - specifies the engine that will be used for this workload. In this case we'll be hosting a model using the **Python** engine.\n",
    "\n",
    "2. `OPTION_MODEL_ID`: Set this to the URI of the Amazon S3 bucket that contains the model. When this is set, the container leverages [s5cmd](https://github.com/peak/s5cmd) to download the model from s3. This enables faster deployments by utilizing optimized approach within the DJL inference container to transfer the model from S3 into the hosting instance.\n",
    "If you want to download the model from huggingface.co, you can set `OPTION_MODEL_ID` to the model id of a pre-trained model hosted inside a model repository on huggingface.co (https://huggingface.co/models). The container uses this model id to download the corresponding model repository on huggingface.co.\n",
    "\n",
    "3. `OPTION_TENSOR_PARALLEL_DEGREE`: Set to the number of GPU devices over which DeepSpeed needs to partition the model. This parameter also controls the number of workers per model which will be started up when DJL serving runs. In this example we use the `ml.g5.12xlarge` instance that has 4 GPUs; hence this is set to 4.\n",
    "\n",
    "4. `OPTION_ROLLING_BATCH`: This parameter enables the use of a particular batching technique for continuous or iteration level batching to enable merging multiple concurrent requests that arrive at different times for inference.\n",
    "In scenarios that involves open ended generation and chatbots, there is a need for having a high throughput. [vLLM](https://arxiv.org/pdf/2309.06180.pdf) is a fast LLM inference and serving framework that uses techniques like PagedAttention and continuous batching to improve the throughput. Hence, we set the `rolling_batch` parameter to `vllm`. When using `vllm`, you can also use some [additional parameters](https://github.com/deepjavalibrary/djl-serving/blob/master/serving/docs/lmi/configurations_large_model_inference_containers.md#vllm).\n",
    "\n",
    "5. `OPTION_MAX_ROLLING_BATCH_SIZE`: The maximum number of concurrent requests to be used in a batch by the model server for inference. Clients can still send more requests to the endpoint, they will be queued.\n",
    "\n",
    "6. `OPTION_ENABLE_LORA`: This config enables support for LoRA adapters. Default: false.\n",
    "\n",
    "7. `OPTION_MAX_LORAS`: This config determines the maximum number of LoRA adapters that can be run at once. Allocates more GPU memory for those adapters. Default: 4\n",
    "\n",
    "8. `OPTION_MAX_LORA_RANK`: This config determines the maximum rank allowed for a LoRA adapter. Setting a larger value will enable more adapters at a greater memory usage cost. Default: 16\n",
    "\n",
    "9. `OPTION_LORA_EXTRA_VOCAD_SIZE`: This config determines the maximum additional vocabulary that can be added through a LoRA adapter. Default: 256\n",
    "\n",
    "10. `OPTION_MAX_CPU_LORAS`: This config determines the maximum number of LoRA adapters to cache in memory. All others will be evicted to disk. Default: None\n",
    "\n",
    "\n",
    "For more information on the available options, please refer to the [DJL Serving - SageMaker Large Model Inference Configurations](https://github.com/deepjavalibrary/djl-serving/blob/master/serving/docs/lmi/configurations_large_model_inference_containers.md)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88c92745-edd2-4187-8b19-b05b0a3c5b5b",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Select the relevant Large Model Inference container\n",
    "SageMaker offers optimized [large model inference containers](https://github.com/aws/deep-learning-containers/blob/master/available_images.md#large-model-inference-containers) that contains different frameworks for model parallelism enabling inference of LLMs on multiple GPUs.\n",
    "\n",
    "In this scenario, since we are leveraging `vllm` as the batching technique, we leverage the `deepspeed` container that has frameworks like deepspeed, vllm, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68716eed-94eb-4f70-88ea-92d15707bc8b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "deepspeed_image_uri = image_uris.retrieve(\n",
    "    framework=\"djl-deepspeed\", \n",
    "    region=sess.boto_session.region_name, \n",
    "    version=\"0.27.0\"\n",
    ")\n",
    "\n",
    "env_generation = {\"HUGGINGFACE_HUB_CACHE\": \"/tmp\",\n",
    "                  \"TRANSFORMERS_CACHE\": \"/tmp\",\n",
    "                  \"SERVING_LOAD_MODELS\": \"test::Python=/opt/ml/model\",\n",
    "                  \"OPTION_MODEL_ID\": \"TheBloke/Llama-2-7B-Chat-fp16\",\n",
    "                  \"OPTION_TRUST_REMOTE_CODE\": \"true\",\n",
    "                  \"OPTION_TENSOR_PARALLEL_DEGREE\": \"max\",\n",
    "                  \"OPTION_ROLLING_BATCH\": \"vllm\",\n",
    "                  \"OPTION_MAX_ROLLING_BATCH_SIZE\": \"32\",\n",
    "                  \"OPTION_DTYPE\": \"fp16\",\n",
    "                  \"OPTION_ENABLE_LORA\": \"true\",\n",
    "                  \"OPTION_GPU_MEMORY_UTILIZATION\": \"0.8\",\n",
    "                  \"OPTION_MAX_LORA_RANK\": \"64\",\n",
    "                  \"OPTION_MAX_CPU_LORAS\": \"4\"\n",
    "                 }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "858e2fbc-30aa-4b89-8c2f-8f1a0438c0f0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# - Select the appropriate environment variable which will tune the deployment server.\n",
    "env = env_generation # use this in case it is 'generation' task \n",
    "# - now we select the appropriate container \n",
    "inference_image_uri = deepspeed_image_uri # use this in case it is 'generation' task \n",
    "#inference_image_uri = trtllm_image_uri # enable this in case your use case is summarization ( high input and medium output sizes ) \n",
    "\n",
    "print(f\"Environment variables are ---- > {env}\")\n",
    "print(f\"Image going to be used is ---- > {inference_image_uri}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1150973-4e6c-4c8e-9d35-23fe50eb90e6",
   "metadata": {},
   "source": [
    "To create the end point the steps are:\n",
    "- Create the Model using the inference image container\n",
    "\n",
    "- Create the endpoint config using the following key parameters\n",
    "\n",
    "In this notebook we leverage the boto3 SDK. You can also use the [SageMaker SDK](https://sagemaker.readthedocs.io/en/stable/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7166e11e-53ea-4888-add5-82711e2d9cf0",
   "metadata": {},
   "source": [
    "### Create the Model\n",
    "Leverage the `inference_image_uri` to create a model object. We will leverage the Least routing algorithim -- [Least Routing Algorithim](https://aws.amazon.com/blogs/machine-learning/minimize-real-time-inference-latency-by-using-amazon-sagemaker-routing-strategies/). This innovation from sagemnaker has shown to reduce latency by 10% or more when we have multiple instances configured to serve the endpoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "784e2b93-783f-411b-9ff9-2b3436e4bb1b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_name = sagemaker.utils.name_from_base(\"lmi-llama2-7b\")\n",
    "print(model_name)\n",
    "\n",
    "create_model_response = sm_client.create_model(\n",
    "    ModelName=model_name,\n",
    "    ExecutionRoleArn=role,\n",
    "    PrimaryContainer={\n",
    "        \"Image\": inference_image_uri,\n",
    "        \"Environment\": env,\n",
    "        \"ModelDataUrl\": s3_code_artifact_accelerate,\n",
    "\n",
    "    }\n",
    ")\n",
    "model_arn = create_model_response[\"ModelArn\"]\n",
    "\n",
    "print(f\"Created Model: {model_arn}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "287be57e-2765-4384-becd-b4e2825a1025",
   "metadata": {},
   "source": [
    "### Create an endpoint config\n",
    "Create an endpoint configuration using the appropriate instance type. Set the `ContainerStartupHealthCheckTimeoutInSeconds` to account for the time taken to download the LLM weights from S3 or the model hub; and the time taken to load the model on the GPUs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d9b9db0-ae2a-4e77-84ab-6b7ca5593a0c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "endpoint_config_name = f\"{model_name}-config\"\n",
    "endpoint_name = f\"{model_name}-endpoint\"\n",
    "\n",
    "endpoint_config_response = sm_client.create_endpoint_config(\n",
    "    EndpointConfigName=endpoint_config_name,\n",
    "    ProductionVariants=[\n",
    "        {\n",
    "            \"VariantName\": \"variant1\",\n",
    "            \"ModelName\": model_name,\n",
    "            \"InstanceType\": \"ml.g5.12xlarge\",\n",
    "            \"InitialInstanceCount\": 1,\n",
    "            # \"ModelDataDownloadTimeoutInSeconds\": 2400,\n",
    "            \"ContainerStartupHealthCheckTimeoutInSeconds\": 500,\n",
    "            \"RoutingConfig\": {\n",
    "                'RoutingStrategy': 'LEAST_OUTSTANDING_REQUESTS'\n",
    "            },\n",
    "        },\n",
    "    ],\n",
    ")\n",
    "endpoint_config_response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c0534b2-aa1c-4cdd-9ac9-0f4466ee5b22",
   "metadata": {},
   "source": [
    "### Create an endpoint using the model and endpoint config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "caaee3e7-784a-42fb-bf5c-d2fb5eb113c0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "create_endpoint_response = sm_client.create_endpoint(\n",
    "    EndpointName=f\"{endpoint_name}\", EndpointConfigName=endpoint_config_name\n",
    ")\n",
    "print(f\"Created Endpoint: {create_endpoint_response['EndpointArn']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db0d2aaf-4a88-4cb2-8467-e1ce677689eb",
   "metadata": {},
   "source": [
    "#### This step can take ~15 mins or longer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1545e376-48e2-4b8b-ac83-cdb8fa8cd63d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "resp = sm_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "status = resp[\"EndpointStatus\"]\n",
    "print(\"Status: \" + status)\n",
    "\n",
    "while status == \"Creating\":\n",
    "    time.sleep(60)\n",
    "    resp = sm_client.describe_endpoint(EndpointName=endpoint_name)\n",
    "    status = resp[\"EndpointStatus\"]\n",
    "    print(\"Status: \" + status)\n",
    "\n",
    "print(\"Arn: \" + resp[\"EndpointArn\"])\n",
    "print(\"Status: \" + status)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fce7be01-f2c5-47fd-bf09-9454764d7e47",
   "metadata": {},
   "source": [
    "### Invoke the endpoint with a sample prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41f2a079-08d7-4ddf-b6f4-68f17b390f42",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "response_model = smr_client.invoke_endpoint(\n",
    "    EndpointName=endpoint_name,\n",
    "    Body=json.dumps({\"inputs\": [\"Piensa en una excusa creativa para decir que no necesito ir a la fiesta.\"],\n",
    "                     \"adapters\": [\"es\"]}),\n",
    "    ContentType=\"application/json\",\n",
    ")\n",
    "\n",
    "response_model[\"Body\"].read().decode(\"utf8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "317a8135-a522-45c5-8b6b-adec54737f7b",
   "metadata": {},
   "source": [
    "## Clean up the environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18e7f0b2-2d78-452b-89ba-c92d2178d924",
   "metadata": {},
   "outputs": [],
   "source": [
    "sm_client.delete_endpoint(EndpointName=endpoint_name)\n",
    "sm_client.delete_endpoint_config(EndpointConfigName=endpoint_config_name)\n",
    "sm_client.delete_model(ModelName=model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29acba11-80fd-4e2b-b4e7-984e644f8309",
   "metadata": {},
   "source": [
    "#### Resource:\n",
    "- [Deep Learning containers for large model inference](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints-large-model-dlc.html)\n",
    "- [Deep Java Library for Large Model Inference](https://docs.djl.ai/docs/serving/serving/docs/large_model_inference.html)"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}