Skip to content

Latest commit

 

History

History
171 lines (103 loc) · 7.74 KB

README.md

File metadata and controls

171 lines (103 loc) · 7.74 KB
Raw

Code Generation

Code-generating LLMs are specialized AI models designed for the task of generating computer code. Such models undergo training with datasets that encompass repositories, specialized documentation, programming code, relevant web content, and other related data. They possess a deep understanding of various programming languages, coding patterns, and software development concepts. Code LLMs are engineered to assist developers and programmers. When these LLMs are seamlessly integrated into the developer's Integrated Development Environment (IDE), they possess a comprehensive understanding of the coding context, which includes elements such as comments, function names, and variable names. This contextual awareness empowers them to provide more refined and contextually relevant coding suggestions.

Capabilities of LLMs in Coding:

  • Code Generation: streamline coding through Code Generation, enabling non-programmers to describe tasks for code creation.
  • Code Completion: accelerate coding by suggesting contextually relevant snippets as developers type.
  • Code Translation and Modernization: translate and modernize code across multiple programming languages, aiding interoperability and updating legacy projects.
  • Code summarization: extract key insights from codebases, improving readability and developer productivity.
  • Code Refactoring: offer suggestions for code refactoring, enhancing code performance and efficiency.
  • AI-Assisted Testing: assist in creating test cases, ensuring code robustness and accelerating development cycles.
  • Error Detection and Debugging: detect errors in code and provide detailed descriptions and potential fixes, expediting debugging processes.

In this example, we present a Code Copilot application to showcase how code generation can be executed on the Intel Gaudi2 platform. This CodeGen use case involves code generation utilizing open source models such as "m-a-p/OpenCodeInterpreter-DS-6.7B", "deepseek-ai/deepseek-coder-33b-instruct" and Text Generation Inference on Intel Gaudi2.

CodeGen architecture shows below:

architecture

Environment Setup

To use 🤗 text-generation-inference on Intel Gaudi2, please follow these steps:

Prepare Gaudi Image

Getting started is straightforward with the official Docker container. Simply pull the image using:

docker pull ghcr.io/huggingface/tgi-gaudi:1.2.1

Alternatively, you can build the Docker image yourself with:

bash ./serving/tgi_gaudi/build_docker.sh

Launch TGI Gaudi Service

Launch a local server instance on 1 Gaudi card:

bash ./serving/tgi_gaudi/launch_tgi_service.sh

Launch a local server instance on 4 Gaudi cards:

bash ./serving/tgi_gaudi/launch_tgi_service.sh 4 9000 "deepseek-ai/deepseek-coder-33b-instruct"

Customize TGI Gaudi Service

The ./tgi_gaudi/launch_tgi_service.sh script accepts three parameters:

  • num_cards: The number of Gaudi cards to be utilized, ranging from 1 to 8. The default is set to 1.
  • port_number: The port number assigned to the TGI Gaudi endpoint, with the default being 8080.
  • model_name: The model name utilized for LLM, with the default set to "m-a-p/OpenCodeInterpreter-DS-6.7B".

You have the flexibility to customize these parameters according to your specific needs. Additionally, you can set the TGI Gaudi endpoint by exporting the environment variable TGI_ENDPOINT:

export TGI_ENDPOINT="http://xxx.xxx.xxx.xxx:8080"

Launch Copilot Docker

Build Copilot Docker Image (Optional)

cd codegen
bash ./build_docker.sh
cd ..

Launch Copilot Docker

docker run -it -e http_proxy=${http_proxy} -e https_proxy=${https_proxy} --net=host --ipc=host -v /var/run/docker.sock:/var/run/docker.sock intel/gen-ai-examples:copilot bash

Start Copilot Server

Start the Backend Service

Make sure TGI-Gaudi service is running and also make sure data is populated into Redis. Launch the backend service:

Please follow this link huggingface token to get the access token and export HUGGINGFACEHUB_API_TOKEN environment with the token.

export HUGGINGFACEHUB_API_TOKEN=<token>
nohup python server.py &

The Copilot backend defaults to listening on port 8000, but you can adjust the port number as needed.

Install Copilot VSCode extension from Plugin Marketplace

Install Neural Copilot in VSCode as below.

Install-screenshot

How to use

Service URL setting

Please adjust the service URL in the extension settings based on the endpoint of the code generation backend service.

Setting-screenshot Setting-screenshot

Customize

The Copilot enables users to input their corresponding sensitive information and tokens in the user settings according to their own needs. This customization enhances the accuracy and output content to better meet individual requirements.

Customize

Code Suggestion

To trigger inline completion, you'll need to type # {your keyword} (start with your programming language's comment keyword, like // in C++ and # in python). Make sure Inline Suggest is enabled from the VS Code Settings. For example:

code suggestion

To provide programmers with a smooth experience, the Copilot supports multiple ways to trigger inline code suggestions. If you are interested in the details, they are summarized as follows:

  • Generate code from single-line comments: The simplest way introduced before.
  • Generate code from consecutive single-line comments:

codegen from single-line comments

  • Generate code from multi-line comments, which will not be triggered until there is at least one space outside the multi-line comment):

codegen from multi-line comments

  • Automatically complete multi-line comments:

auto complete

Chat with AI assistant

You can start a conversation with the AI programming assistant by clicking on the robot icon in the plugin bar on the left:

icon

Then you can see the conversation window on the left, where you can chat with AI assistant:

dialog

There are 4 areas worth noting:

  • Enter and submit your question
  • Your previous questions
  • Answers from AI assistant (Code will be highlighted properly according to the programming language it is written in, also support streaming output)
  • Copy or replace code with one click (Note that you need to select the code in the editor first and then click "replace", otherwise the code will be inserted)

You can also select the code in the editor and ask AI assistant question about it. For example:

  • Select code

select code

  • Ask question and get answer

qna

SCRIPT USAGE NOTICE:  By downloading and using any script file included with the associated software package (such as files with .bat, .cmd, or .JS extensions, Docker files, or any other type of file that, when executed, automatically downloads and/or installs files onto your system) (the “Script File”), it is your obligation to review the Script File to understand what files (e.g.,  other software, AI models, AI Datasets) the Script File will download to your system (“Downloaded Files”). Furthermore, by downloading and using the Downloaded Files, even if they are installed through a silent install, you agree to any and all terms and conditions associated with such files, including but not limited to, license terms, notices, or disclaimers.