COSMADS: Composing SMArt Data Services through Large Language Models

This repository provides the code and resources for replicating the experiments from "Composing Smart Data Services in Shop Floors through Large Language Models", presented at the 22nd International Conference on Service-Oriented Computing (ICSOC 2024)

Prerequisites

• Miniconda
• OpenAI API Key

Setup

• Create a virtual environment and install the dependencies

conda create -n pyllm python=3.9
conda activate pyllm
pip install -r requirements.txt

• Create a .env file in the root directory of the project and add the following line

OPENAI_API_KEY=<YOUR_OPENAI_API_KEY>

Usage

• Define the query in the json file.
  As an example:

  "q5": {
      "query": "Please provide a table for the upcoming 30 cardboard pieces processed by the diecutter with ID 7, detailing (i) how many cardboard pieces are defect-free and (ii) how many contain defects.",
  }

• In the main file, specify the <query_number> to be executed.
  As an example:

  ...
  if __name__ == "__main__":
      q = "q5"
      ...

• Run the LLM:

  cd src
  python main.py

• The LLM will generate a temp_pipeline.py file with the Python pipeline leveraging the proper data services to generate the requested information.
  Given the Example, the LLM will generate a schema as follows.

  +----+--------------------+---------------------+
  |    |   no_defects_count |   with_errors_count |
  |----+--------------------+---------------------|
  |  0 |                 17 |                  13 |
  +----+--------------------+---------------------+

How to replicate the experiments

To run the experiments, execute the following command:

cd src
python run_evaluation.py

The script will create different .csv in evaluation folder containing the results of the run and computed metrics.

Experiments results

evaluation folder contains the results of the experiments:

• COSMADS:
  • evaluation results
  • metrics
• COSMADS (w/o similar pipelines):
  • evaluation results
  • metrics
• COSMADS (w/o pipelines):
  • evaluation results
  • metrics
• GitHub Copilot:
  • evaluation results
  • metrics