diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 1a01a02da..ad822dee2 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -1,7 +1,7 @@ - [Developer Guide](#developer-guide) - [Forking and Cloning](#forking-and-cloning) - [Install Prerequisites](#install-prerequisites) - - [JDK 14](#jdk-14) + - [Java](#java) - [Setup](#setup) - [Build](#build) - [Building from the command line](#building-from-the-command-line) @@ -10,6 +10,7 @@ - [Publishing](#publishing) - [Publishing to Maven Local](#publishing-to-maven-local) - [Generating artifacts](#generating-artifacts) + - [Adding Workflow Steps](#adding-workflow-steps) ## Developer Guide @@ -19,9 +20,11 @@ Fork this repository on GitHub, and clone locally with `git clone`. ### Install Prerequisites -#### JDK 14 +See [OpenSearch requirements](https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#install-prerequisites). -OpenSearch components build using Java 14 at a minimum. This means you must have a JDK 14 installed with the environment variable `JAVA_HOME` referencing the path to Java home for your JDK 14 installation, e.g. `JAVA_HOME=/usr/lib/jvm/jdk-14`. +#### Java + +Flow Framework code currently maintains compatibility with JDK 11. Other plugins may require newer Java versions if used. ### Setup @@ -60,6 +63,7 @@ merged to main, the workflow will create a backport PR to the `2.x` branch. ### Publishing #### Publishing to Maven Local + Run the below command to publish the artifacts to maven local. ```./gradlew publishToMavenLocal``` @@ -93,8 +97,17 @@ snapshots/ ├── maven-metadata.xml.sha1 ├── maven-metadata.xml.sha256 └── maven-metadata.xml.sha512 - - ``` + 1. Change the url from ``"https://aws.oss.sonatype.org/content/repositories/snapshots"`` to your local path and comment out the credentials under publishing/repositories in build.gradle. 2. Run ```./gradlew publishPluginZipPublicationToSnapshotsRepository```. + +### Adding Workflow Steps + +To add functionality to workflows, add new Workflow Steps to the [`org.opensearch.flowframework.workflow`](https://github.com/opensearch-project/flow-framework/tree/main/src/main/java/org/opensearch/flowframework/workflow) package. +1. Implement the [Workflow](https://github.com/opensearch-project/flow-framework/blob/main/src/main/java/org/opensearch/flowframework/workflow/WorkflowStep.java) interface. See existing steps for examples for input, output, and API execution. +2. Choose a unique name for the step which is not used by other steps. This will align with the `step_type` field in the templates and should be descriptive of what the step does. +3. Add a constructor and call it from the [WorkflowStepFactory](https://github.com/opensearch-project/flow-framework/blob/main/src/main/java/org/opensearch/flowframework/workflow/WorkflowStepFactory.java). +4. Add a configuration to the [`workflow-steps.json`](https://github.com/dbwiddis/flow-framework/blob/main/src/main/resources/mappings/workflow-steps.json) file specifying required inputs, outputs, required plugins, and optionally a different timeout than the default. +5. If your step provisions a resource that should be deprovisioned, create the corresponding step and add both steps to the [`WorkflowResources`](https://github.com/opensearch-project/flow-framework/blob/main/src/main/java/org/opensearch/flowframework/common/WorkflowResources.java) enum. +6. Write unit and integration tests. diff --git a/README.md b/README.md index 51bd59d8e..86510708b 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,14 @@ -## OpenSearch AI Flow Framework +## OpenSearch Flow Framework -This project is an OpenSearch plugin that enables builders to innovate AI apps on OpenSearch. +This project is an OpenSearch plugin that enables builders to innovate AI applications on OpenSearch. The current process of using ML offerings in OpenSearch, such as Semantic Search, requires users to handle complex setup and pre-processing tasks, and send verbose user queries, both of which can be time-consuming and error-prone. -The directional idea is to provide OpenSearch users with use case templates, which provide a compact description (e.g., JSON document). These templates would describe configurations for automated workflows such as Retrieval Augment Generation (RAG), AI connectors and other components that prime OpenSearch as a backend to leverage generative models—once primed, builders can query OpenSearch directly without building middleware logic to stitch together data flows and ML models. +We want to introduce our customers to a new no-code/low-code builder experience ([Backend RFC](https://github.com/opensearch-project/OpenSearch/issues/9213) and [Frontend RFC](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/4755)) that empowers users to compose AI-augmented query and ingestion flows, integrate ML models supported by ML-Commons, and streamline the OpenSearch app development experience through a drag-and-drop designer. The front end will help users create use case templates, which provide a compact description of configuration steps for automated workflows such as Retrieval Augment Generation (RAG), AI connectors and other components that prime OpenSearch as a backend to leverage generative models. Once primed, builders can query OpenSearch directly without building middleware logic to stitch together data flows and ML models. -See the [RFC](https://github.com/opensearch-project/OpenSearch/issues/9213) on the OpenSearch project for initial design discussions. +While the initial development has focused on ML use cases, the framework is generic and can be adapted to other use cases. + +See the [Development Plan](https://github.com/opensearch-project/flow-framework/issues/475) to view or comment on current incremental development priorities. ## Security