- Developer Guide
Welcome to the opensearch-api-specification
developer guide! Glad you want to contribute. Here are the things you need to know while getting started!
Fork opensearch-api-specification repository to your GitHub account and clone it to your local machine. Whenever you're drafting a change, create a new branch for the change on your fork instead of on the upstream repository.
You will also need Java Development Kit (JDK) 17 or later to build the project. In your terminal, run the following command to build the project:
./gradlew build
This command generates API specs for Smithy and also converts them to OpenAPI specs. The specs can be found at:
- Smithy specs:
build/smithyprojections/opensearch-api-specification/full/model
- OpenAPI specs:
build/smithyprojections/opensearch-api-specification/full/openapi
To format the Smithy model files, use
./gradlew spotlessCheck
./gradlew spotlessApply
Popular IDEs for Smithy models include Visual Studio Code and IntelliJ, and they both have plugins that improve the editing experience for this project.
The OpenSearch API is composed of over 300 operations. These operations are grouped into API actions based on the functionality they provide (This grouping is done through the @xOperationGroup
Smithy trait). Each API action is later translated to an API method in each OpenSearch client. For example:
- The
cat.health
action will turn intoclient.cat.health()
- While the
index
action will turn intoclient.index()
This grouping influences the file structure of the Smithy models:
- Operations of
cat.health
action will be defined inmodel/cat/health
folder - Operations of
index
action will be defined inmodel/_global/index
folder
Each action folder contains 2 files
operations.smithy
defines all the operations of that action.structures.smithy
defines the input and output structures of said operations.
The path and querystring parameters are often reused across multiple operations regardless of the action they belong to. These parameters are defined in the root folder, model/
, and grouped by data-type in files of common_<data_type>.smithy
format.
Overall, the file structure of the Smithy models looks like this:
model
├── _global
│ ├── index
│ │ ├── operations.smithy
│ │ └── structures.smithy
│ └── search
│ ├── operations.smithy
│ └── structures.smithy
├── cat
│ └── health
│ ├── operations.smithy
│ └── structures.smithy
│
├── common_strings.smithy
└── common_enums.smithy
As mentioned in the previous section, each API action is composed of multiple operations that are defined in the same operations.smithy
file. The search
action, for example, is consisted of 4 operations:
GET /_search
POST /_search
GET /{index}/_search
POST /{index}/_search
To group these operations together in the search
action, we mark them with the @xOperationGroup
trait with the same value of search
. Note that this trait tells the client generators that these operations serve identical purpose and should be grouped together in the same API method. The xOperationGroup
trait also tells the generators the namespace and the name of the API method. For example, operations with xOperationGroup
trait value of indicies.create
will result in client.indices.create()
method to be generated.
We name each operation using the following format [ActionName]_[HttpVerb]_[PathParameters]
- ActionName: The name of the action. CamelCase and without the
.
character. E.g.search -> Search
,cat.health -> CatHealth
. - HttpVerb: The HTTP verb of the operation. E.g.
Get
,Post
,Put
,Delete
. In actions where all operations share the same HTTP verb, we omit the verb from the operation name. - PathParameters: This part is prefixed with
With
and is followed by the names of the path parameters. E.g.WithIndex
,WithId
, andWithIndexId
. This part can be omitted if the operation does not have any path parameters, or if all operations of the action share the same path parameters.
The search
action mentioned above will have the following operations:
Search_Get
Search_Post
Search_Get_WithIndex
Search_Post_WithIndex
Operations of the same API action share:
- Identical Output structure
- Similar Input structure:
- Identical set of querystring parameters
- Identical schema of the request body, if any
- Only differ in the path parameters
Due to these characteristics, these operations share the same output structure, and their input structures reuse the same querystring parameters and request body schema. The search
action, for example, will have the following input and output structures:
Search_Output
Search_Get_Input
Search_Post_Input
Search_Get_WithIndex_Input
Search_Post_WithIndex_Input
These structures are defined in the structures.smithy
file along with the shared querystring parameters and request body schema. The search
action's structures.smithy
file will look like this:
@mixin
structure Search_QueryParams {
...
}
structure Search_BodyParams {
...
}
@input
structure Search_Get_Input with [Search_QueryParams] {
}
@input
structure Search_Post_Input with [Search_QueryParams] {
@httpPayload
content: Search_BodyParams,
}
@input
structure Search_Get_WithIndex_Input with [Search_QueryParams] {
@required
@httpLabel
index: PathIndices,
}
@input
structure Search_Post_WithIndex_Input with [Search_QueryParams] {
@required
@httpLabel
index: PathIndices,
@httpPayload
content: Search_BodyParams,
}
structure Search_Output {
_scroll_id: String,
took: Long,
timed_out: Boolean,
_shards: ShardStatistics,
hits: HitsMetadata
}
Note that all input structures utilize the Search_QueryParams
mixin, and The Search_BodyParams
structure is used as @httpPayload
for the both POST operations as seen in Search_Post_Input
and Search_Post_WithIndex_Input
.
The bodies of request and response are also defined inside the structures.smithy
file. The request body is a member (usually named content
) of the input structure and MUST be accompanied by the @httpPayload
trait. It is also often accompanied by the @required
trait since most operations that accept a request body also require it. The response body, on the other hand, is the output structure itself.
@input
structure Operation_Input {
@required
@httpPayload
content: Search_BodyParams,
}
structure Request_Body {
// Request body members are defined here
}
structure Operation_Output {
// Response body members are defined here
}
Common parameters that are used across OpenSearch namespaces are defined in the root folder, model/
, and grouped by data-type in files of common_<data_type>.smithy
format. There are a few things to note when defining global common parameters:
- All path parameters should be prefixed with
Path
likePathIndex
andPathDocumentID
. - Smithy doesn't support enum or list as path parameters. We, therefore, have to define such parameters as string and use
x-data-type
vendor extension to denote their actual types (More on this in the traits section). - Parameters of type
time
are defined asstring
and has@pattern("^([0-9]+)(?:d|h|m|s|ms|micros|nanos)$")
trait to denote that they are in the format of<number><unit>
. E.g.1d
,2h
,3m
,4s
,5ms
,6micros
,7nanos
. We usex-data-type: "time"
vendor extension for this type. - Path parameters that are defined as strings must be accompanied by a
@pattern
trait and should be default to^[^_][\\d\\w-*]*$
to signify that they are not allowed to start with_
to avoid URI Conflict errors. - The
@documentation
,@default
, and@deprecation
traits can later be overridden by the operations that use these parameters.
Common parameters that are used within a namespace, especially namespaces representing plugins like security
are defined in the namespace folder (e.g. model/security
)
We use Smithy traits extensively in this project to work around some limitations of Smithy and to deal with some quirks of the OpenSearch API. Here are some of the traits that you should be aware of:
@suppress(["HttpMethodSemantics.UnexpectedPayload"])
: Used in DELETE operations with request body to suppress the UnexpectedPayload error.@suppress(["HttpUriConflict"])
: Used to suppress the HttpUriConflict error that is thrown when two operations have conflicting URI. Unfortunately, this happens a lot in OpenSearch API. When in doubt, add this trait to the operation.@pattern("^(?!_|template|query|field|point|clear|usage|stats|hot|reload|painless)")
: Required for most Path parameters to avoid URI Conflict errors. This is often used in tandem with the @suppress trait above. To denote the actual pattern of the parameter, usex-string-pattern
vendor extension.@readonly
: Should accompany most GET operations to denote that they are read-only.@idempotent
: Should accompany most PUT operations to denote that they are idempotent.
This repository includes several custom Smithy traits that map to OpenAPI Specification Extensions to fill in any metadata not directly supported by Smithy or OpenAPI. These traits are used to add the following metadata:
@xOperationGroup("{namespace}.{operation}")
: Used to group operations into API actions.@xVersionAdded("{version}")
: OpenSearch version when the operation/parameter was added.@xVersionDeprecated("{version}")
: OpenSearch version when the operation/parameter was deprecated.@xDeprecationDescription("{description}")
: Reason for deprecation and guidance on how to prepare for the next major version.@xSerialize("bulk")
: Denotes that the request body should be serialized as bulk data.@xDataType("{type}")
: Denotes the actual data-type of the parameters. This extension is used where a certain data-type is not supported by Smithy/OpenAPI (liketime
), or not supported in a certain context (likeenum
andlist
as path parameters).@xEnumOptions(["{opt1}", "{opt2}", ...])
: List of options for anenum
path parameter.@xOverloadedParam("{param}")
: Denotes that the parameter is overloaded with another parameter. This is used in the/_nodes/{node_id}
operation where you can also treat{node_id}
as{metric}
. Future operations should avoid this situation because it is bad API design. See Client Generator Guide for more info.@xIgnorable(true)
: Denotes that the operation should be ignored by the client generator. This is used in operation groups where some operations have been replaced by newer ones, but we still keep them in the specs because the server still supports them.
@xOperationGroup("search")
@xVersionAdded("1.0")
@suppress(["HttpUriConflict"])
@http(method: "POST", uri: "/{index}/_search")
@documentation("Returns results matching a query.")
operation Search_Post_WithIndex {
input: Search_Post_WithIndex_Input,
output: Search_Output
}
@xDataType("list")
@xEnumOptions(["settings", "os", "process", "jvm", "thread_pool", "transport", "http", "plugins", "ingest"])
@pattern("^(?!_|template|query|field|point|clear|usage|stats|hot|reload|painless)")
@documentation("Comma-separated list of metrics you wish returned. Leave empty to return all.")
string PathNodesInfoMetric
@xDataType("time")
@pattern("^([0-9]+)(?:d|h|m|s|ms|micros|nanos)$")
@documentation("The maximum time to wait for wait_for_metadata_version before timing out.")
string WaitForTimeout
Once you've finished with the model API, follow the steps below to create a test-case.
Let's suppose we have test-cases for put mapping and search api at first. Structure of the test folder's project tree:
test
├── scripts
└── model
├── _global
│ └── search
│ ├── hooks.js
│ └── OpenSearchModel.json
└── indices
└── put_mapping
├── hooks.js
└── OpenSearchModel.json
We'd want to include the Index-Aliases API now. The project-tree structure will be as follows:
test
├── scripts
└── model
├── _global
│ └── search
│ ├── hooks.js
│ └── OpenSearchModel.json
└── indices
├── put_mapping
│ ├── hooks.js
│ └── OpenSearchModel.json
└── aliases
├── hooks.js
└── OpenSearchModel.json
Two files must be defined:
- OpenSearchModel.js: This is a json file that includes the API model's test-case.
- The steps to create this file are listed below.
- Move to the project-directory.
- Run
cd test/scripts
. - Run
python operation-filter.py --operation <operation-id_1,operation-id_2> --output <complete-path>
. In case of the Index-aliases API, for examplepython operation-filter.py --operation PostAliases --output /Users/xxx-xxx/Desktop/
. - When the preceding step is completed successfully, a file named
model.openapi.json
will be generated in the defined directory. Copy the contents of the file into theOpenSearchModel.json
file.
- hooks.js: This file contains the API model's setup and teardown procedures.
NOTE:
- The arguments
--operation
and--output
are necessary. - For the
--output
parameter, provide the full directory path.
References: If you're having trouble while writing API test cases, check out the Index Aliases API.
The procedures outlined here will assist you in ensuring that the API model accurately represents the OpenAPI specification while testing it against the API's backend implementation. To do so, follow the steps below.
Following the instructions below will allow you to test the API documentation locally.
- In Docker go-to Preferences > Resources, set RAM to at least 4 GB.
- Move to project directory then run
cd test/
. - Install all node-modules using
npm install
. - Install all python dependencies using
pipenv install --system
. - Run docker using
docker-compose up -d
. - Wait for around 1 minute (for opensearch domain to be operational).
- Run
cd scripts/
.
We are ready with the setup now, for finally testing our API implementation use below commands:
- To test API implementation on default endpoint and all APIs.
- Run
python driver-code.py
- To test API implementation on default endpoint and specific API.
- Run
python driver-code.py --testname <test_name>
.
- To test all the APIs implementation with custom OpenSearch service endpoint.
- Run
python driver-code.py --endpoint <ENDPOINT_NAME> --user <USERNAME>:<PASSWORD>
.
- To test API implementation with custom OpenSearch service endpoint and specific APIs.
- Run
python driver-code.py --endpoint <ENDPOINT_NAME> --user <USERNAME>:<PASSWORD> --path <TEST_DIRECTORY>
.
Arguments supported while testing are mentioned below:
- --endpoint: (String) To specific the custom OpenSearch service URL for testing.
- --user: (String) To specify the username and password associated with the endpoint used.
- --path: (String) To specify the directory path of specific test to be tested.
- --testname: (String) To specify the name of API to be tested if not provided then all the tests are run.
- --testpass: (Boolean) When this option is set to True, a table of passed test cases will be printed as well. (By default, only the table for failed test-cases is printed.)
NOTE: Due to Ubuntu security updates, the version of Ubuntu mentioned in the CI workflow file may not be compatible with the Continuous Integration framework.