From dc6d01a65e7eed6820da765127097b75857f4538 Mon Sep 17 00:00:00 2001 From: otto-ifak <87818460+otto-ifak@users.noreply.github.com> Date: Tue, 12 Nov 2024 11:01:24 +0100 Subject: [PATCH] Improve README (#72) --- README.md | 130 ++++++++++++++++++++++++++++---------------- bin/check_readme.py | 66 ++++++++++++++++++++++ bin/run_tests.sh | 2 + doc/api.md | 43 +++++++++++++++ doc/file.md | 15 +++++ 5 files changed, 208 insertions(+), 48 deletions(-) create mode 100755 bin/check_readme.py create mode 100644 doc/api.md create mode 100644 doc/file.md diff --git a/README.md b/README.md index 17b90ff..98fbe93 100644 --- a/README.md +++ b/README.md @@ -11,14 +11,16 @@ The tools in this repository offer measures to validate compliance of AAS implem You can install the AAS Test Engines via pip: + ```sh python -m pip install --upgrade aas_test_engines ``` ## Command Line Interface -You may want to invoke the test tools using the simplified command line interface: +You may want to invoke the test tools using the command line interface: + ```sh # Check file aas_test_engines check_file test.aasx @@ -30,56 +32,85 @@ aas_test_engines check_server https://localhost https://admin-shell.io/aas/API/3 # Generate test data aas_test_engines generate_files output_dir -# Alternative output formats +# Alternative output formats (work for all commands) aas_test_engines check_file test.aasx --output html > output.html aas_test_engines check_file test.aasx --output json > output.json ``` -## Supported Versions and Suites - -By default, the Test Engines test against the latest version 3.0 (file and api). -For v3.0 api testing, the following suites are defined: - - - `Asset Administration Shell API` - - `Submodel API` - - `Serialization API` - - `AASX File Server API` - - `Asset Administration Shell Registry API` - - `Submodel Registry API` - - `Asset Administration Shell Repository API` - - `Submodel Repository API` - - `Concept Description Repository API` - - `Asset Administration Shell Basic Discovery API` - - `Description API` - - `Asset Administration Shell Service Specification` - - `Submodel Service Specification` - - `AASX File Server Service Specification` - - `Asset Administration Shell Registry Service Specification` - - `Submodel Registry Service Specification` - - `Discovery Service Specification` - - `Asset Administration Shell Repository Service Specification` - - `Submodel Repository Service Specification` - - `ConceptDescription Repository Service Specification` - - `https://admin-shell.io/aas/API/3/0/AssetAdministrationShellServiceSpecification/SSP-001` - - `https://admin-shell.io/aas/API/3/0/AssetAdministrationShellServiceSpecification/SSP-002` - - `https://admin-shell.io/aas/API/3/0/SubmodelServiceSpecification/SSP-001` - - `https://admin-shell.io/aas/API/3/0/SubmodelServiceSpecification/SSP-002` - - `https://admin-shell.io/aas/API/3/0/SubmodelServiceSpecification/SSP-003` - - `https://admin-shell.io/aas/API/3/0/AasxFileServerServiceSpecification/SSP-001` - - `https://admin-shell.io/aas/API/3/0/AssetAdministrationShellRegistryServiceSpecification/SSP-001` - - `https://admin-shell.io/aas/API/3/0/AssetAdministrationShellRegistryServiceSpecification/SSP-002` - - `https://admin-shell.io/aas/API/3/0/SubmodelRegistryServiceSpecification/SSP-001` - - `https://admin-shell.io/aas/API/3/0/SubmodelRegistryServiceSpecification/SSP-002` - - `https://admin-shell.io/aas/API/3/0/DiscoveryServiceSpecification/SSP-001` - - `https://admin-shell.io/aas/API/3/0/AssetAdministrationShellRepositoryServiceSpecification/SSP-001` - - `https://admin-shell.io/aas/API/3/0/AssetAdministrationShellRepositoryServiceSpecification/SSP-002` - - `https://admin-shell.io/aas/API/3/0/SubmodelRepositoryServiceSpecification/SSP-001` - - `https://admin-shell.io/aas/API/3/0/SubmodelRepositoryServiceSpecification/SSP-002` - - `https://admin-shell.io/aas/API/3/0/SubmodelRepositoryServiceSpecification/SSP-003` - - `https://admin-shell.io/aas/API/3/0/SubmodelRepositoryServiceSpecification/SSP-004` - - `https://admin-shell.io/aas/API/3/0/ConceptDescriptionRepositoryServiceSpecification/SSP-001` +Note that the Test Engines return zero in case of compliance and non-zero otherwise so that you can integrate them into ci. + +For more detailed instructions on how to test your AAS Software, see [Test Setups](#test-setups). +If you want to include the Test Engines into your software, see [Python Module Interface](#python-interface). + +## Supported Versions, Suites and Templates + +By default, the Test Engines test against the latest version 3.0, precisely: metamodel => 3.0.1 and API => 3.0.3. +In case of Files, the IDTA specifies submodel templates. +For a full list, please visit [AAS Submodel Templates](https://industrialdigitaltwin.org/content-hub/teilmodelle). +The following templates are supported: +| Name | Semantic ID | Support in test-engine | +| :--- | :--- | :--- | +| Contact Information | https://admin-shell.io/zvei/nameplate/1/0/ContactInformations | ✅ | +| Digital Nameplate for Industrial Equipment | https://admin-shell.io/zvei/nameplate/2/0/Nameplate | ✅ | + +For a detailed list of what is checked (and what is not), see [here](doc/file.md). + +In case of API, the IDTA specifications define service specifications and profiles. Below tables describes the supported API profiles by the current test-engine. For more information about these profiles, please visit [IDTA Specifications for API](https://industrialdigitaltwin.org/wp-content/uploads/2024/10/IDTA-01002-3-0-3_SpecificationAssetAdministrationShell_Part2_API.pdf). + +| Name | Profile Identifier | Description | Support in test-engine | +| :--- | :--- | :--- | :--- | +| AAS Repository Read Profile | https://admin-shell.io/aas/API/3/0/AssetAdministrationShellRepositoryServiceSpecification/SSP-002 | Only read operations for the AAS Repository Service | ✅ | +| Submodel Repository Read Profile | https://admin-shell.io/aas/API/3/0/SubmodelServiceSpecification/SSP-002 | Only read operations for the Submodel Repository Service | ✅ | +| AAS Registry Read Profile | https://admin-shell.io/aas/API/3/0/AssetAdministrationShellRegistryServiceSpecification/SSP-002 | Only reads operations for AAS Registry Service | (✔️) | + +For a detailed list of what is checked (and what is not), see [here](doc/api.md). + +## Test Setups + + +### Check File for Compliance +Given a file in either AASX package format or XML/JSON "raw" format, you can use the Test Engines to check compliance of these files. +Assuming your file is named `my_aas.aasx`, you can invoke the Test Engines using the command line: + +```sh +aas_test_engines check_file my_aas.aasx +``` +This will first check, if your aasx package is correct (in terms of relationships etc.). +Then it checks if all contained AAS are compliant to the meta-model and all constraints hold as defined by Part 1 and Part 3a of the specification. +Finally, the Test Engines will iterate over all submodels in your AAS and check if these are compliant to the corresponding submodel templates. + +In case you want to check other formats, use the `--format` parameter: + +```sh +aas_test_engines check_file my_aas.json --format json +aas_test_engines check_file my_aas.xml --format xml +``` + +### Check Server for compliance +To test compliance of an AAS server to the HTTP/REST API, the Test Engines send a series of requests. +Your server should then answer according to the behavior as defined by Part 2 of the specification. +The Test Engines check the conformance of the responses. +For each operation (aka endpoint) we execute a set of negative tests. +These set parameters to invalid values like negative integers for the `limit` parameter. +Afterwards we execute a set of positive tests which set all parameters to valid values and check the response accordingly. + +Before starting the actual testing procedure, you need to populate some test data at your server which is stored at `bin/check_servers/test_data`. +Then you start the testing by running passing the url of your server and a profile name: + +```sh +aas_test_engines check_server http://my-server.com/api/v3.0 https://admin-shell.io/aas/API/3/0/AssetAdministrationShellRepositoryServiceSpecification/SSP-002 +``` + +This starts the testing procedure which may take some seconds. +You may prefer the HTML output for better readability by running: + + +```sh +aas_test_engines check_server http://my-server.com/api/v3.0 https://admin-shell.io/aas/API/3/0/AssetAdministrationShellRepositoryServiceSpecification/SSP-002 --output html > result.html +``` ## Python Module Interface + ### Check AAS Type 1 (File) @@ -132,7 +163,7 @@ with open('aas.xml') as f: # Or check data directly data = ElementTree.fromstring( '') -result = file.check_xml_data(aas) +result = file.check_xml_data(data) # result.ok() == True result.dump() @@ -159,7 +190,8 @@ Check a running server instance: ```python from aas_test_engines import api -result = api.execute_tests("http://localhost", "https://localhost https://admin-shell.io/aas/API/3/0/AssetAdministrationShellRepositoryServiceSpecification/SSP-002") +conf = api.ExecConf(server="http://localhost") +result, mat = api.execute_tests(conf, "https://admin-shell.io/aas/API/3/0/AssetAdministrationShellRepositoryServiceSpecification/SSP-002") result.dump() ``` @@ -170,7 +202,8 @@ from aas_test_engines import api print(api.supported_versions()) print(api.latest_version()) -result = api.execute_tests("http://localhost", 'Asset Administration Shell API', version="3.0") +conf = api.ExecConf(server="http://localhost") +result, mat = api.execute_tests(conf, "https://admin-shell.io/aas/API/3/0/AssetAdministrationShellRepositoryServiceSpecification/SSP-002", "3.0") result.dump() ``` @@ -179,6 +212,7 @@ result.dump() If you develop an AAS application like an AAS editor you may want to use test data to verify correctness of your application. The test engines allow to generate a set of AAS files which are compliant with the standard and you can therefore use to assess your application as follows: + ```python from aas_test_engines import file diff --git a/bin/check_readme.py b/bin/check_readme.py new file mode 100755 index 0000000..357d50d --- /dev/null +++ b/bin/check_readme.py @@ -0,0 +1,66 @@ +#! /usr/bin/env python3 + +import os +import subprocess +from typing import List, Optional +from tempfile import NamedTemporaryFile, TemporaryDirectory +from dataclasses import dataclass + +DISABLE_CHECK = "" +ROOT_DIR = os.path.abspath(os.path.join(os.path.dirname(os.path.realpath(__file__)), "..")) +README = os.path.join(ROOT_DIR, 'README.md') + +print(f"Checking {README}") + +with open(README) as f: + lines = f.readlines() + + +@dataclass +class Block: + lang: str + check: bool + content: str = "" + + +blocks: List[Block] = [] + +last_block: Optional[Block] = None +check_next = True +for line in lines: + if line.startswith("```"): + if last_block is None: + last_block = Block(line[3:].strip(), check_next) + check_next = True + else: + blocks.append(last_block) + last_block = None + elif line.startswith(DISABLE_CHECK): + if last_block: + raise Exception(f"Found '{DISABLE_CHECK}' within block") + else: + check_next = False + else: + if last_block is not None: + last_block.content += line + +print(f"Found {len(blocks)} code blocks:") + +with TemporaryDirectory() as tmp_dir: + # Create these files as some snippets rely on them + open(os.path.join(tmp_dir, "aas.aasx"), "w") + open(os.path.join(tmp_dir, "aas.json"), "w") + open(os.path.join(tmp_dir, "aas.xml"), "w") + + skipped = 0 + for block in blocks: + if not block.check: + skipped += 1 + continue + with NamedTemporaryFile(mode="w") as f: + print("-"*10) + print(block.content) + print("-"*10) + f.write(block.content) + f.flush() + subprocess.check_call(["python", f.name], env={**os.environ, 'PYTHONPATH': ROOT_DIR}, cwd=tmp_dir) diff --git a/bin/run_tests.sh b/bin/run_tests.sh index e6ae0eb..181273a 100755 --- a/bin/run_tests.sh +++ b/bin/run_tests.sh @@ -6,6 +6,8 @@ SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) cd "$SCRIPT_DIR/.." +./bin/check_readme.py + coverage run \ --source=aas_test_engines \ -m unittest diff --git a/doc/api.md b/doc/api.md new file mode 100644 index 0000000..044ac28 --- /dev/null +++ b/doc/api.md @@ -0,0 +1,43 @@ +# API Tests + +The following sections list all special conditions when checking an AAS REST API. +Please note, that in case of checking responses, the conditions described [here](file.md) apply, too. + +## Query Parameters + +Validation around duplicate query parameters is not in the scope of test-engine. +See https://github.com/admin-shell-io/aas-specs-api/issues/313 for details. + +## Operations + +Methods like `InvokeOperation` are not executed as we cannot check their side-effects in a generic way + +## Non-Applicability of Serialization Modifiers + +In Part 2, Section 11.3 "Applicability of SerializationModifiers" it is not specified what Applicability means in the context of HTTP/REST. +For current scope/release, the test-engine expects below behavior: + +**For level and extent:** +A server should not throw any error for not applicable query parameters, i.e., Level and Extent. The server should in this case, ignore the query parameters and return the serialization of the element. + +**For content:** +The server should result in an error, usually 404. + +See https://github.com/admin-shell-io/aas-specs-api/issues/327 for details. + +## GetAllSubmodelElements + +`/submodel-elements` is under-specified for the different SerializationModifiers. +See the following issues for details: +* https://github.com/admin-shell-io/aas-specs-api/issues/265 +* https://github.com/admin-shell-io/aas-specs-api/issues/185 +* https://github.com/admin-shell-io/aas-specs-api/issues/246 + +## Pagination + +It is possible that the server is modified during the pagination request. However, the specifications are underdefines for such scenario. +Thus, for the current scope of the test-engine, modification around pagination is not being considered. + +Cursor has to be serialized as string, even though it is an integer. No other data type is allowed. +As the specification allows the cursor in the response to be both Base64UrlEncoded as well as not-encoded, the test-engine supports both. + diff --git a/doc/file.md b/doc/file.md new file mode 100644 index 0000000..fc8689c --- /dev/null +++ b/doc/file.md @@ -0,0 +1,15 @@ +# File Tests + +The following sections list all special conditions when checking an AAS file. + +## Constraints + +All constraints of Part 1 and Part 3a are checked except for the following: +* AASd-012 +* AASd-115 +* AASd-020 +* AASd-021 +* AASd-022 +* AASd-077 +* AASc-3a-003 +* AASc-3a-050