From 00a77b9c0478869c165753fcdc99d64bff168f98 Mon Sep 17 00:00:00 2001 From: SergioLangaritaBenitez Date: Thu, 4 Jul 2024 13:36:51 +0200 Subject: [PATCH 1/4] update documentation --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ba3b3807..faa4b01e 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ DCNiOS is an open-source command-line tool to easily manage the creation of even [OSCAR](https://oscar.grycap.net) is an open-source platform for serverless event-driven data processing of containerized applications across the computing continuum. -Together with [DCNiOS](http://github.com/grycap/dcnios) (Data Connector + NiFi + OSCAR), you can manage the creation of event-driven data processing flows. As shown in the figure, when an event occurs in the external component, dCache in this case, events are ingested in Apache NiFi, which can queue them up depending on the (modifiable at runtime) ingestion rate, to be then delegated for processing into a scalable OSCAR cluster, where a user-defined application based on a Docker image can process the data file. +Together with [DCNiOS](http://github.com/grycap/dcnios) ( NiFi + OSCAR), you can manage the creation of event-driven data processing flows. As shown in the figure, when an event occurs in the external component, dCache in this case, events are ingested in Apache NiFi, which can queue them up depending on the (modifiable at runtime) ingestion rate, to be then delegated for processing into a scalable OSCAR cluster, where a user-defined application based on a Docker image can process the data file. DCNiOS Workflow From db13732d8d0719221d31e2ba83ba4a66b27df029 Mon Sep 17 00:00:00 2001 From: SergioLangaritaBenitez Date: Thu, 4 Jul 2024 13:38:39 +0200 Subject: [PATCH 2/4] update documentation --- docpage/docusaurus.config.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docpage/docusaurus.config.js b/docpage/docusaurus.config.js index d2f9bd17..de0d44eb 100644 --- a/docpage/docusaurus.config.js +++ b/docpage/docusaurus.config.js @@ -15,7 +15,7 @@ const config = { tagline: 'Data Connector through Apache NiFi for OSCAR', favicon: 'img/favicon.ico', - // Set the production url of your site here #3f3f9f + // Set the production url of your site here url: 'https://intertwin-eu.github.io/', // Set the // pathname under which your site is served // For GitHub pages deployment, it is often '//' From 2c77d70d2d79204993b356330e36ca133541a3f7 Mon Sep 17 00:00:00 2001 From: SergioLangaritaBenitez Date: Thu, 4 Jul 2024 13:42:23 +0200 Subject: [PATCH 3/4] update documentation --- docpage/docs/Introduction.md | 4 ++-- docpage/src/pages/markdown-page.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docpage/docs/Introduction.md b/docpage/docs/Introduction.md index e47a3178..c13b4fb5 100644 --- a/docpage/docs/Introduction.md +++ b/docpage/docs/Introduction.md @@ -12,8 +12,8 @@ DCNiOS is an open-source command-line tool that easily manages the creation of e Apache NiFi Process Group is a group of Processors that compose a dataflow. DCNiOS uses predefined Process Groups that make simple actions like interacting with a third-party component (e.g., consuming from Kafka) or changing the data content (e.g.encoding the data in base64) to compose a complete dataflow. In DCNiOS documentation, the Process Groups are split by purpose into three main groups: 'Sources', 'Destinations', and 'Alterations'. -- 'Sources' interact with a third-party component as the input data receiver. -- 'Destinations' interact with a third-party component as output data sender. +- 'Sources' interact with a third-party component as the input data receiver. +- 'Destinations' interact with a third-party component as an output data sender. - 'Alterations' that do not interact with third-party components and change the format of the data flow. diff --git a/docpage/src/pages/markdown-page.md b/docpage/src/pages/markdown-page.md index 1cad08ea..03e201d5 100644 --- a/docpage/src/pages/markdown-page.md +++ b/docpage/src/pages/markdown-page.md @@ -6,7 +6,7 @@ title: Main Page DCNiOS is an open-source command-line tool to easily manage the creation of event-driven data processing flows. DCNiOS, Data Connector through [Apache NiFi](https://nifi.apache.org/) for [OSCAR](https://oscar.grycap.net/), facilitates the creation of event-driven processes connecting a Storage System like [dCache](https://www.dcache.org/) or [S3](https://aws.amazon.com/s3) to a scalable OSCAR cluster by employing predefined dataflows that are processed by Apache NiFi. -DCNiOS was developed within the interTwin project. DCNiOS creates dataflows in Apache NiFi that captures events or messages, and ingests them in an OSCAR cluster at a customized rate, where an OSCAR service is run based on a user-defined application (containerized in a Docker image). +DCNiOS was developed within the interTwin project. DCNiOS creates dataflows in Apache NiFi that capture events or messages and ingest them in an OSCAR cluster at a customized rate, where an OSCAR service is run based on a user-defined application (containerized in a Docker image). The DCNiOS command-line application is available in the Source Code repository. Additionally, the corresponding TOSCA templates and the ansible roles that are required to deploy an Apache Nifi cluster via the Infrastructure Manager (IM) have been provided. Any user can self-deploy such a cluster via the [IM Dashboard](https://im.egi.eu). From 788c5ff28bf94a15b3e9f75a4023fc047f8a587c Mon Sep 17 00:00:00 2001 From: SergioLangaritaBenitez Date: Fri, 2 Aug 2024 12:39:23 +0200 Subject: [PATCH 4/4] Documentation of alterations --- cli/description.yaml | 12 ++++++++++-- cli/destinations/{s3sqs.py => s3aws.py} | 0 cli/env.py | 2 +- docpage/docs/05.- Alterations/Decode.md | 18 ++++++++++++++++++ docpage/docs/05.- Alterations/Encode.md | 16 ++++++++++++++++ docpage/docs/05.- Alterations/Merge.md | 23 +++++++++++++++++++++++ docpage/docs/Users.md | 12 ++++++++++++ 7 files changed, 80 insertions(+), 3 deletions(-) rename cli/destinations/{s3sqs.py => s3aws.py} (100%) create mode 100644 docpage/docs/05.- Alterations/Decode.md create mode 100644 docpage/docs/05.- Alterations/Encode.md create mode 100644 docpage/docs/05.- Alterations/Merge.md diff --git a/cli/description.yaml b/cli/description.yaml index 44901ea9..6213edd5 100644 --- a/cli/description.yaml +++ b/cli/description.yaml @@ -35,6 +35,14 @@ nifi: ssl_context: Truststore_Filename: Truststore_Password: "" + alterations: + - action: Encode + Encoding: base64 + - action: Decode + Encoding: base64 + - action: Merge + maxMessages: 10 + windowSeconds: 7 SQS: - name: SQS-events AWS_DEFAULT_REGION: @@ -57,5 +65,5 @@ nifi: connection: - from: dcache to: invoke - - from: dcache - to: invoke2 + - from: kafka + to: invoke diff --git a/cli/destinations/s3sqs.py b/cli/destinations/s3aws.py similarity index 100% rename from cli/destinations/s3sqs.py rename to cli/destinations/s3aws.py diff --git a/cli/env.py b/cli/env.py index 309560b1..fda01672 100644 --- a/cli/env.py +++ b/cli/env.py @@ -19,7 +19,7 @@ from sources.kafka import createKafka from sources.generic import createGeneric from destinations.oscar import createOSCAR -from destinations.s3sqs import createPutS3 +from destinations.s3aws import createPutS3 folderSource = "template/sources/" diff --git a/docpage/docs/05.- Alterations/Decode.md b/docpage/docs/05.- Alterations/Decode.md new file mode 100644 index 00000000..28e868b6 --- /dev/null +++ b/docpage/docs/05.- Alterations/Decode.md @@ -0,0 +1,18 @@ +--- +sidebar_position: 2 +--- + +# Decode + +Alteration's Decode decodes the data flow from the chosen encoding. The user must ensure the input data is encoded using the selected encoding. +Three encodes are available: `base64`, `base32` and `hex`. It is similar to the command `base64 -d` or `base32 -d`. For example, If the input data is a string in base64 with the value `aGVsbG8K` or in base32 with the value `NBSWY3DPBI======`. The output data will be the same in both cases, `hello`. + + +Here is the YAML example. + + +``` +alterations: + - action: Decode + Encoding: base64 +``` \ No newline at end of file diff --git a/docpage/docs/05.- Alterations/Encode.md b/docpage/docs/05.- Alterations/Encode.md new file mode 100644 index 00000000..03c12bfb --- /dev/null +++ b/docpage/docs/05.- Alterations/Encode.md @@ -0,0 +1,16 @@ +--- +sidebar_position: 1 +--- + +# Encode + +Alteration's Encode changes the data flow to the chosen encoding. Three encodings are available: `base64`, `base32` and `hex`. It is similar to the command `base64` `base32` or `hexdump`. +For example, If the input data is a string with the message `hello`. The output message encode in base64 will be `aGVsbG8K`, in base32 will be `NBSWY3DPBI======` and in hex `0000000 6568 6c6c 0a6f 0000006` + +Here is the YAML example. + +``` +alterations: + - action: Encode + Encoding: base64 +``` \ No newline at end of file diff --git a/docpage/docs/05.- Alterations/Merge.md b/docpage/docs/05.- Alterations/Merge.md new file mode 100644 index 00000000..a9de665c --- /dev/null +++ b/docpage/docs/05.- Alterations/Merge.md @@ -0,0 +1,23 @@ +--- +sidebar_position: 3 +--- + +# Merge + +Alteration's Merge put together some messages in one. There are two variables to set: the maximum messages `maxMessages` can get and the waiting window time `windowSeconds`. +The final output will aggregate the input messages separated by a lane break. If the receive data order is `I am the message 1` and `I am the message 2` the output will be: + +``` +I am the message 1 +I am the message 2 +``` + +Here is the YAML example. + + +``` +alterations: + - action: Merge + maxMessages: 10 + windowSeconds: 2 +``` \ No newline at end of file diff --git a/docpage/docs/Users.md b/docpage/docs/Users.md index 037106ac..cb3dd82d 100644 --- a/docpage/docs/Users.md +++ b/docpage/docs/Users.md @@ -88,6 +88,18 @@ components: ``` +#### Alterations + +The subsection `alterations`, inside Sources, change the data format. These alterations are applied as a descendent definition. In this example, the input data is merged into one message. Then, the merge message is encoded. + +``` + - action: Merge + maxMessages: 10 + windowSeconds: 7 + - action: Encode + Encoding: base64 +``` + ### Connections