From 250c67f20017de7f5727607fd72528afafe0b5d5 Mon Sep 17 00:00:00 2001
From: esparig <5576714+esparig@users.noreply.github.com>
Date: Thu, 18 Jul 2024 15:11:14 +0200
Subject: [PATCH] README changes: - New diagram with updated DCNiOS logo, -
Clarification of some sentences, - Removed redundancies.
---
README.md | 53 +++++++++---------
.../images/OSCAR-dcnios-workflow-drawio.png | Bin 0 -> 143696 bytes
2 files changed, 27 insertions(+), 26 deletions(-)
create mode 100644 docpage/docs/images/OSCAR-dcnios-workflow-drawio.png
diff --git a/README.md b/README.md
index ba3b3807..ec03bf0b 100644
--- a/README.md
+++ b/README.md
@@ -1,34 +1,44 @@
# DCNiOS
-DCNiOS is an open-source command-line tool to easily manage the creation of event-driven data processing flows. DCNiOS, Data Connector through Apache NiFi for OSCAR, facilitates the creation of event-driven processes connecting a Storage System like [dCache](http://dcache.org) or [S3](https://aws.amazon.com/s3) to a scalable OSCAR cluster by employing predefined dataflows that are processed by Apache NiFi.
+[DCNiOS](https://intertwin-eu.github.io/dcnios) is an open-source command-line tool to easily manage the creation of event-driven data processing flows. DCNiOS, Data Connector through Apache NiFi for OSCAR, facilitates the creation of event-driven processes connecting a Storage System like [dCache](http://dcache.org) or [Amazon S3](https://aws.amazon.com/s3) to a scalable OSCAR cluster by employing predefined dataflows that are processed by Apache NiFi.
[Apache NiFi](http://nifi.apache.org) is a reliable system to process and distribute data through powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
[OSCAR](https://oscar.grycap.net) is an open-source platform for serverless event-driven data processing of containerized applications across the computing continuum.
-Together with [DCNiOS](http://github.com/grycap/dcnios) (Data Connector + NiFi + OSCAR), you can manage the creation of event-driven data processing flows. As shown in the figure, when an event occurs in the external component, dCache in this case, events are ingested in Apache NiFi, which can queue them up depending on the (modifiable at runtime) ingestion rate, to be then delegated for processing into a scalable OSCAR cluster, where a user-defined application based on a Docker image can process the data file.
+The DCNiOS command-line interface allows you to create a NiFi dataflow though a YAML file (see figure).
-
+After creating a dataflow using DCNiOS, when an event is captured (e.g., Server-sent event by dCache), Apache NiFi stores it in an internal queue and then sends it for processing to an OSCAR cluster at a customized rate. In the OSCAR cluster, a user-defined service creates a job to process the stored data.
-Therefore, DCNiOS has been made to interact with NiFi and deploy a complete dataflow. It uses HTTP calls to communicate with a Nifi cluster, which can be automatically deployed by the [Infrastructure Manager (IM)](https://im.egi.eu). Apache NiFi is deployed on a dynamically provisioned Kubernetes. It does not require a Nifi registry.
+![DCNiOS Workflow](docpage/docs/images/OSCAR-dcnios-workflow-drawio.png)
-All the dataflow information is described in a YAML file, and by executing the DCNiOS command-line interface, this dataflow is deployed on Nifi.
+Therefore, DCNiOS has been made to interact with NiFi and deploy a complete dataflow. It uses HTTP calls to communicate with a Nifi cluster.
-From predefined recipes (ProcessGroup in Nifi, .json files) created before, DCNiOS inserts a general flow and changes the variables to create a concrete workflow.
+Apache NiFi can be automatically deployed using the [Infrastructure Manager (IM)](https://im.egi.eu) on a dynamically provisioned Kubernetes Cluster. **Note** that a NiFi registry is not required.
+## Achievements
+
+
+
+
+This software has received a silver badge according to the [Software Quality Baseline criteria](https://www.eosc-synergy.eu/for-developers/) defined by the [EOSC-Synergy](https://www.eosc-synergy.eu) project.
+
+
## Getting Started
### Prerequisites
-- OSCAR cluster with services deployed
-- Nifi Cluster deployed
-- A package provider such as [Anaconda](https://www.anaconda.com/)
+- OSCAR cluster with services deployed.
+- NiFi Cluster deployed.
+- A Python distribution such as [Anaconda](https://www.anaconda.com/).
### Installation
-Create an environment with conda and use it.
+#### Recommended
+Create an environment with conda and activate it.
``` bash
conda create --name dcnios python=3.7.6
@@ -40,8 +50,8 @@ Install all the requirements defined in `requirements.txt`
``` bash
pip install -r requeriments.txt
```
-
-Or only install the minimal requirements that DCNiOS needs.
+#### Minimum
+Install the minimum requirements for DCNiOS:
``` bash
@@ -51,17 +61,17 @@ pip install pyyaml==6.0 requests==2.28.2 oscar_python==1.0.3
## Authors
- Germán Moltó mailto:gmolto@dsic.upv.es
-- Estibaliz Parcero mailto:esparig@i3m.upv.es
+- Estíbaliz Parcero mailto:esparig@i3m.upv.es
- Sergio Langarita mailto:slangarita@i3m.upv.es
-Instituto de Instrumentación para Imagen Molecular (I3M), Centro Mixto CSIC — Universitat Politècnica de València, Camino de Vera s/n, 46022 Valencia, España
+Instituto de Instrumentación para Imagen Molecular (I3M), Centro Mixto CSIC — Universitat Politècnica de València, Camino de Vera s/n, 46022 Valencia, España.
## Versions and Maintenance
There is only one version in maintenance:
- The main branch in the source code repository maintains a working state version of the software component.
-- Documentation is updated with the new software versions involving any substantial or minimal change in the application's behavior. Any issue can be reported in the [Issues section of the GitHub project](https://github.com/interTwin-eu/dcnios/issues)
+- Documentation is updated with the new software versions involving any substantial or minimal change in the application's behavior. Any issue can be reported in the [Issues section of the GitHub project](https://github.com/interTwin-eu/dcnios/issues).
- Documentation is updated whenever reported as inaccurate or unclear.
## Licensing
@@ -70,19 +80,10 @@ DCNiOS is licensed under the Apache License, Version 2.0. See LICENSE for the fu
## Acknowledgements
-This work was supported by the project “An interdisciplinary Digital Twin Engine for science’’ (interTwin), which has received funding from the European Union’s Horizon Europe Programme under Grant 101058386.
+
-
+This work was supported by the project “An interdisciplinary Digital Twin Engine for science’’ (interTwin), which has received funding from the European Union’s Horizon Europe Programme under Grant 101058386.
## More information
You can find more [information](https://oscar.grycap.net/blog/data-driven-processing-with-dcache-nifi-oscar/ ) in the [OSCAR's blog.](https://oscar.grycap.net/blog/)
-
-
-
-
-
-
-
-This software has received a silver badge according to the [Software Quality Baseline criteria](https://www.eosc-synergy.eu/for-developers/) defined by the [EOSC-Synergy](https://www.eosc-synergy.eu) project. Please acknowledge the use of DCNiOS by citing the following scientific
-publications ([preprints available](https://www.grycap.upv.es/gmolto/publications)):
\ No newline at end of file
diff --git a/docpage/docs/images/OSCAR-dcnios-workflow-drawio.png b/docpage/docs/images/OSCAR-dcnios-workflow-drawio.png
new file mode 100644
index 0000000000000000000000000000000000000000..dcc21bef9bf79408f649b85fdf6a1a014900f65b
GIT binary patch
literal 143696
zcmd?S%d+ao)-JYxH%>TC#Cd@ej&Q3Y?8dIpl}l3>1OfyIBtUc=k`O{5Bmn{;-oP(&
z9>w9;up4f=>F_s%n3Flzs|VLnf&jQQTM<6=l}Yj|M8E1
z{1?5aPXF)k4Ws08!|Scp
z_}?1B|B81~mc_v$@CFf{tBHK`XTCwV40@I^{E1WWK92SQUb=z5Pw5tAdrA+uqOn}#
zXNLa8#hJTtPti*kG>0H}!^ijsSL37y{i2z$vwpI6(4OxSGUQ)EFXHO3$rgMOIH`sI^RCpwz%eFEOfmWMB(=NWVX?Md~CH*(g_V>l_2GBg18W`Xd#BLs4Uwh#g
zw`TnCtsUu%A7Ku(sd?+$V<8Rxi3PtpPqvnbs~cbHG5*9gVvt7Oq)^n*WN#eNI9`bL
zt1td}uCxK^FILR3N?4QU_qeupbN8Rr!9UL@zarrC%Wa;McEF7q{-Y-6pC5|T&7$+q
zetN)V|BNQu#yvzaE%`L{Qn1eJ12%5G%+G$mY|{v7u+}LGL+IUlQuQk=`gXlwY$+F3?k$D95EY)8e
zqV<}ziF^vcF0NN;7LwmAjj-;-7oz-=sN%|(68&{{t<@AgDR|nk$Gx?Ysh@&t-_+v|
zS$+H^S$UDzy4!y;KeJ7~7LVFMf-XdwAC~EF%joAZOo%Gzz8*kroS(WP-E7WJ(1rE-
zjv0B*3@u+K_wDiUKvBQ5c6re2gL3)ATkkf6%tf({@?e%dgu4K0!L=8A0dyKGX`DV_
z6yIskzi4vKPw75*`9_o*N>h6K=7|mfy1v3`we9$Vr@l?CCxu_g`?hp!Q+&+4<{Lq%
z4R-MkQ`lyGZTkvQnEK*#)&KPtf&;9>(4)`kDjAM`^P|kWNNiqdehL~zUTK>V(?)r
z0r;TBK0u_u0^0daIPDWeVuCAq7{7!SzM=*G8mQo@qQ21J9PvU#;v()6;l=S=qm{?9=sp__==*e^?pn
zSc|TWzYiORPY+l3=@I{B)jXoZkG1*Zoj-)2c(G=MH@*9K`k$lpYaL%d`#*v!SX1*|
zeE$2^_;rl>Pt^E7qJ)2c^vpdYHoq8P{NYgEQ)8VoH-U)mU4BS~XG{kSx~U8V%Rl{+*-$mb&qiF5{S2*xPBF
zyUlVZc=6Wp1o7mq#r1=4zU|LHLk1thY<~`;y`kEg)!1hg@5yVAY}0Q*x$l$u6k+-N
zNKWA4s(fJZ-wdsv5B=p3(T`Y&XD@(_FH7?6HT?ap3R@H9mtp$fJSJ0_sK1>Rs`c2P
zV1w-
znxK2m9(6gJ=tkp^Q3q31GaYW2W48U-EzZKv$lg|E6*_{cXoHWBcj|{$7b5*PVr5?_
zg>T!8r=58*GH>ZMj{Ar|pLY2ryY{u@+Vk2|fd2xywf`8v@4HnuUw3vt0>rQ8`z!kV
z21faVEkd24Z+y;&JsI)IVSgZ2`w^l%WawRzK7mc))A0GHfIxHpPgEY7S$-d)TuH|`+4CPDpjk-li(Dg9+dFWxATdw0h6!V!+H@v
zMOuEu7JMpl`*XOECksK*xta^#>Y$!n_}4a&JX@=bdiZ~i0sHm#`bo{-9FO`}XI*Q2
z_{%2qUE+V-Eq%oDb>l}Z)zkOc1~(U_?hx$kMQ-b>Z=y`@)OV{D;Z!}{FEoFYOshP3
zUJqaKwJwpI&8KfhYUY#ksa5~r`qNE(7TLbJhsa>~Q4-3vyQqSB(aXn*$u|B$)$OBq
z?Nfbq?H@i%=pU8ZoJh6WB2{bS4-M+edTo3`729a~tl7rfXVvx#8L6-nBZL;D34Y-&
z3r(TPD){E1Grm%9@jLpEywH9|R%_|zSmAl)#gSy6D(o*_runrr8)@Em$>E+19v!NE
zHE*2sivfq2YQAOH--lI|?lau+6UVY
z$8M&FHsiGO6>3VpVS^i|OnM`3Rmy_5xLy1^6S=)9A
zDDC_D;759Q8@r2jx>lKJbhj^~yLmUPkk=!bcNbH}#dUlxhC7oT&GgV+m~B15bM8Ke
z>yh4D#eOo%JWgoqyUxWxoAn`%*G&H!RQK+M@l3iKcl~0hhN!C@!B7NdWZ8u=llak0
z;76))8>{