Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/ted 314 #89

Merged
merged 5 commits into from
Apr 27, 2022
Merged

Feature/ted 314 #89

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
eec4264
Updated Makefile
Apr 27, 2022
7e1de94
Merge remote-tracking branch 'origin/main' into feature/TED-314
Apr 27, 2022
6d193e4
Updated Make target to support the correct rmlmapper.jar installation
Apr 27, 2022
7379ba2
Updated airflow Dockerfile to support the correct rmlmapper.jar insta…
Apr 27, 2022
106d95b
Updated airflow Dockerfile to support the correct rmlmapper.jar insta…
Apr 27, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ PROJECT_PATH = $(shell pwd)
AIRFLOW_INFRA_FOLDER ?= ${PROJECT_PATH}/.airflow
RML_MAPPER_PATH = ${PROJECT_PATH}/.rmlmapper/rmlmapper.jar
XML_PROCESSOR_PATH = ${PROJECT_PATH}/.saxon/saxon-he-10.6.jar

LOGGING_TYPE = PY
#-----------------------------------------------------------------------------
# Dev commands
#-----------------------------------------------------------------------------
Expand Down Expand Up @@ -154,8 +154,8 @@ stop-metabase:

init-rml-mapper:
@ echo -e "RMLMapper folder initialisation!"
@ wget -c https://github.com/RMLio/rmlmapper-java/releases/download/v5.0.0/rmlmapper-5.0.0-r362-all.jar -P .rmlmapper/
@ mv .rmlmapper/rmlmapper-5.0.0-r362-all.jar .rmlmapper/rmlmapper.jar 2>/dev/null
@ mkdir -p ./.rmlmapper
@ wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1MckjzpvrCoChy_IRYC2S54tf3dFQNwEt' -O- | sed -rn \'s/.*confirm=\([0-9A-Za-z_]+).*/\1\n/p\'\)&id=1MckjzpvrCoChy_IRYC2S54tf3dFQNwEt" -O ./.rmlmapper/rmlmapper.jar && rm -rf /tmp/cookies.txt

init-saxon:
@ echo -e "$(BUILD_PRINT)Saxon folder initialization $(END_BUILD_PRINT)"
Expand Down Expand Up @@ -193,7 +193,7 @@ staging-dotenv-file: guard-VAULT_ADDR guard-VAULT_TOKEN vault-installed
@ echo ELK_HOST=localhost >> .env
@ echo ELK_PORT=5959 >> .env
@ echo ELK_VERSION=1 >> .env
@ echo LOGGING_TYPE=PY,ELK >> .env
@ echo LOGGING_TYPE=${LOGGING_TYPE} >> .env
@ echo XML_PROCESSOR_PATH=${XML_PROCESSOR_PATH} >> .env
@ echo AIRFLOW_INFRA_FOLDER=~/airflow-infra/staging >> .env
@ vault kv get -format="json" ted-staging/airflow | jq -r ".data.data | keys[] as \$$k | \"\(\$$k)=\(.[\$$k])\"" >> .env
Expand All @@ -212,7 +212,7 @@ dev-dotenv-file: guard-VAULT_ADDR guard-VAULT_TOKEN vault-installed
@ echo ELK_HOST=localhost >> .env
@ echo ELK_PORT=5959 >> .env
@ echo ELK_VERSION=1 >> .env
@ echo LOGGING_TYPE=PY,ELK >> .env
@ echo LOGGING_TYPE=${LOGGING_TYPE} >> .env
@ echo XML_PROCESSOR_PATH=${XML_PROCESSOR_PATH} >> .env
@ echo AIRFLOW_INFRA_FOLDER=${AIRFLOW_INFRA_FOLDER} >> .env
@ vault kv get -format="json" ted-dev/airflow | jq -r ".data.data | keys[] as \$$k | \"\(\$$k)=\(.[\$$k])\"" >> .env
Expand All @@ -232,7 +232,7 @@ prod-dotenv-file: guard-VAULT_ADDR guard-VAULT_TOKEN vault-installed
@ echo ELK_HOST=localhost >> .env
@ echo ELK_PORT=5959 >> .env
@ echo ELK_VERSION=1 >> .env
@ echo LOGGING_TYPE=PY,ELK >> .env
@ echo LOGGING_TYPE=${LOGGING_TYPE} >> .env
@ echo XML_PROCESSOR_PATH=${XML_PROCESSOR_PATH} >> .env
@ echo AIRFLOW_INFRA_FOLDER=~/airflow-infra/prod >> .env
@ vault kv get -format="json" ted-prod/airflow | jq -r ".data.data | keys[] as \$$k | \"\(\$$k)=\(.[\$$k])\"" >> .env
Expand All @@ -259,7 +259,7 @@ elk-add-dotenv-file:
logging-add-dotenv-file:
@ echo -e "$(BUILD_PRINT)Add logging config to local .env file $(END_BUILD_PRINT)"
@ sed -i '/^LOGGING_TYPE/d' .env
@ echo LOGGING_TYPE=PY,ELK >> .env
@ echo LOGGING_TYPE=${LOGGING_TYPE} >> .env

refresh-mapping-files:
@ python -m ted_sws.data_manager.entrypoints.generate_mapping_resources
Expand Down
21 changes: 20 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,15 @@
#TED Semantic Web Services
# TED Semantic Web Services

<hr>

## Table of contents

- [Developer documentation](#developer-documentation)
- [Installation](#installation)
- [Usage](#usage)


<hr>

## Developer documentation

Expand All @@ -7,6 +18,8 @@ If you contribute to this project please refer to the following project document
* [Enterprise architecture model file (in development)](https://drive.google.com/file/d/1YB2dPYe9E9bAR2peVraQaUANS-hXetms/view?usp=sharing)
* [Meaningfy google Drive of the project (restricted)](https://drive.google.com/drive/folders/1wfWYDAtcaJrYTuB14VzTixr1mJUkCHYl?usp=sharing)

<hr>

## Installation
### Installation of ted-sws package within external projects using terminal
#### 1. Using the package manager pip
Expand Down Expand Up @@ -36,6 +49,8 @@ install:
make setup
```

<hr>

## Usage
### CLI tools (commands/console-scripts) for notice mapping suite manipulation

Expand Down Expand Up @@ -180,6 +195,10 @@ mapping_suite_processor -c COMMAND1 -c COMMAND2 ...
```
to set custom commands (order) to be executed

<hr>

## Contributions

<hr>

## Licence
4 changes: 2 additions & 2 deletions infra/airflow/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ COPY requirements.txt /opt/airflow

# working in the /opt/airflow
WORKDIR /opt/airflow
RUN wget -c https://github.com/RMLio/rmlmapper-java/releases/download/v5.0.0/rmlmapper-5.0.0-r362-all.jar -P .rmlmapper/
RUN mv .rmlmapper/rmlmapper-5.0.0-r362-all.jar .rmlmapper/rmlmapper.jar 2>/dev/null
RUN mkdir -p ./.rmlmapper
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just mkdir -p .rmlmapper

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

asa a pus Stefan la artefacts, ca sa fie 100% sigur

RUN wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1MckjzpvrCoChy_IRYC2S54tf3dFQNwEt' -O- | sed -rn \'s/.*confirm=\([0-9A-Za-z_]+).*/\1\n/p\'\)&id=1MckjzpvrCoChy_IRYC2S54tf3dFQNwEt" -O ./.rmlmapper/rmlmapper.jar && rm -rf /tmp/cookies.txt

RUN pip install --upgrade pip
RUN pip install --no-cache-dir -r requirements.txt --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.5/constraints-no-providers-3.8.txt"
2 changes: 2 additions & 0 deletions ted_sws/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ def RML_MAPPER_PATH(self) -> str:


class ELKConfig:

@property
def ELK_HOST(self) -> str:
return VaultAndEnvConfigResolver().config_resolve()
Expand All @@ -82,6 +83,7 @@ def ELK_VERSION(self) -> int:


class LoggingConfig:

@property
def LOGGING_TYPE(self) -> str:
return VaultAndEnvConfigResolver().config_resolve()
Expand Down