Feature/connector sync #100

arejula27 · 2023-07-13T14:14:51Z

This Pr refactos the current connectors, separating them from the server and exposing a uniformed interface for executing them from external scrips.

For completing the issue #16 is needed more development:

Create the script src/connectors/synchronization.py
Create synchronize.sh

josvandervelde

A step in the right direction :)

src/connectors/abstract/resource_connector_by_date.py

src/connectors/example/example_dataset_connector.py

src/connectors/example/example_connector.py

src/tests/connectors/huggingface/test_huggingface_dataset_connector.py

src/tests/connectors/zenodo/test_get_datasets_zenodo.py

src/connectors/zenodo/zenodo_dataset_connector.py

src/main.py

src/connectors/abstract/resource_connector_by_date.py

src/connectors/huggingface/huggingface_dataset_connector.py

src/connectors/example/example_dataset_connector.py

src/connectors/example/example_connector.py

src/connectors/huggingface/huggingface_dataset_connector.py

src/connectors/openml/openml_dataset_connector.py

src/connectors/example/example_dataset_connector.py

…into feature/connector-sync

#16

…adata model)

… dont think anybody uses it. If anybody wants it back, feel free to set it back

Taniya-Das

looks good to me.

arejula27 added 7 commits July 6, 2023 07:56

example changed

143bf70

connector by date

d698a6a

zenodo conector

5128a38

openml connector

f848db7

remove connectors from api

acaf5d1

error on zenodo

d4fd687

openml

6eba04a

arejula27 requested a review from josvandervelde July 13, 2023 14:14

josvandervelde suggested changes Jul 14, 2023

View reviewed changes

arejula27 added 21 commits July 14, 2023 09:26

delete retun

ac7e485

delete code duplication dataset example

6a182df

change exception to Value error

eaf8d85

change excepcion to value error

3860b61

change id to id_ test zenodo

06ebd63

change a nmethod to static

daf0408

change var name

0869102

remove commnets

ea2fb14

example filter by time

0344c82

change default example date dataset

4f0a358

comments

14efe89

reduce code duplication

f16823f

type hints in error class

d4b4b29

record error

eba1311

add record to abstact connector class

fb062d6

change id

2bc4465

zenodo retry error handling

a237ad5

remove comment sflag dataset

320f474

fix test

b51cc9a

fix setup fetch

2735d98

limit remove

424bdff

arejula27 added 5 commits July 24, 2023 12:27

add check_valid_id

bc48b59

main connector

e396e3d

comments

b1b0c5a

config

e4d218d

delete useless line

cdbaa19

josvandervelde suggested changes Jul 26, 2023

View reviewed changes

arejula27 and others added 19 commits July 26, 2023 13:40

create new abstract class

ed42f01

change id type to int

6090839

delete useless comment

18ecf50

connector hugg changed to on startup

970448d

typo

b36a84b

error handling huggingface

91be909

add header error

9549cc9

delete raise exception

27fe77c

typo

2cf514b

reason why zenodo uses 2 protocols

322efe8

ad until to the protocol

a4c7ee3

Merge branch 'develop' of https://github.com/aiondemand/AIOD-rest-api …

3ef74d2

…into feature/connector-sync

Updated the synchronization and connectors to be more like described in

9e3e8c2

#16

Fixed merge conflicts between connector-sync and develop (the new met…

e773777

…adata model)

Moved example connectors to separate docker container

1905e81

Openml connector running as cron job inside a separate container

45c5054

Huggingface and zenodo connectors in containers

70d330e

Removed devcontainer (dev docker compose) because its outdated, and I…

1df23e4

… dont think anybody uses it. If anybody wants it back, feel free to set it back

Cleanup

778f86f

josvandervelde approved these changes Aug 22, 2023

View reviewed changes

josvandervelde requested a review from Taniya-Das August 22, 2023 13:13

Taniya-Das approved these changes Aug 23, 2023

View reviewed changes

josvandervelde merged commit 21c9498 into develop Aug 23, 2023

josvandervelde deleted the feature/connector-sync branch August 23, 2023 13:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/connector sync #100

Feature/connector sync #100

arejula27 commented Jul 13, 2023 •

edited

Loading

josvandervelde left a comment

Taniya-Das left a comment

Feature/connector sync #100

Feature/connector sync #100

Conversation

arejula27 commented Jul 13, 2023 • edited Loading

josvandervelde left a comment

Choose a reason for hiding this comment

Taniya-Das left a comment

Choose a reason for hiding this comment

arejula27 commented Jul 13, 2023 •

edited

Loading