Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configure ingestion pipeline from a configuration file #1393

Open
ire-and-curses opened this issue Jun 11, 2019 · 0 comments
Open

Configure ingestion pipeline from a configuration file #1393

ire-and-curses opened this issue Jun 11, 2019 · 0 comments
Assignees
Labels
ingest New ingestion system

Comments

@ire-and-curses
Copy link
Contributor

Currently the ingestion pipeline is built in code, e.g. like this.

It would be a big improvement if the description of a pipeline made up of the generic components we provide could be specified from a configuration file. The file should be a standard format (e.g. YAML) if possible. This would allow users to create custom pipelines without editing code, and also decouples user decisions from the code base, making it convenient to record or version control user's pipeline setups.

@ire-and-curses ire-and-curses added the ingest New ingestion system label Jun 11, 2019
@bartekn bartekn self-assigned this Jul 24, 2019
@bartekn bartekn added this to the Horizon 0.22.0 milestone Sep 4, 2019
@bartekn bartekn removed this from the Horizon 0.22.0 milestone Sep 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ingest New ingestion system
Projects
None yet
Development

No branches or pull requests

2 participants