-
Notifications
You must be signed in to change notification settings - Fork 337
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor as a (self hosted) service #339
Comments
in term of implementation, maybe ray eg #272 can help guide things / compare with the http path |
https://github.com/ml6team/fondant is doing some good things in term of packaging, a bit similar to what jina is doing. They're doing docker though which is not clear how it can be made to work looks like they're becoming dependent on dask though https://github.com/ml6team/fondant/blob/main/components/load_from_parquet/src/main.py which is what I'd like to avoid (being dependent on any given distribution framework) https://github.com/ml6team/fondant/blob/main/src/fondant/component.py they're also locking themselves into kubeflow things to consider taking from them in term of design:
|
https://github.com/rom1504/img2dataset/tree/streaming_refacto some work I started on that some 8 months ago
I still think it's the right direction
may try to finish it soon
would close #82 #188 and #135
The text was updated successfully, but these errors were encountered: