Usage of Domino

[isoctcolo]

Hello as far as I understood domino is primarily used for AI. Can this tool also be used for data intensive airflow applications? If this works I want to implement pieces for my internal data pipelines.

[vinicvaz]

Hey @isoctcolo
The short answer is yes.
Domino pieces can be used to run any type of processing tasks, including data intensive tasks. We provide ways to set the computation resources needs for each piece directly from the UI, including GPU access, so basically you can run anything, the just need is to have available resources in your cluster / environment.
Nice you are thinking in implementing your own pieces, let me know if you need any help with that!
Thanks

[isoctcolo]

Perfect thanks a lot. btw your domino page https://domino-py.readthedocs.io/en/latest/pages/rest.html seems to not reachable

[vinicvaz]

Oh, this docs is outdated, please use the following link:

https://domino-workflows.io/docs/intro

Just to know, where did you find the read the docs link? I'll update there, probably we missed it.
Thanks

[isoctcolo]

it is directly on the github repo page https://github.com/Tauffer-Consulting/domino?tab=readme-ov-file#rest

[luiztauffer]

@isoctcolo great to see you're interested in writing your own pieces!
If you think they might be general enough for more people to use, and decide to make them open source, let us know and we can list them in the gallery of publicly available pieces: https://domino-workflows.io/gallery

[isoctcolo]

When I have pieces they will be open sourced. I don’t know it it is possible to transfer data between pieces and not to save them?

[vinicvaz]

Yes, you can transfer data between pieces in two ways:

1. Using XCOM: This is recommended only for small data. By default all values returned by a piece, defined in the OuputModel, are returned to xcom and can be read by downstream pieces.

2. Shared Storage: this is recommended for larger files. You can configure an external provider to be the "file system" of your workflows and pieces in the same workflow can access other pieces results. If running in docker compose you can use local file system as shared storage, if running in kubernetes we currently accept AWS S3.
   Check the docs for this here

[isoctcolo]

We are running Digitalocean this has AWS S3 compatible storage. Thanks for the answers.