Possibility of deploying to one central composer for multiple projects

[faderskd]

The problem

We'd like to create a deployment with multiple independent dag groups. The specific group of dags is actually connected to the specific env. To save few bucks on GCP it would be convenient to have a one central composer for managing various tasks in different projects. In our case, one composer for each env: dev/test/prod. The current solution doesn't allow such a setup because there is no way to deploy group of dags. The only option to replace dags with newer version is to pass the --clear-dags-folder arg to the bf deploy-dags command. It clears all currently deployed dags and uploads the new ones.

Possible solution

To enable deployment only for specific group of dags e.g. dev/test/prod dags we would like to introduce the concept of dags grouping.

1. Add metadata to the generated dags files. Each dag should contain some indicator(s) to which group it belongs to. In our case, the group id is env: dev/test/prod
2. Pass the group id variable: --group-id dev during deployment: bf deploy-dags --group-id dev
3. Deployment deletes dags only within this group and replaces them with the new ones.
4. The --clear-dags-folder option could be removed. We can make it just a dummy option for backward compatibility (bigflow options parses raises error if invalid options are given)
5. The concept of groups would allow to grouping by different projects too, e.g.: bf deploy-dags --group-id {some-project-id}:{dev}

[faderskd]

There are three problems to solve:

1. How to generate specific group of dags ? We can generate all dags upfront. There should be as many dags as many different configurations. So let's say I'd like to create dags doing some operations on two different GCP projects projA, projB. I'd need to manually create configs:

current_workflow_config_projA = resolve_konfig(
    {
        'dev_projA': DevWorkflowProjAConfig,
        'test_projA': TestWorkflowProjAConfig,
        'prod_projA': ProdWorkflowProjAConfig
    },
    default='dev_projA')

current_workflow_config_projB = resolve_konfig(
    {
        'dev_projB': DevWorkflowProjBConfig,
        'test_projB': TestWorkflowProjBConfig,
        'prod_projB': ProdWorkflowProjBConfig
    },
    default='dev_projB')

The build-dags command will generate: 6 files upfront. The current bigflow version generates dags based on bf_env environment variable. We can remove it and have always the same behaviour: generate all possible dags. If we want to preserve it, we should have an option to differentiate between the case when we want to generate dag for specific env or group of envs, e.g additional bf_envs variable. We can add a command param too: --configs dev_projA, dev_projB,....
The generated dags will contain metadata in its files:

# This file was generated by `bigflow build-dags`
# bigflow-workflow:  	...
# bigflow-build-ver:    ...
# bigflow-startdate: 	...
# biglfow-imageid:   	...
# config:               dev_projA  <----------- New field

2. How to select specific group of dags to deploy ? We should clearly differentiate between config/environment during dags building and dags deploying. The bf_env or --config variables designates deployment config when calling deploy-dags command. We need additional variable to select specific group of dags: bf deploy-dags --dags-configs dev_projA, dev_projB,.... When the option is empty it behaves like before for backward compatibility.
3. Stop rely on '--config', '{{var.value.env}}' in dags' KubernetesPodOperator definitions. Bigflow should set them upfront during dags generation.

[faderskd]

And do you see any potential use cases for having multiple DAG groups in a single project?

Yes, you can use one GCP project to share airflow tasks for multiple other GCP projects. Our use case is to have a one composer for all environments: dev/test/prod. It gives us the following advantages:

1. Saving money - we often create composer to perform only one periodic task. Most of the time it is idle, but we're paying for the resources it is consuming. If you have few GCP projects like that it is just a waste. The more efficient solution is to have a one composer doing multiple tasks.
2. Saving time - each time we need a composer, we have to go through a long way to create a new one.

[beczkowb]

And do you see any potential use cases for having multiple DAG groups in a single project?

Yes, you can use one GCP project to share airflow tasks for multiple other GCP projects. Our use case is to have a one composer for all environments: dev/test/prod. It gives us the following advantages:

1. Saving money - we often create composer to perform only one periodic task. Most of the time it is idle, but we're paying for the resources it is consuming. If you have few GCP projects like that it is just a waste. The more efficient solution is to have a one composer doing multiple tasks.
2. Saving time - each time we need a composer, we have to go through a long way to create a new one.

And do you see any potential use cases for having multiple DAG groups in a single project, besides lowering the number of composers (which is a great argument)?*

I just wonder if there are some other use cases for this DAG group thing. For example, I can come up with one: being able to divide a bigflow project into DAG groups, allows you to keep multiple business projects inside a single bigflow project (monorepo).

[beczkowb]

Same thing to environments, how the build system will find available configurations?

The available configuration will be searched the same way as now: by searching Konfig classes. The only additional logic is for defining groups from the configurations.

Currently, Konfig classes have nothing to do with dag generation or build/deployment process in general. I think we should keep it that way. But if you really want to use configuration to implement the concept of groups, then you should also implement that in bigflow.config which is used in every single project implicitly and explicitly.

I suggest not touching configuration utils at all and instead, adding some tooling to bigflow.workflow, because it is the toolset for describing and generating DAGs, so it looks like the perfect place for implementing DAG groups, and solving the "bf_env" problem, I suggest adding some explicit tool, for example (it's pseudocode):

import bigflow.workflow as workflow

some_workflow = workflow.Workflow(
    id="blabla", 
    definition=[bla_job],
    environments=['dev_project1', 'prod_project1'], # will generate 2 DAGs, 1 for each env, setting bf_env to either dev_project1 or prod_project1
    dag_group='project1',
)

bf build-dags --group project1
bf deploy-dags --group project1 --env dev_project1

Something like that, wdyt?

[faderskd]

Agree, looks cool, what about backward compatibility ? If the group is not given it should take some defaults for dag_group and environments too. We can set them to None in such a case and treat it as a disabled grouping mode too.

[beczkowb]

If you are ok with the idea in general, then @alicja-nowak-allegro will prepare a detailed design description.