feat: variant-index refactoring

[project-defiant]

Context

This PR closes opentargets/issues#3431

The main development being done was moving the code from variant_index dag to the genetics_etl dag to port new tasks (wrapped into variant annotation task group) that are required to run before the variantIndexStep can be successfully executed.

Things implemented:

• Operator for annotating variant source with VEP command
• Operator for partitionning the variant sources into chunks required by VEP batch job
• TaskGroup in the configuration
• Addition of biosample index step
• New filters for validation steps
• Deprecation of old hydra configuration

Limitations

The orchestration of variant_index and variant_annotation currently runs as:

1. convert variant source to vcf (batch job) - 4 tasks | ~ 4min
2. sort and partition vcfs into small chunks (python operator) - partitionned to 850 vcfs | very heavy in computations on local machine that runs airflow and requires pandas as a dependency ~ 3min
3. annotate partitionned vcfs (batch job) - 850 tasks ~ 25 min
4. perform variantIndexStep (merge variants from gnomad and variant sources) ~25min

The steps 1 and 2 should be merged and run as a dataproc job (the 1. requires only 4 batch tasks) for silplifying the configuration, although this is out of scope of this PR.

Test perfomed on successful run of the pipeline ending at the variant_index step (including) resulted in ~1h10min run

[javfg]

I think this is a good start!

I haven't gone in detail into it. As we discussed yesterday, we should try to figure out a way to separate concerns in a similar way as it is being done in the platform, with a configuration file for the pipeline itself and other files for the settings for the internals of running an application inside a step.

We could do this by either overriding settings in a hierarchical way (tool setting < pipeline settings), by templating, or by trying to isolate settings in one of the two places.

I realize this is harder in Gentropy due to it being cloud-agnostic. That is a target for the platform part too. Even if it's impossible to do completely, we should strive for it. I think this separation will enable us to separate concerns in the orchestrator code too.