[query] VDS combiner will not load a successfully completed plan causing combiner to be rerun when a successful run has been completed

[chrisvittal]

The constructor to the VDS Combiner has this sanity check:

hail/hail/python/hail/vds/combiner/variant_dataset_combiner.py

Lines 226 to 227 in 3e0b213

	if not (vdses or gvcfs):
	raise ValueError("one of 'vdses' or 'gvcfs' must be nonempty")

A complete combiner will not have any vdses or gvcfs present, so that sanity check will fail and the combiner will be rerun in its entirety.

It is a valid state for a VariantDatasetCombiner to have no vdses or gvcfs (when it is done), and so the fix is straightforward, remove the sanity check. A similar one already exists in new_combiner and VariantDatasetCombiner.__init__ isn't really part of the public interface.

I'm undecided if we should add a different sanity check to maybe_load_from_saved_path to see if the final file is present if the combiner is done. Though better logging will be added to that function so that the message from the exception is logged.

[danking]

Either as part of this issue or in a new issue can we also add a check that prevents deletion of a VDS whose reference and variant MTs have _SUCCESS files?

[chrisvittal]

There's nothing in the code that currently can delete the final VDS, but I will add a check for existence to the various ways of obtaining a combiner object.