[query] rework flags and fix QoB flags

[danking]

Flags now use the same user configuration machinery we use for Batch and QoB. I am not certain this is the right choice. Feedback very welcome. The configuration_of function lets us uniformly treat any configuration by checking, in order: explicit argument, envvar, config file, or a fallback.

I added a bit of code to allow us to support the envvars which do not conform to the new envvar scheme.

I also removed a few flags that are no longer used.

I kind of think these flags should actually be under a new section like "query_compiler" or something.

@tpoterba, thoughts?

[danking]

I also couldn't remove the flags from Scala because of the Scala tests.

[danking]

@tpoterba this is ready for a look

[danking]

Hmm. Those two tests still fail. They don't fail on my laptop when I run them in isolation.

[danking]

bump @tpoterba this is passing tests now.

[jigold]

Can we medium prioritize this? I don't think my VEP PR will pass until flags are fixed, but I'm not 100% sure.

[danking]

@tpoterba OK, this is ready for final review. Flags are now duplicated in Python so that service backend can perform all actions without starting a JVM. I have a test that verifies the flag sets, their envvars, and the default values, are all the same. I preserved the randomness behavior. We can address that in a separate PR.

The flags now use the Hail configuration_of machinery which checks, in order:

• an explicit value (not relevant to flags)
• a deprecated environment variable (these are the current flag envvars)
• an environment variable with a mechanically derived name (e.g. HAIL_QUERY_NO_WHOLE_STAGE_CODEGEN)
• the hail configuration file (usually: "~/.config/hail/config.ini") under the section "query"

FWIW, hail configuration files look like this:

(base) dking@wm28c-761 hail % cat ~/.config/hail/config.ini 
[query]
backend = spark
jar_url = gs://hail-query/jars/dking/0wfcw2e6sma9/f4fb19e3d387d6efc6cf0f19b95bec59c95b793a.jar

[batch]
remote_tmpdir = gs://1-day/dktmp/
billing_project = hail

[tpoterba]

🎉

[danking]

This includes the changes from #12691, so I put a WIP on. I'm still not certain why this PR uncovered those issues. It's possible that I was just incredibly unlucky?

[danking]

What a helluva lifecycle for a PR. Rebased, labels removed, ready to merge. I expect it to pass.