Impossible to override default config with DASK_INTERNAL_INHERIT_CONFIG

[jacobtomlinson]

What happened:

When passing config with DASK_INTERNAL_INHERIT_CONFIG it appears it is not possible to override the default values.

What you expected to happen:

I expected the config passed with DASK_INTERNAL_INHERIT_CONFIG to be applied.

Minimal Complete Verifiable Example:

Generate config

>>> from distributed.utils import serialize_for_cli
>>> serialize_for_cli({"distributed": {"comm": {"default-scheme": "tls"}}})
'eyJkaXN0cmlidXRlZCI6IHsiY29tbSI6IHsiZGVmYXVsdC1zY2hlbWUiOiAidGxzIn19fQ=='

Start scheduler with DASK_INTERNAL_INHERIT_CONFIG.

$ DASK_INTERNAL_INHERIT_CONFIG="eyJkaXN0cmlidXRlZCI6IHsiY29tbSI6IHsiZGVmYXVsdC1zY2hlbWUiOiAidGxzIn19fQ==" dask-scheduler

Connect a client and check the config.

>>> from dask.distributed import Client
>>> import dask.config
>>> client = Client("tcp://localhost:8786")
>>> client.run_on_scheduler(lambda: dask.config.get("distributed.comm.default-scheme"))
'tcp'

Anything else we need to know?:

It appears that existing config is prioritised, even if the existing config is just detected from the defaults.

distributed/distributed/cli/dask_scheduler.py

Lines 170 to 173 in f2b09c6

	if "DASK_INTERNAL_INHERIT_CONFIG" in os.environ:
	config = deserialize_for_cli(os.environ["DASK_INTERNAL_INHERIT_CONFIG"])
	# Update the global config given priority to the existing global config
	dask.config.update(dask.config.global_config, config, priority="old")

Environment:

• Dask version: 2020.12.0+6.g1d669480 (master)
• Distributed version: 2020.12.0+7.gc2d87738 (master)
• Python version: 3.8.5
• Operating System: macOS
• Install method (conda, pip, source): Source

[jacobtomlinson]

cc @madsbk