Allow MQTT device based auto discovery #118757

jbouwh · 2024-06-03T21:19:09Z

Proposed change

Rework of #109030 which introduces a new device based MQTT discovery schema.

The original PR was reverted during the beta of HA Core 2024.6.0. With PR and branch we aim to test and optimize large MQTT setups that run via MQTT discovery.

The new device based discovery payload allows multiple mqtt components to set up with one configuration payload.

Context

The current MQTT discovery model only allows to setup per component, if a device has multiple entities to published, the device context, availability and origin information needs to be duplicated.

This PR reduces IO overhead in the paho client on processing discovery packages: The baseline was 10000 packets, with device discovery it 6250 packet reads so a 37.5% reduction in I/O. Processing less I/O reduced the paho client run time by ~18%. The grouped discovery reduced the run time in the HA client by ~12%.

This PR adds a way to discover all components for a device with one discovery. The device, availability and origin mapping is only submitted once. Also the following options are allowed in the shared context:

state_topic
command_topic
encoding
qos

Except for device and origin options, supported shared options can be overridden in the component config.

Discovery updates and removal is fully supported.

The device based discovery coexists with the component based discovery, so no deprecation is planned for discovery with the single component schema to guarantee older devices keep working.

Note that some code and schemas that are related to device, availability and origin validation was moved to a new module schemas.py.

Example

The example below is of a device based auto discovery that supplies a sensor, a binary_sensor and a tag.

Discovery topic:

homeassistant/device/test123/config

Payload:

{
  "dev": {
    "identifiers": [
      "mysensors123"
    ]
  },
  "avty": {
    "topic": "avty-topic"
  },
  "cmp": {
    "bins1": {
      "platform": "binary_sensor",
      "unique_id": "bins1_key",
      "name": "Beer",
      "state_topic": "test-topic"
    },
    "sens1": {
      "platform": "sensor",
      "unique_id": "sens1_key",
      "name": "Milk",
      "state_topic": "test-topic"
    },
    "tag1": {
      "platform": "tag",
      "topic": "foobar/tags/tag1/see"
    }
  },
  "o": {
    "name": "My org",
    "sw": "bla2mqtt"
  },
  "qos": 0
}

Note that a required platform option is added to identify the component platform. The keys under the components (abbreviated cmp), are treated as object_id and are used to create a unique device discovery_id. In this case the discover_id's are test123 bins1 and test123 sens1.
For entity components defined in the device based configuration payload the unique_id is required.

homeassistant/device/node123/test123/config

... this will become part of the object_id.
The discovery_id 's become test123 node123 bins1 and test123 node123 sens1.

Migration from single topic to device based discovery

To allow a smooth migration from single topic discovery to device based discovery, the single platform platform discovery configs must have a unique_id and device config. Note that migration is only supported if the entities in the device based payload have the same unique_id and device identifier.

To allow a take over via the device based discovery all included items must be made ready for migration first by publishing the {"migrate_discovery": true } payload to the existing platform based discovery topics. This will allow migration and clearing the migrated discovery topics without removing the registry settings with possible customization's. Migration is supported in both directions. When a rollback is intended {"migrate_discovery": true } is to be published to the device based discovery topic, and then included components can be migrated back to single component discovery topics.

Example (device automation and 2 sensors):

Discovery topic single: `homeassistant/device_automation/0AFFD2/bla1/config`

Discovery id: 0AFFD2 bla1
Discovery payload single:

{
  "automation_type": "trigger",
  "device": {
    "identifiers": [
      "0AFFD2"
    ]
  },
  "o": {
    "name": "foobar"
  },
  "payload": "short_press",
  "topic": "foobar/triggers/button1",
  "type": "button_short_press",
  "subtype": "button_1"
}

Discovery topic single: `homeassistant/sensor/0AFFD2/bla2/config`

Discovery id: 0AFFD2 bla2
Discovery payload single:

{
  "automation_type": "trigger",
  "device": {
    "identifiers": [
      "0AFFD2"
    ]
  },
  "o": {
    "name": "foobar"
  },
  "state_topic": "foobar/bla2/state",
  "unique_id": "bla2"
}

Migrate to a one device config discovery payload:

First enable discovery migration:
Publish:

{"migrate_discovery": true }

To the topics:

homeassistant/device_automation/0AFFD2/bla1/config
homeassistant/sensor/0AFFD2/bla2/config

Now the new device based discovery payload must be send to migrate the discovery topic.

Discovery topic device: homeassistant/device/0AFFD2/config
Discovery id: 0AFFD2
Discovery payload device:

{
  "device": {
    "identifiers": [
      "0AFFD2"
    ]
  },
  "o": {
    "name": "foobar"
  },
  "cmp": {
    "bla1": {
      "automation_type": "trigger",
      "payload": "short_press",
      "topic": "foobar/triggers/button1",
      "type": "button_short_press",
      "subtype": "button_1",
      "platform": "device_automation"
    },
    "bla2": {
      "state_topic": "foobar/bla2/state",
      "platform": "sensor",
      "unique_id": "bla2"
    }
  },
}

Note that in this example the node_id becomes the object_id in the device based discovery topic, and the object_id from the single component is placed as a key under the components (cmp) key. The device part is now shared and all entities have a unique_id.

The source integration, responsible for the entities and the migration, is also responsible for cleaning up the old discovery topics after the migration has been completed. This the source integration should publish empty payloads to the old configuration topics, if these topic are retained. Note that retained configs must be cleared with a retained empty payload.

In the example a retained empty payload is to be published to the topics:

homeassistant/device_automation/0AFFD2/bla1/config
homeassistant/sensor/0AFFD2/bla2/config

Each migration step is logged as an info message to the HA Core logs. This allow you to check that each step was executed correctly.

Removing the device or components

Updating works the same as with single component based discovery.
To remove all components at once a (retained) empty payload must be send to the discovery topic homeassistant/device/0AFFD2/config.

To remove a component from an existing example configuration, only the platform key must be specified.
To remove the device_automation component from the the example discover publish:

{
  "device": {
    "identifiers": [
      "0AFFD2"
    ]
  },
  "o": {
    "name": "foobar"
  },
  "cmp": {
    "bla1": {
      "platform": "device_automation"
    },
    "bla2": {
      "state_topic": "foobar/bla2/state",
      "platform": "sensor",
      "unique_id": "bla2"
    }
  },
}

Type of change

Dependency upgrade
Bugfix (non-breaking change which fixes an issue)
New integration (thank you!)
New feature (which adds functionality to an existing integration)
Deprecation (breaking change to happen in the future)
Breaking change (fix/feature causing existing functionality to break)
Code quality improvements to existing code or addition of tests

Additional information

This PR fixes or closes issue: fixes #
This PR is related to issue:
Link to documentation pull request: Add device based MQTT auto discovery home-assistant.io#33072

Checklist

The code change is tested and works locally.
Local tests pass. Your PR cannot be merged unless tests pass
There is no commented out code in this PR.
I have followed the development checklist
I have followed the perfect PR recommendations
The code has been formatted using Ruff (ruff format homeassistant tests)
Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

The manifest file has all fields filled out correctly.
Updated and included derived files by running: python3 -m script.hassfest.
New or updated dependencies have been added to requirements_all.txt.
Updated by running python3 -m script.gen_requirements_all.
For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
Untested files have been added to .coveragerc.

To help with the load of incoming pull requests:

I have reviewed two other open pull requests in this repository.

home-assistant · 2024-06-03T21:19:18Z

Hey there @emontnemery, @bdraco, mind taking a look at this pull request as it has been labeled with an integration (mqtt) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of mqtt can trigger bot actions by commenting:

@home-assistant close Closes the pull request.
@home-assistant rename Awesome new title Renames the pull request.
@home-assistant reopen Reopen the pull request.
@home-assistant unassign mqtt Removes the current integration label and assignees on the pull request, add the integration domain after the command.
@home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
@home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

corporategoth · 2024-06-06T17:24:06Z

Since this is difficult to read for someone not involved. Does this still support specifying topics device-wide (ie. same for all entities), and then overriding them on a per-entity basis if required?

jbouwh · 2024-06-06T20:30:48Z

Since this is difficult to read for someone not involved. Does this still support specifying topics device-wide (ie. same for all entities), and then overriding them on a per-entity basis if required?

Effectively the PR is the same is the previous PR, but we love to seem some test results. We will try to update this branch as good as possible. A documentation PR is linked.

homeassistant/components/mqtt/debug_info.py

homeassistant/components/mqtt/discovery.py

homeassistant/components/mqtt/tag.py

home-assistant

There is a merge conflict.

home-assistant · 2024-08-20T15:17:53Z

Please take a look at the requested changes, and use the Ready for review button when you are done, thanks 👍

Learn more about our pull request process.

tests/components/mqtt/test_discovery.py

emontnemery

A few nits to pick.

homeassistant/components/mqtt/const.py

homeassistant/components/mqtt/discovery.py

homeassistant/components/mqtt/entity.py

Co-authored-by: Erik Montnemery <erik@montnemery.com>

jbouwh · 2024-10-11T15:08:30Z

tests/components/mqtt/test_tag.py

-@pytest.fixture
-def tag_mock() -> Generator[AsyncMock]:
-    """Fixture to mock tag."""
-    with patch("homeassistant.components.tag.async_scan_tag") as mock_tag:
-        yield mock_tag
-
-
 @pytest.mark.no_fail_on_log_exception


This was moved because the fixture was needed for some added discovery tests with device-based discovery

jbouwh · 2024-10-11T15:25:57Z

tests/components/mqtt/test_init.py

@@ -1197,7 +1197,6 @@ async def test_mqtt_ws_get_device_debug_info(
    }
    data_sensor = json.dumps(config_sensor)
    data_trigger = json.dumps(config_trigger)
-    config_sensor["platform"] = config_trigger["platform"] = mqtt.DOMAIN


This key is no longer added, see:
https://github.com/home-assistant/core/pull/118757/files#diff-189ecd07b98c2a9602a647f97dfe63e919c95af2d3856701044958b230591ef1L274

jbouwh · 2024-10-11T15:26:32Z

tests/components/mqtt/test_init.py

@@ -1254,7 +1253,6 @@ async def test_mqtt_ws_get_device_debug_info_binary(
        "unique_id": "unique",
    }
    data = json.dumps(config)
-    config["platform"] = mqtt.DOMAIN


The platform is is no longer added to the discovery message, see:
https://github.com/home-assistant/core/pull/118757/files#diff-189ecd07b98c2a9602a647f97dfe63e919c95af2d3856701044958b230591ef1L274

home-assistant bot added cla-signed core has-tests integration: mqtt new-feature labels Jun 3, 2024

home-assistant bot assigned emontnemery and bdraco Jun 3, 2024

home-assistant bot added by-code-owner Quality Scale: platinum labels Jun 3, 2024

bdraco mentioned this pull request Jun 3, 2024

Allow MQTT device based auto discovery #109030

Merged

20 tasks

This was referenced Jun 4, 2024

Allow MQTT device based auto discovery #118750

Closed

Revert "Allow MQTT device based auto discovery" #118746

Merged

jbouwh mentioned this pull request Jun 4, 2024

Add device based MQTT auto discovery home-assistant/home-assistant.io#33072

Open

8 tasks

This was referenced Jun 6, 2024

[Feature Request] Adding firmware version to Home Assistant technyon/nuki_hub#395

Closed

[Feature request]: Incorporate multi-threading Koenkk/zigbee2mqtt#20648

Closed

jbouwh force-pushed the mqtt-device-based-discovery branch from 9d516dd to 317bb4b Compare June 6, 2024 21:10

jbouwh force-pushed the mqtt-device-based-discovery branch 2 times, most recently from bdb0bf7 to 85d2546 Compare June 25, 2024 14:19

jbouwh commented Jun 25, 2024

View reviewed changes

homeassistant/components/mqtt/debug_info.py Outdated Show resolved Hide resolved

homeassistant/components/mqtt/discovery.py Outdated Show resolved Hide resolved

homeassistant/components/mqtt/tag.py Outdated Show resolved Hide resolved

jbouwh force-pushed the mqtt-device-based-discovery branch from ffd153b to 51fda9a Compare June 27, 2024 18:44

jbouwh marked this pull request as ready for review August 16, 2024 11:58

jbouwh requested review from emontnemery and bdraco as code owners August 16, 2024 11:58

jbouwh force-pushed the mqtt-device-based-discovery branch from fbc9829 to d23973b Compare August 20, 2024 15:17

home-assistant bot previously requested changes Aug 20, 2024

View reviewed changes

home-assistant bot marked this pull request as draft August 20, 2024 15:17

jbouwh commented Aug 22, 2024

View reviewed changes

tests/components/mqtt/test_discovery.py Outdated Show resolved Hide resolved

jbouwh added 6 commits September 19, 2024 19:22

Allow MQTT device based auto discovery

2445489

Fix merge error

3fa1c76

Remove unused import

f1309d7

Fix discovery device based topics

1d543d3

Fix cannot delete twice

2002991

Improve cleanup test

b280d9a

jbouwh force-pushed the mqtt-device-based-discovery branch from 5c7ee65 to b280d9a Compare September 19, 2024 19:23

Merge branch 'dev' into mqtt-device-based-discovery

644d636

emontnemery reviewed Oct 1, 2024

View reviewed changes

jbouwh requested a review from emontnemery October 4, 2024 17:59

jbouwh and others added 3 commits October 4, 2024 18:07

Follow up comment

2795809

Typo

d385e25

Co-authored-by: Erik Montnemery <erik@montnemery.com>

Explain more

bc932a5

jbouwh added Hacktoberfest and removed Hacktoberfest labels Oct 4, 2024

Use tuple

c68f4b1

jbouwh marked this pull request as draft October 7, 2024 15:44

jbouwh and others added 3 commits October 7, 2024 17:45

Merge branch 'dev' into mqtt-device-based-discovery

fc47f6f

Default a device payload to have priority over a platform based payload

1b5b16c

Merge branch 'dev' into mqtt-device-based-discovery

6689d4b

jbouwh marked this pull request as ready for review October 8, 2024 06:18

jbouwh added 5 commits October 9, 2024 09:01

Add unique_id to sensor test data

0be4c24

Set migration flag to mark a discovery topic for migration

a22b843

Correct type hint

4ba2664

Make unique_id required for components in device based discovery payload

efe4a53

Remove CONF_MIGRATE_DISCOVERY from platform schema

42a9224

jbouwh commented Oct 11, 2024

View reviewed changes

jbouwh and others added 2 commits October 15, 2024 20:47

Unload discovered MQTT item to allow migration

0d59336

Merge branch 'dev' into mqtt-device-based-discovery

d957963

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow MQTT device based auto discovery #118757

Allow MQTT device based auto discovery #118757

jbouwh commented Jun 3, 2024 •

edited

Loading

home-assistant bot commented Jun 3, 2024

corporategoth commented Jun 6, 2024

jbouwh commented Jun 6, 2024 •

edited

Loading

home-assistant bot left a comment

home-assistant bot commented Aug 20, 2024

emontnemery left a comment

jbouwh Oct 11, 2024

jbouwh Oct 11, 2024

jbouwh Oct 11, 2024

Allow MQTT device based auto discovery #118757

Are you sure you want to change the base?

Allow MQTT device based auto discovery #118757

Conversation

jbouwh commented Jun 3, 2024 • edited Loading

Proposed change

Context

Example

Migration from single topic to device based discovery

Example (device automation and 2 sensors):

Discovery topic single: homeassistant/device_automation/0AFFD2/bla1/config

Discovery topic single: homeassistant/sensor/0AFFD2/bla2/config

Migrate to a one device config discovery payload:

Removing the device or components

Type of change

Additional information

Checklist

home-assistant bot commented Jun 3, 2024

corporategoth commented Jun 6, 2024

jbouwh commented Jun 6, 2024 • edited Loading

home-assistant bot left a comment

Choose a reason for hiding this comment

home-assistant bot commented Aug 20, 2024

emontnemery left a comment

Choose a reason for hiding this comment

jbouwh Oct 11, 2024

Choose a reason for hiding this comment

jbouwh Oct 11, 2024

Choose a reason for hiding this comment

jbouwh Oct 11, 2024

Choose a reason for hiding this comment

jbouwh commented Jun 3, 2024 •

edited

Loading

Discovery topic single: `homeassistant/device_automation/0AFFD2/bla1/config`

Discovery topic single: `homeassistant/sensor/0AFFD2/bla2/config`

jbouwh commented Jun 6, 2024 •

edited

Loading