MQTT auto discovery is entity based, not device based

[corporategoth]

The problem

The MQTT discovery protocol (with Z2M, not sure if anything else uses it) is explicitly entity based, NOT device based.

This causes a problem for devices with many entities. At least with Z2M.
I have 180 Z2M devices ... but this explodes to 12,752 /config publishes from Z2M, because the discovery protocol is entity based. ALL entities for the same device share the same availability, command and state topics.

This is kind of being tracked on the Z2M side with Koenkk/zigbee2mqtt#20648

HA should support device-based auto-configuration, where that config can just enumerate the entities involved and their types/names, and home assistant could then create/update the entities using a single MQTT message. It could even support per-entity topics if necessary, but also the entities sharing the same topic (with a single 'default_topic' field which is used if no entity-specific topic is specified in the enumeration or some such), which is currently the case.

This would reduce the number of MQTT messages being sent back and forth during discovery drastically, in my personal case, from 12752 messages down to 180! A 70x reduction! Which is huge.

This of course requires coordination with the Z2M guys to ensure both sides understand this new device-based discovery mechanism, but I believe the benefits would be significant in supporting this. And it could always fall back to entity based if necessary.

What version of Home Assistant Core has the issue?

core-2024.1.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

mqtt

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant]

Hey there @emontnemery, @jbouwh, mind taking a look at this issue as it has been labeled with an integration (mqtt) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of mqtt can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign mqtt Removes the current integration label and assignees on the issue, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

mqtt documentation
mqtt source
_{^{(message by IssueLinks)}}

[jbouwh]

I agree that would reduce the number of config messages needed. But as you already indicated, services would need to adopt to a device based discovery.

[mundschenk-at]

At least for some time (probably several years), v1 and v2 of the discovery protocol could coexist.

[jbouwh]

Sure, if we go such a route, it should be backwards compatible and coexist.

[corporategoth]

Given the syntax of the config URI with Z2M - homeassistant/light/0x6c5cb1fffe56cbd6/entity/config it would not be hard to make co-exist. Say using just homeassistant/device/0x6c5cb1fffe56cbd6/config instead. And just have the reply enumerate:

{
  name = "My Device",
  default_availability_topic = "topic",
  default_command_topic = "topic",
  default_status_topic = "topic",
  light {
    entity{
        name = "My Entity",
    }
  }
}

Then if you want a specific status/command/availability topic for an entity, you could put it under the individual entity config, but if (as in the case of Z2M) they all share the same topic, you only need to mention it once.

But yeah, making it <discovery_topic>/device/<device_id>/config should not conflict with anything else, and would not prevent backward compatibility, or supporting both at once in HA. It would then just be a flag in Z2M that they could enable "V2 discovery" or "V1 discovery", with a note that V2 is supported in HA 20xx.y.z and later only.

It's just a matter of defining the layout so that anything using the discovery protocol knows what syntax to publish.

[jbouwh]

It is clear to me what you mean. Discovery does not seem too complex, it is mainly different. I am working on something that could cope with such a discovery.

[jbouwh]

I think they should both be available as in some cases it might be needed to split. This draft PR implements device based auto discovery for MQTT:
#109030

[jbouwh]

Strictly this is not an issue with Home Assistant, but a feature request. Since there is an open PR, I do not think it is needed to keep this one open, so i'll mark it as resolved.