Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logbook speedup #12566

Merged
merged 4 commits into from
Feb 21, 2018
Merged

Logbook speedup #12566

merged 4 commits into from
Feb 21, 2018

Conversation

amelchio
Copy link
Contributor

Description:

The logbook often turns event data into a State object twice before dropping an event.

This PR optimizes that down to zero State constructions for discarded events:

  • Some filters are moved earlier, before the final State construction
  • All filters are rewritten to work directly on event data, removing the first State construction completely

This gives me a solid 45% reduction on logbook loading so I wanted to get it out for comments even if I have not tested it much yet.

Credits to @OverloadUT for pointing me in this direction.

Related issue (if applicable): fixes #

Checklist:

  • The code change is tested and works locally.

If the code does not interact with devices:

  • Local tests with tox run successfully.
  • Tests have been added to verify that the new code works.

@@ -189,9 +189,6 @@ def humanify(events):
if event.event_type == EVENT_STATE_CHANGED:
entity_id = event.data.get('entity_id')

if entity_id is None:
continue

if entity_id.startswith(tuple('{}.'.format(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Creating a tuple for each event is also slow. We should move this out of the for loop.

if hidden:
continue

domain = to_state.domain
entity_id = to_state.entity_id

elif event.event_type == EVENT_LOGBOOK_ENTRY:
domain = event.data.get(ATTR_DOMAIN)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's confusing that this var name is the same as the one used above.

Will also mess with if statement below??

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is the same variable. We look for domain and entity_id in each kind of event and do the include/exclude filtering if one is found.

if hidden:
continue

domain = to_state.domain
entity_id = to_state.entity_id

elif event.event_type == EVENT_LOGBOOK_ENTRY:
domain = event.data.get(ATTR_DOMAIN)
entity_id = event.data.get(ATTR_ENTITY_ID)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should migrate the filters below to helpers/entity_filter.py, as that one returns an optimized function for the passed in configuration.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the excluded_domains etc. filters? Agreed, though that seems like a separate opportunity that could fit in a separate PR :)

@balloob
Copy link
Member

balloob commented Feb 21, 2018

So if you want to be real scientific, and actually come up with measurements to proof your awesome work, consider writing a benchmark. (optional)

Home Assistant benchmarks

@amelchio amelchio changed the title [RFC] Logbook speedup Logbook speedup Feb 21, 2018
@amelchio
Copy link
Contributor Author

I will try to add a benchmark tonight.

@amelchio amelchio mentioned this pull request Feb 21, 2018
@amelchio
Copy link
Contributor Author

Added a benchmark that is 35% faster for events that we keep and 90% faster for dropped events. I don't think we should treat this as too scientific but it definitely seems worth the trouble.

@balloob balloob merged commit f9ee29a into home-assistant:dev Feb 21, 2018
@amelchio amelchio mentioned this pull request Feb 22, 2018
3 tasks
@balloob balloob mentioned this pull request Feb 22, 2018
@home-assistant home-assistant locked and limited conversation to collaborators May 29, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants