Fix time-based statistics (peaks, incorrect values)

[unfug-at-github]

Fixing issues with time-based statistics (peaks, incorrect values) and provide values for statistical function even if only a single input value exists

Breaking change

Proposed change

The statistics functions has issues with the time based functions:

• Currently, time-based functions (average step / linear) don't compute a proper average over the specified time interval. Instead the average is only computed between the first and last change within the interval. This produces unexpected results when values are not changing over a longer period.
• Currently, the computation of the time-based functions is only triggered when input values change or values are dropping out of the age range. However, the actual average changes over time, because the time interval is shifting.

The changes will fix the above mentioned issues. To do so, they add a new parameter "refresh interval" that allows to updated the computed averages even if the input states are not changing.

Type of change

• Dependency upgrade
• Bugfix (non-breaking change which fixes an issue)
• New integration (thank you!)
• New feature (which adds functionality to an existing integration)
• Deprecation (breaking change to happen in the future)
• Breaking change (fix/feature causing existing functionality to break)
• Code quality improvements to existing code or addition of tests

Additional information

• This PR fixes or closes issue: fixes Spikes in statistics averages when HA restarts #119738 Statistics sensor logs bad data right after HA restart #98262 Statistics sensor doesn't handle zero values correctly #67627
• This PR is related to issue:
• Link to documentation pull request:

Checklist

• The code change is tested and works locally.
• Local tests pass. Your PR cannot be merged unless tests pass
• There is no commented out code in this PR.
• I have followed the development checklist
• I have followed the perfect PR recommendations
• The code has been formatted using Ruff (ruff format homeassistant tests)
• Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

• The manifest file has all fields filled out correctly.
  Updated and included derived files by running: python3 -m script.hassfest.
• New or updated dependencies have been added to requirements_all.txt.
  Updated by running python3 -m script.gen_requirements_all.
• For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.

To help with the load of incoming pull requests:

• I have reviewed two other open pull requests in this repository.

[home-assistant]

Hi @unfug-at-github

It seems you haven't yet signed a CLA. Please do so here.

Once you do that we will be able to review and accept this pull request.

Thanks!

[home-assistant]

Please take a look at the requested changes, and use the Ready for review button when you are done, thanks 👍

Learn more about our pull request process.

[home-assistant]

Hey there @ThomDietrich, mind taking a look at this pull request as it has been labeled with an integration (statistics) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of statistics can trigger bot actions by commenting:

• @home-assistant close Closes the pull request.
• @home-assistant rename Awesome new title Renames the pull request.
• @home-assistant reopen Reopen the pull request.
• @home-assistant unassign statistics Removes the current integration label and assignees on the pull request, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the pull request.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the pull request.

Stale

[ThomDietrich]

Hello @unfug-at-github,
its generally not a good idea to mix so many changes in one PR.
Still, please give me some time to review, this looks like some good additions! Thanks for the contribution :)

[unfug-at-github]

Hello @unfug-at-github, its generally not a good idea to mix so many changes in one PR. Still, please give me some time to review, this looks like some good additions! Thanks for the contribution :)

I agree. I should have at least put the fixes for the average and the changes for the "compute with only one value" functions into different requests.

With the average functions it's a little harder to split things up, because these changes are somewhat depending on each other.

Let me know if things need to be put into separate requests. It's my first contribution here, and I have to admit that I only read some of the how-tos after I fixed all the issues that I had with my solar management solution.

[Zamtakk]

I could really use this fix from this PR, @ThomDietrich could you give an estimate on when you have time to approve this PR?

[joostlek]

I think this PR should try to split up more, for me as a reviewer its not clear why we exactly change what since this PR tries to solve 4 bugs, so I can't discover if this is the proper way to solve the bug.

Also, there is a merge conflict.

[unfug-at-github]

I have created a separate pull request that only addresses the issue of the spikes (#127268). I'll add further ones to reduce the scope of this one.

[unfug-at-github]

Added another PR for the single value case #127305

[home-assistant]

Hello @unfug-at-github,

When attempting to inspect the commits of your pull request for CLA signature status among all authors we encountered commit(s) which were not linked to a GitHub account, thus not allowing us to determine their status(es).

The commits that are missing a linked GitHub account are the following:

• 3bfdebee41ca6e7af537b065bd5218a194aef8a3 - This commit has something that looks like an email address (dr.dirk.schmidt@web.de). Maybe try linking that to GitHub?.

Unfortunately, we are unable to accept this pull request until this situation is corrected.

Here are your options:

1. If you had an email address set for the commit that simply wasn't linked to your GitHub account you can link that email now and it will retroactively apply to your commits. The simplest way to do this is to click the link to one of the above commits and look for a blue question mark in a blue circle in the top left. Hovering over that bubble will show you what email address you used. Clicking on that button will take you to your email address settings on GitHub. Just add the email address on that page and you're all set. GitHub has more information about this option in their help center.

2. If you didn't use an email address at all, it was an invalid email, or it's one you can't link to your GitHub, you will need to change the authorship information of the commit and your global Git settings so this doesn't happen again going forward. GitHub provides some great instructions on how to change your authorship information in their help center.

   • If you only made a single commit you should be able to run

     git commit --amend --author="Author Name <email@address.com>"
     

     (substituting "Author Name" and "email@address.com" for your actual information) to set the authorship information.
   • If you made more than one commit and the commit with the missing authorship information is not the most recent one you have two options:
     1. You can re-create all commits missing authorship information. This is going to be the easiest solution for developers that aren't extremely confident in their Git and command line skills.
     2. You can use this script that GitHub provides to rewrite history. Please note: this should be used only if you are very confident in your abilities and understand its impacts.
   • Whichever method you choose, I will come by to re-check the pull request once you push the fixes to this branch.

We apologize for this inconvenience, especially since it usually bites new contributors to Home Assistant. We hope you understand the need for us to protect ourselves and the great community we all have built legally. The best thing to come out of this is that you only need to fix this once and it benefits the entire Home Assistant and GitHub community.

Thanks, I look forward to checking this PR again soon! ❤️

[unfug-at-github]

I created a new PR to fix the async issues in general #128796