Statistics sensor doesn't handle zero values correctly

[Flox-Muc]

The problem

As you can see in the picture below the statistics sensor seems to ignore 0 values. Both "average linear" and "average step" types jump to higher values as soon as the source sensor (in the picture the graph at the bottom) has a value of "0". The dark blue line in the upper graph ("Windgeschwindigkeit mit Jitter") shows how the "average step"-sensor should behave correctly. (It is created using an automation that adds an artificial jitter to the original value every second.)

What version of Home Assistant Core has the issue?

core-2022.2.2

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Core

Integration causing the issue

Statistics

Link to integration documentation on our website

https://www.home-assistant.io/integrations/statistics/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[probot-home-assistant]

statistics documentation
statistics source
_{^{(message by IssueLinks)}}

[probot-home-assistant]

Hey there @fabaff, @ThomDietrich, mind taking a look at this issue as it has been labeled with an integration (statistics) you are listed as a code owner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[ThomDietrich]

Hey @Flox-Muc,
I can see the issue but am not quite clear on the reason for it. The spike-jumps are strange, I am baffled where they come from.
Could you share a few more details:

• how many wind measurement updates are in those zero blocks (e.g. how many samples in the time between 0:00 and 8:00)?
• please share the configuration of your sensor, especially the limits.
• How does your graph look for average_timeless?
• Could you share your automation? Maybe there is a piece of logic in your code, specific to the quirks of your sensor, that should be added as an improvement to the statistics component.

Unrelated: I have other averaging characteristics on my todo list, so irrespective of the issue here, look out for those.

[Flox-Muc]

Thank you for your quick response! And sorry that I didn't answer earlier. I probably have an idea what the problem could be. Take a look at my simplified example below:

The blue line represents the sensor. The red dots are the stored values in the database.

The green line represents the average of the last ten "hours" at "time" 10.5.
If my thinking is correct it is calculated by the "average step" algorithm in this way (please look especially at the first value):
(5.5 * 0 + 1 * 5 + 1 * 3 + 2.5 * 0) / 10 = 0.8
The exact average would be:
(0.5 * 4 + 5 * 0 + 1 * 5 + 1 * 3 + 2.5 * 0) / 10 = 1
So at that moment it's pretty accurate.

Now look at the orange line at "time" 11.5:
Average step: (5.5 * 5 + 1 * 3 + 3.5 * 0) / 10 = 3.05 (as the data point at time 1 falls out of the range)
Correct average: (4.5 * 0 + 1 * 5 + 1 * 3 + 3.5 * 0) / 10 = 0.8

The error gets bigger the longer a sensor value is unchanged and the more consecutive stored values differ - so it's probably not a special problem with the zero.

[Flox-Muc]

* how many wind measurement updates are in those zero blocks (e.g. how many samples in the time between 0:00 and 8:00)?

Probably none. The wind sensor gets its value via the Ecowitt integration every minute. But as the value is unchanged Home Assistant doesn't store any additional data point in the database, I think. Can you tell me how to extract the samples of the database? Then I could attach it.

* please share the configuration of your sensor, especially the limits.

The wind sensor from the Ecowitt integration is named "sensor.ws_windgeschwindigkeit".

The two statistics sensors that show the error:

• platform: statistics
  name: ws_windgeschwindigkeit_durchschnitt_tag_test
  entity_id: sensor.ws_windgeschwindigkeit
  state_characteristic: average_linear
  sampling_size: 1440
  max_age:
  hours: 24

• platform: statistics
  name: ws_windgeschwindigkeit_durchschnitt_tag_test_AS
  entity_id: sensor.ws_windgeschwindigkeit
  state_characteristic: average_step
  sampling_size: 1440
  max_age:
  hours: 24

The sensor that I use to create the correct average:

• platform: statistics
  name: ws_windgeschwindigkeit_durchschnitt_tag
  entity_id: input_number.ws_windgeschwindigkeit_jitter
  sampling_size: 1440
  max_age:
  hours: 24

The "input_number" is constantly changed by this automation that creates the mentioned jitter which forces Home Assistant to evenly store values:

• alias: trigger_1min
  id: A0014
  trigger:
  • platform: time_pattern
    minutes: /1
    action:
  • service: input_number.set_value
    data_template:
    entity_id: input_number.ws_windgeschwindigkeit_jitter
    value: "{%- if ((now().strftime("%M") | int) % 2) -%}\n {{ ((states('sensor.ws_windgeschwindigkeit')
    \ | float) + 0.01) | round(2) }}\n{%- else -%}\n {{ ((states('sensor.ws_windgeschwindigkeit')
    \ | float) - 0.01) | round(2) }}\n{%- endif -%}\n"

* How does your graph look for `average_timeless`?

I just created such a sensor - so I probably can tell you in a few days.

[Flox-Muc]

Now I can show you the "average timeless" graph, too:

[marcobe]

I am having a similar issue, values are incorrect. Also when I do a avg over 24 hrs and 7 days, you would say that the averages should be different, however they are the same. Something's very off.

[exus85]

I have the same issue with a solar radiation sensor, during the night my statistic sensor reports wrong value

• platform: statistics
  name: outdoor solar radiation 24h
  entity_id: sensor.solar_radiation
  state_characteristic: average_linear
  sampling_size: 1500
  max_age:
  hours: 24

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[ThomDietrich]

Still relevant.

[ThomDietrich]

Hey guys, just a quick update from my side.
Honestly speaking, I didn't find much time to look into this so far. Appologies.

Based on what was discussed here and in other places, these are the conclusions I drew:

• Left time window bound: The issue described in Statistics sensor doesn't handle zero values correctly #67627 (comment) is that values outside of the given time windows are not considered. For all previous timeless characteristics this was correct. As I introduced time-driven characteristics I did not change this limitations, which leads to weird jumps experienced by you guys. Please let me know if that summary is correct.
  @Flox-Muc from your example it became clear that the youngest value outside of the time window needs to be considered to include 0.5 * 4 for your example. The implementation currently ignores this area, hence the correct first formula would be (5.0 * 0 + 1 * 5 + 1 * 3 + 2.5 * 0) / 9.5 = 0.842
• Right time window bound: Beware, that the statistics value is only calculated upon source sensor update. @Flox-Muc your example diagram would be more correct with two red dots where your green and orange brackets end. Seldom source sensor updates could be an issue. I suggest to observe this next. If this is in fact an issue, we need to introduce a service to proactively update the statistics sensor. Thinking of an automation similar to "Every 30min: call service 'statistics.force_update' on 'sensor.ws_windgeschwindigkeit_durchschnitt'".
• Irrelevant here but just so you are aware:
  • I intend to remove the need to define both max_age and sampling_size.
  • I intend to add trace logging for values and computation results, to help with debugging, like @Flox-Muc "Can you tell me how to extract the samples of the database?"
  • I intend to introduce attribute_characteristics for secondary characteristics.

Please let me know what you think.

[sophof]

I just ran into this issue somewhat while trying to create a short window statistics sensor and it becoming unknown if the value is stable. What you could do is follow the logic I added for the derivative sensor. In effect I only update the sensor values if an update is received and then calculate the values for the window based on the assumption that it increased linearly. It solved the exact same glitch that started this thread.

The disadvantage of this approach is that the derivative trends to 0 but not always becomes 0, the advantage of the approach is that the sensor doesn't have to constantly check its state. I'm thinking of adding some logic to recalculate the state if it hasn't received an update for some time. Either based on the time window itself or via an optional parameter. With that I think we can get the best of both worlds (accuracy and speed) and add some customisation for the user if required.

Since this sensor can also calculate more 'continuous' values such as standard deviation, I think the extra step of an optional polling parameter is required.

For more info see: #31579
And: #65528

Most important implementation detail is that you have to keep the last value outside the window for your calculations.

[flo-wer]

After spending too much time digging though many issues and the not so well conceived implementations of both the statistics and derivative integrations, I conclude that there should first be a consensus on which features (i.e. different kinds of derivative functions in the mathematical sense) should be implemented, because there are multiple ways of using a derivative function and this depends on the desired result.

First of, the smoothing implemented in #65528 by @sophof in the derivative integration using a simple moving average should not be part of a derivative function and is actually already implemented in home assistant using the filter integration. Only in some cases the smoothing is desired, but it is not even clear if the smoothing takes place before or after the derivative function. Both cases are valid but produce different results.

Second, usage of the derivative function can be summarized to the following cases:
(1) A "real" derivative of discrete values using only the last (newest) two values divided by the time between them. No matter the length of the time between them and the time between the last value and now. There are always two values (as long as the source sensor has enough values).
(2) A "constantly" moving window of time, using the newest and the oldest value still inside the time window. The end time of the window is always now. There could be only one or zero values, which must be handled explicitly (i.e. value 0).
(3) A "lagging" moving window of time, using the newest and the oldest value still inside the time window. The end time of the window is the timestamp of the newest value. There could be only one value, which must be handled explicitly (i.e. value 0).
(4) More "obscure" cases which mix a moving time window with a limited count of samples or more then two samples with no time limit. These cases are obscure because they try to smooth the resulting values by choosing larger/smaller sample sizes. These cases are better handled in combination with the filter integration.

I did a research on the history of commits changing behavior: As the author of the original derivative function (see commit 8a2bc99) in the statistics integration, I adapted to the sampling_size and max_age config to support case (1) and (3). At that time, the value only updated when the source sensor changed. Then later commit 4149bd6 changed the behavior to (1) and (2), because of a new timer that gets fired at the time, the oldest value would expire out of the time window. Next, commit b8f4b76 (change from attributes to separate sensors) broke the "correct" implementation because unknown is returned if there are less than two values. The latest change was commit ad8b882, which improves case (2) as there is no longer a limit to the sample size (for large windows).

For the competing derivative integration, case (1) is supported if no time window is configured and case (3) is supported with some arbitrary smoothing applied.

The conclusion is that the statistics integration has the bug described in this report (and #19800 #61624 #71997 #80648 #82063) und should return value 0 instead of unknown if there are not enough values in the time window (not for other cases like missing sensor etc.). The derivative integration has the same bug (#31395 #31579 #45822 #48717 #52538 #53425 #56488 #83496) but because it never reaches value 0 with its algorithm.

At the last I am curious why there are two different integrations for the same purpose, why does the derivative integration even exist when the statistics integration does the same and even more. I would vote for statistics to get better UI integration and derivative to be removed entirely. (@balloob)

[Zocker1999NET]

I'm in with @flo-wer's proposals however I want to point out that statistics has features where simply returning 0 when no values are in is wrong. For example when using it to calculate averages about the last time frames.

Maybe it fits for most cases (where at least one value is required) to do the following if no values were recorded: Use the current value of the sensor two times (one for the beginning & the other for the end of the time window). Then calculating the derivative should "automatically" return 0 (which is the derivative for a constant) while calculating the average will "automatically" return the current value (which is the average for a constant).
Still it doesn't fit for count & sum but those implementations should naturally be able to return 0 when no values were recorded.

[sophof]

I must admit I was aware of the same (in my own way) as @flo-wer describes when I wrote my pull request, but I chose to be practical about it. For me this was simply the quickest way to achieve what I wanted and the old implementation was definitely incorrect in my opinion. With the limited time I have this was the best option for me. But yes, in essence I put a band-aid on something that probably shouldn't have existed in the first place.

I'm not 100% sure, but I expect the same results could indeed be achieved by having the derivative sensor only calculate a naive derivative between two values and then handling the rest via the filter sensor. I'm personally not a big fan of intermediate sensors that you don't use, but that really is a different issue (and one can hide them I guess). This way it would be less redundant and in general you don't want each entity handling it's own 'filtering' probably.

So I assume for statistics the same result could be achieved with the change per second characteristic? In that case for me at least it was just a case of not finding it (the last time I looked at the statistics sensor there was no derivative alternative afaik).

I'm rambling a little, but I guess I'm saying I agree with this:

I would vote for statistics to get better UI integration and derivative to be removed entirely. (@balloob)

I do strongly think a good UI integration and an overall rethink of how to approach the subject is required in that case though. It seems we need to think about the scope of the integration in this case (where does this end?).
I also think a similar argument applies to the filter integration.

[mekaneck]

I was pointed to this issue after struggling with both a statistics: mean sensor and a derivative sensor. I've tested how they operate, and after reading through this issue as well as #83496 I can say that the very first thing that needs to be done is to decide what can be assumed about the source sensor. I've put together these plots for clarity because depending on how the source sensor reports data, the 'correct' average or derivative calculation will change.

Here's some example data and 4 different assumptions about what the data could represent:

Assumption 1: Sensor sends new data at state change

Assumption 2: Sensor sends stale data at state change

Assumption 3: Sensor sends updates at unknown time-based intervals

Assumption 4: Sensor sends updates at defined time-based intervals (and possibly also fresh data on state changes)

Note that in the plots above I intentionally stopped the line at the point closest to 'now' (zero on the x-axis) where it is no longer possible to determine what the line will look like.

In addition to picking which assumption is made about the data, there is also the issue about the oldest data point used in the max_age window.

• The derivative sensor will use the closest data points on either side of max_age and perform linear interpolation to determine what value should be used at the edge of the window. This is fine if your data is represented by assumption 3.
• The statistics sensor will ignore any data outside the window and will pick the oldest data point inside the window. This doesn't seem appropriate for any scenario.

IMHO assumption 4 is probably the most-often-encountered representation of data where people are noticing issues with these sensors. (Meaning the data actually represents assumption 4 but the statistics/derivative sensors are assuming assumption 1 or 3).

What I would love to see would be some configuration options in the statistics sensors (and derivative sensor) where we can choose what assumptions it should make about the source data.

[sophof]

Maybe I misunderstand your post, but there's only one valid assumption for both integrations. AFAIK right now they both only update when a new sensor value is posted, so on calculation the last value is always at 'now'.

Put in another way: when there's no state update both assume a constant derivative, not a constant sensor value, which isn't any of your examples I think. Or is that your assumption 4?
My point is, it really doesn't matter what you assume about the update interval of the sensor, it matters what you assume if there's no update at all. My personal sensors all update regularly or if there's a significant change. With that setup there's no issue ever really.

To be clear, that is almost guaranteed to be wrong (since with time the derivative should then trend to 0), it just only really shows if there's a long gap in the sensor updates and the last derivative value was far from zero.

[mekaneck]

Maybe I misunderstand your post, but there's only one valid assumption for both integrations. AFAIK right now they both only update when a new sensor value is posted, so on calculation the last value is always at 'now'.

This is almost true; the statistics sensor will update upon a new sensor value as you mentioned but will also update when the oldest sensor value ages outside of the max_age setting. The derivative sensor appears to only update when a new value is posted.

Put in another way: when there's no state update both assume a constant derivative, not a constant sensor value, which isn't any of your examples I think. Or is that your assumption 4? My point is, it really doesn't matter what you assume about the update interval of the sensor, it matters what you assume if there's no update at all. My personal sensors all update regularly or if there's a significant change. With that setup there's no issue ever really.

There's still an issue, and that is when a value is repeated. When a sensor has an update but it's the same value as the previous value, it isn't stored in the recorder database is this is where weird things happen. Lots of sensors have enough noise and small enough resolution where they don't see this. But if the sensor has a hard min or max (zero being the common one), or if it has low resolution and/or low noise, this issue pops up. The sensor I'm having an issue with is a water meter reader, and when I don't use water overnight, even though the sensor reports the reading every 2 minutes, the recorder database has a single datapoint for the reading when water was last used. Then the database can have 6+ hours before another data point appears when we use water in the morning. The statistics and derivative sensors don't assume a constant derivative when there aren't updates; they simply don't update at all (except as mentioned before when data points fall outside of max_age). If you plot them in the history tab, it will appear to "hold the last value" but that's just how the history tool plots data. There's really only one data point, and it's at the same point in time of the last data point of the source sensor. No data points exist afterwards (i.e. no data points at 'now').

To be clear, that is almost guaranteed to be wrong (since with time the derivative should then trend to 0), it just only really shows if there's a long gap in the sensor updates and the last derivative value was far from zero.

The derivative does not trend to zero as one would expect, because the derivative never recalculates after the last data point (and there is no data point at 'now'). If a change is implemented to force the sensor to update on command or at some interval, you'd see some odd behavior at times. A constantly increasing sensor updated every 2 minutes would have a sawtooth derivative if the derivative sensor were updated every 1 minute. (That is assuming the derivative sensor still interpolates data on the left/old side of the age window and holds the last value on the right/new side of the window.)

[mekaneck]

The main point of my first comment was that whatever changes are made to the statistics sensor, the new behavior should be logically consistent. Specifically, whatever assumptions are made about the source sensor data on the right side of a moving time window (i.e. 'now'), the same assumptions should be made on the left side of the window (in the past).

So for example, if it is going to be allowed to have a statistics sensor update (recalculate) 10 minutes after the last sensor data point (by assuming the stale source sensor data is still valid), then it should also be assumed that the sensor data stayed flat for 10 minutes as it moves to the left in the time window. So if there was an update to the sensor 11 minutes after the data point in question, then the statistics sensor should always perform calculations assuming the data point was flat for 10 minutes and then jumped to a new value at 11 minutes. It should never interpolate between the first data point and the 11-minute data point, forgetting that at one point in time it assumed there was another data point.

The only possible logically-consistent assumptions I can come up with are the 4 examples that I listed in my post above. Anything else just makes an assumption about the data now for convenience, and then makes a different assumption about the data when it becomes older. I can't come up with a scenario where that is a good idea.

[juicejuice]

I love the direction all this is headed, it sounds very exciting to have sophisticated statistics capabilities.
However, I thought I would post a workaround for those, like me, who just need to get a statistics sensor working correctly. Perhaps the HA Community is better for this, but Google seems to lead people here. Adding a time-based attribute to the source sensor forces the HA database to store the entity state at each time interval. This avoids the original issue highlighted by @Flox-Muc.
Sample code:

  - sensor:
      - name: "Desired Charge Current"
        unit_of_measurement: "A"
        state_class: "measurement"
        state: >
            {{ charge_current }}
        attributes:
            minute_counter: "{{ now().minute }}"

Source: https://community.home-assistant.io/t/add-force-update-support-to-template-sensor/106901/31

[brenank]

#88655 is another solution that would avoid the statistics sensor from going into an unknown state given repeated sensor values or infrequent source sensor updates.

It does not change the current timing of statistic sensor updates which is discussed above.

[ThisWayToo]

I just want to chime in here since I see that a lot of discussion is about the averaging case for statistics. I want to be sure that whatever fix that is implemented for this issue also works for the 'state_characteristic: total' use case as well.

My use case is using the statistics sensor to generate rain amount totals for N periods of time looking back from the present time, e.g. 1 hour, 6 hours, etc. As such utility meter is not suitable for my needs since its time boundaries are chronology based. What I have setup is working well with the oldest data aging out as I would expect, it is just this zero handling issue that is making things not work quite right.

But like the averaging case, I too get Unknown states when there are long periods of no data updates to the statistics sensor before hitting the max_age timeout. The statistics sensor is getting 0.00 values once per minute the entire time. So it is being fed information from a ESP Home device continuously, but the values are zero for hours, days, and maybe even a week or more stretches. So the basic problem that I'm seeing is well captured by this issue.

[jasonmp85]

Chiming in because once again I've picked up a new part of Home Assistant and once again, it's sent me on an hours-long goosechase. It didn't take me long to figure out "oh, the temperature isn't changing much outdoors, is that why this 1m average sensor is 'unavailable'?", but even after making the jump to "I probably need force_update?", finding any documentation of this bizarre behavior has been tough.

I figured I could ditch the mess of JSONata and Node-RED flows I used to post weather data to PWS pre-Home Assistant. After all, I have all the sensors in here over MQTT, why can't I just slap an average on them and have it post every minute?

So… what exactly are we doing here? I'm hearing "making fancy new additions to statistics", but… what can I do to work around this now, and what can I do to fix the codebase now?

[jasonmp85]

By the way, this is making mean sensors fail for a weather sensor which is sending data every fifteen seconds. It's just that the temperature often won't change by more than 0.1° every minute, and the humidity is measured in integer percent… it's been pouring rain all day and it's pegged at 91%, so Home Assistant simply doesn't see any data going back, oh, probably an hour.

Why tie a moving window of values to a persistent storage backend? Shouldn't we be able to disentangle "I want a moving average of incoming data" from "I want to only store changes"?

[juicejuice]

Why tie a moving window of values to a persistent storage backend? Shouldn't we be able to disentangle "I want a moving average of incoming data" from "I want to only store changes"?

If you want to skip a whole bunch of learning just use the workaround I posted above. I've had it in place for two months now and it works great.

[ThisWayToo]

There's still an issue, and that is when a value is repeated. When a sensor has an update but it's the same value as the previous value, it isn't stored in the recorder database is this is where weird things happen. Lots of sensors have enough noise and small enough resolution where they don't see this. But if the sensor has a hard min or max (zero being the common one), or if it has low resolution and/or low noise, this issue pops up. The sensor I'm having an issue with is a water meter reader, and when I don't use water overnight, even though the sensor reports the reading every 2 minutes, the recorder database has a single datapoint for the reading when water was last used. Then the database can have 6+ hours before another data point appears when we use water in the morning. The statistics and derivative sensors don't assume a constant derivative when there aren't updates; they simply don't update at all (except as mentioned before when data points fall outside of max_age). If you plot them in the history tab, it will appear to "hold the last value" but that's just how the history tool plots data. There's really only one data point, and it's at the same point in time of the last data point of the source sensor. No data points exist afterwards (i.e. no data points at 'now').

In addition to seeing the repeated zero value manifestation of this bug, I'm now 98% convinced that I'm seeing the above comment issue too when there are back to back non zero values that are identical. In my case it is when tracking rainfall. The issue I'm seeing is described in this community post:

https://community.home-assistant.io/t/statistics-sensor-becomes-intermittent/591213/4

In my implementation I'm am using the 'state_characteristic: total' for the statistics sensor.

[ThomDietrich]

Hey all,
I was blocked by other topics in my life and didn't get around to look into this and other improvements I thought about the last couple of months.

Thanks for summarizing the problem in such detail!! I would certainly like to fix it, but can't promise any ETA. If anyone wants to help with a fix, I'd happily review a PR.

[brenank]

@ThomDietrich The main issue is the missing data from the sensor due to repeated sensor values not being stored in the recorder. I'm guessing this was an optimization to reduce db usage, and anything that is not changed is assumed to be in the same state. This is good enough for most use-cases, but not so great when running statistics on the data.

@juicejuice's workaround is a pretty good one. Essentially you're duplicating the sensor, and forcing an update to the db on a regular interval (every minute), rather than only updating when the source sensor is updated/polled. This may update the sensor much more or less frequently than the actual sensor though, so individuals will need to consider the chosen interval based on their use-case. This may cause issues with algorithms that interpolate between the data points though, as this sensor's data can be much more fine grained than the source sensor.

I believe #88655 may also provide an easy way for others to work around the issue for most people using the max_age setting, as it also essentially implements the shared assumption with recorder: that the sensor is in the same state as the last known value. It does not cover the case where only sampling_size is used (as the old values would never be pushed out of the samples, given the repeated values are ignored). This solution also still suffers from the issue that there might be missing data in the middle of your samples if the value was the same for a while, but you might be able to solve this by using an algorithm that considers time depending on your use-case.

Perhaps there's an argument that the new feature in this PR should be on by default, to avoid some of these issues from being filed.

[mekaneck]

I invite people to vote on a feature request to enable a method to solve this problem. Or, if you don't agree with that feature request, it would be a good forum to suggest changes or to link to a different feature request which has your preferred solution.

[issue-triage-workflows]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[flo-wer]

Still relevant.

[HotNoob]

still broken

[juicejuice]

I haven't tested yet but I think this issue might be fixed in an upcoming release: https://developers.home-assistant.io/blog/2024/03/20/state_reported_timestamp/

[gdt]

I've read the comments somewhat quickly, but want to throw ideas for semantics.

First, we need to define the meaning when there aren't data points. For that we need to define the meaning of a sensor at times other than when it is reported. For a sensor that is once a minute, it is understood that these are samples of a perhaps-more-complicated process and we don't know about the state in between. For a sensor that is updated when the value changes, we still are fuzzy on exactly when the real-world state changed, and the shape between updates when those updates are at the max rate it uses. But when there is no update for hours, it is pretty safe that the value is unchanged. As an example, consider an energy meter on a well pump. And consider "well energy last hour". (Really, energy is converted to gallons via calibration, so it's "well gallons last hour".)

This leads to saying "if there is no data point in the hour, that means the value over that hour was equal to the previous report", and that lets us create synthetic points for beginning and end of hour, and then use those for subtraction, min, or whatever. And, if there is a single data point in the hour, then that's the value at the end, and we need to look at the most recent point before that (before the hour) to get the start value.

I think this gets users what they expect, treating "most recent report at time t" as the value at t.

The second thing is that the sensor's semantics are defined as being over some time interval. It would be far cleaner if the sensor always had that value, rather than only updating when the underlying sensor updated. Well gallons last hour in particular should go to zero 1 hour after a report of a new energy value, if no new reports are received. I think it should not be that hard to determine the time at which the values used in the computation will age out (or might age out) and schedule a wakeup/recompute for that time. Of course it needs recomputing on new values too. But this way, the sensor can be defined to be what it says, without any hard-to-understand and not-what-you-want semantics.

Thanks for listening!

[erkr]

My two cents, the last value counts, for sensors that don't change/update.
Now the average doesn't update until a new sensor update.

I tried to configure a daily average for a methane sensor. That methane sensor spikes a few times over the week and is zero in between.
The calculated average can remain positive for days until the methane sensor updates. Really? The average of zero is zero!

[unfug-at-github]

I have created a pull request to fix this issue #124644

[unfug-at-github]

My pull request will solve most of the issues discussed here, let's hope they will integrate it soon.

However, some level of weirdness will still remain. The linear average will be "jumpy" . If you look at a value that is constant for a long time and then suddenly changing to a different value, the linear average will show a sudden drastic change,

Assuming you have an equally spaced measurement like this:

0 (0) (0) (0) (0) 10 (10) (10)

The linear average would compute this:

0 0 0 0 0 5 5.83 6.42

The average step would compute this:

0 0 0 0 0 0 1.66 2.85

The linear average will "jump" to 5 because all (0) values will be ignored, and hence, the linear model assumes a linear increase from the first 0 to the first 10. The average step function will behave more like what you would expect from a moving average and will change slowly after a sudden change of the input values.

In my opinion the linear average is pretty useless since it doesn't give you the smoothed curve you would expect from an average function. It will only behave nicely when you have values that are noisy, so that the values will always change, and hence no value will be ignored.

In the case of noisy values both average functions compute very similar values though, and I don't see the need to have both options then.

Another thing to consider is the loss of precision you get for longer intervals. As far as I understood, all states older than 15 minutes are replaced by a 15 minutes average that home assistant is computing elsewhere. This average computation is most likely not taking into account the time between changes, so the precision of the statistics will most likely not be super exact anyway.

Here is a screenshot of the differences between the two functions:

The values are generated by a test template:

- trigger:
    - platform: time_pattern
      seconds: "/2"
  sensor:
    - name: "xxx_test_float"
      unit_of_measurement: 'x'
      state_class: measurement
      availability: "true"
      state: >   
        {% set now = now() %}
        {% set secs = now.second + (60 * now.minute) %}
        {% if (secs % 2) == 1 %}
          {% set secs = secs-1 %}
        {% endif %}
        {% set modsecs = secs % 90 | int %}
        {% if modsecs <= 10 %}
          {{ modsecs }}
        {% elif modsecs <= 30 %}
          10
        {% elif modsecs <= 40 %}
          {{ 40 - modsecs }}
        {% elif modsecs <= 60 %}
          0
        {% elif modsecs <= 70 %}
          10
        {% else %}
          0
        {% endif %}

and the averages are computed like this:

- platform: statistics
  name: "xxx_test_avg_step_10s_refresh"
  entity_id: sensor.xxx_test_float
  state_characteristic: average_step
  max_age:
    seconds: 10
  refresh_interval:
    seconds: 1

- platform: statistics
  name: "xxx_test_avg_linear_10s_refresh"
  entity_id: sensor.xxx_test_float
  state_characteristic: average_linear
  max_age:
    seconds: 10
  refresh_interval:
    seconds: 1

[unfug-at-github]

Since getting this into the core system seems to take quite some time, I have published a
custom component
to test the changes I made to fix this issue.

It would be great to receive feedback regarding the changes.

[issue-triage-workflows]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.