Test of Exponential Moving Average calculation for temperature slope calculation

[jmcollin78]

content

1. Add ema calculation class (ema.py),
2. calculate an ema temperature named ema_temp in VTherm attributes,
3. use the ema_temp in auto_window detection

DO NOT USE if YOU ARE NOT INVITED TO TEST IT

---

This discussion was created from the release Test of Estimated Moving Average calculation for temperature slope calculation.

[maia]

I only have my phone with me at the moment, but is there a way to expose the ema as entity (at least in the alpha) to be able to look at charts in comparison with the temperature? Thanks!

[maia]

I can't find a way to display an attribute in a chart. How do you do it? I want to have a chart of measured temperature and ema. Thanks!

[maia]

Oh, I don't think the algorithm should be exposed in that way, so I would rather call the attribute/entity/… smoothed_temperature or smoothed_temp.

And ideally all attributes would either use _temp or _temperature in their name, currently it's completely mixed when looking at developer tools -> state. There's no obvious reason why is it called target_temperature_step and ext_current_temperature but ema_temp and comfort_temp. :)

[maia]

Oh, it should be called ExponentialMovingAverage, not EstimatedMobileAverage, it's neither estimated nor mobile. :)

[jmcollin78]

I will change that... Sorry

[jmcollin78]

done

[maia]

I'm seeing some rooms with "unknown" slope. I cannot think of any reason why a slope could be unknown, at least if the HA server has been running for a while and the temperature sensor is reporting? Because even if there's no "new" measurement available, the slope is zero, as it means that the temperature hasn't changed and therefor the sensor didn't report anything new. Might this be a bug that I should try to narrow down?

[maia]

@jmcollin78 Jean-Marc, has this above been added to alpha4?

[maia]

And if not: I will try to export my measured values and recreate your EMA code in ruby and check the results minute by minute. I will need to learn Python soon so I don't need that Ruby translation first, sigh.

[maia]

After a good nights sleep I now think that the algorithm should be able to handle such irregular time series. It's just a question of the half life used. Because when we sometimes have NO measurement at all for 20 minutes (because the temperature stays exactly the same), the algorithm will return a very "old" value that might be influenced by much, much older measurements.

[jmcollin78]

So at 20:25 the temperature was still 21.5°C, and in the next 5 minutes the temperature dropped by 1°C

This assumption is not correct from my point of view. You are creating point that doesn't exists. If temperature drops from 1° after 30 min without message it is more likely that the sensor was off. Else we should have the intermediate value.
And the algo is taking the irregular events natively by the calculation of the alpha value.

If I exaggerate the reasoning, why not considering we have 21.5° value at 20:29:59 ? And then the value at 20:30 will have a too small factor and will not be taken into account.

After a good nights sleep I now think that the algorithm should be able to handle such irregular time series. It's just a question of the half life used.

Yes. This is my opinion too. In Big data (more in my skills), a good practice is to never create missing point else you influence the decision.

So now we have to find a good MAX_ALPHA which smooth the pics (but don't discard it) and avoid too much latency.

[jmcollin78]

Done

[maia]

The slope does not look correct to me, partly because it seems to be rounded to the second decimal (is that possible?), but mainly I assume because you're currently ignoring quick temperature changes. But look at 22:43: the sensor reports a temperature increase of 0,1°C, but only for a very short moment, reverting to the "old" value quickly. The EMA looks as if you're using the first value (the increase) but ignoring the second value that comes in a few seconds later (the decrease), falsely leading to a long term increase of the slope to 0,01.

Also I think that whenever the sensor does not report a new measurement over the past 60 seconds, we should simply assume that the current temperature is the same as 60 seconds ago and update the EMA with this value. Because if the temperature would have changed, we would have a new measurement. And if "every minute" isn't possible, then at least every cycle.

EDIT: It seems the ema_temp is rounded to .1 decimal? Or is it just my bad luck that I'm not seeing values like 20.363943°C for ema? I would consider rounding the ema to be a bug.

[maia]

Here's the chart showing the measured temperature, see the lower part (the part above is the slope), I've already posted it above. Your assumption is correct. My sensor quickly switches between two 0.1°C back and forward if the temperature is somewhere inbetween:

[jmcollin78]

The new code will take this into account and never discard any temperature measurement. Only if it is in the past.

Remember me to never buy you temperature sensor.

[maia]

I had 4 of these (Govee H5075) before I installed Home Assistant. They have a display, run for a year or two on 2 AA batteries and cost ~20 EUR. And are BLE, so HA picked up their data instantly.

[jmcollin78]

I have SONOFF Zigbee SNZB02 with 2 years battery (small round battery) also, less that 20€ also, and also display (and no strange artifacts). But we will deal with yours !

[jmcollin78]

Done

[jmcollin78]

With stepline:

[maia]

Your temperature sensor is very bad.... I have never seen that before.

Do you really think so? I don't find that so bad, here is a 12 hour range. It shows that it tends to oscillate between two 0.1°C values when the real temperature is inbetween. This should be a perfect condition for EMA.

[jmcollin78]

Compare to my temperature graph which is in fact already smooth, definitively yes.

This should be a perfect condition for EMA.

Exactly. I will publish a new alpha.

[maia]

Your charts are so smooth because your temperature changes by 2°C within an hour. :) Do you have a room with only 0.1°C changes over hours? Would love to see a chart for that (and if it's smooth, please tell me which sensors you're using. I have a dozen Aqara Zigbee sensors too, just in some rooms I have these Govee sensors)

[jmcollin78]

Done

[jmcollin78]

No even on room with small variations, it is very smooth:

[jmcollin78]

The EMA temp is quite equal to the current_Temp everytime. That is expected.

[jmcollin78]

Done

[jmcollin78]

https://github.com/jmcollin78/versatile_thermostat/releases/tag/4.2.0.alpha4

WIth all your feedbacks. See you tommorrow (I will be off tomorrow).

[maia]

As for the question if it is necessary to update the ema value every cycle even when there is no new temperature measurement:

1. To calculate the slope we need to know what the current (ema-smoothed) temperature is and also what it was over the past X (for example 10) minutes.
2. If the sensor reported a new measurement recently, we know what this current (ema-smoothed) temperature is.
3. But what if the sensor didn't report any new measurement over the past 30 minutes, because the temperature didn't change at all? The easy solution is to set the slope to 0 when there is not a single measurement in the past X minutes.
4. But what if the last two measurements are: 21.0°C 60 minutes ago and 20.5°C 9 minutes ago? The "current" ema-smoothed value would be the calculation from 60 minutes ago and 9 minutes ago (as this was when it was calculated the last time), and would completely ignore that in the past 9 minutes the temperature stayed constant. And that would be wrong.

This is why I believe that it is necessary to use the latest measurement as input for the EMA in each cycle.

EDIT: Take a look at this chart (I've increased the rounding precision in the code):

Here are the measured values:

10:35:12  20.7
10:35:42  20.6
10:36:11  20.7
10:57:11  20.6
10:57:33  20.7
10:59:23  20.6
11:00:02  20.7
11:01:15  20.6
11:02:28  20.7
11:02:57  20.6
11:03:50  20.7
11:05:25  20.6
11:24:17  20.7
11:24:34  20.6
11:31:23  20.7
11:31:39  20.6

The increase to 20.7°C at 11:24:17 causes the EMA to increase sharply, because the last measurement is 19 minutes ago and the alpha used for calculation is huge. But 17 seconds later the temperature drops again to 20.6°C and this time the EMA is hardly reduced, as the alpha is tiny at 17 seconds. I'm not sure how to correctly solve the issue, but currently the algorithm cannot handle it well if sometimes the time difference is 19 minutes and sometimes 17 seconds. The more measurements we can use for the calculation (even if they are the same), the smaller this problem will be.

EDIT2:
The algo clearly has a problem with these short jumps up and down:

[maia]

I think I found the solution: Logic tells us that a minute ago the temperature was most likely so close to the last reported temperature, that the sensor didn't publish an updated measurement. Therefor any changed temperature must have happened in the past minute. This can be implemented by setting MAX_ALPHA to 0.13 (which is the calculated alpha for a 1 minute time distance at a 5 minute cycle).

I'm now testing how this much lower alpha works and if it causes other issues. I won't be able to report much before Monday afternoon.

[maia]

MAX_ALPHA = 0.13 works well in this situation with an open window. The open window detection works well too. I'd put the trigger at -0.05 and not -0.1, but we'd need more data points for the suggested value in the documentation.

Maybe with these settings the half life could be reduced from 5 minutes to 2.5 minutes? Will need to test.

[maia]

And here a situation with a pretty constant temperature. Using MAX_ALPHA = 0.13 works fine as well. Obviously there is a lag, but adjusting the half life to 2 minutes might help.

[jmcollin78]

Hello @maia ,

I see you have done a lot of work and thinking. That's fine. Here is some remarks :

To calculate the slope we need to know what the current (ema-smoothed) temperature is and also what it was over the past X (for example 10) minutes.

Currently the slope is not calculated considering the past. I just use two last points : the last known and the current event. And globally I make d emaTemp / dt. Because the ema takes care of the past, I think this is sufficient to have something well,

But what if the sensor didn't report any new measurement over the past 30 minutes, because the temperature didn't change at all? The easy solution is to set the slope to 0 when there is not a single measurement in the past X minutes.

I nothing special to do. If dt is 30 the slope will be 0 or near 0 automatically with the formula above. Between the two measure, I agree that we should recalculate the slope at each cycle as if a measure point have been set by the temp sensor. Then the magie will occurs without having anything special to do.

But what if the last two measurements are: 21.0°C 60 minutes ago and 20.5°C 9 minutes ago? The "current" ema-smoothed value would be the calculation from 60 minutes ago and 9 minutes ago (as this was when it was calculated the last time), and would completely ignore that in the past 9 minutes the temperature stayed constant

Because the alpha is depending of the last timestamp the 21° will be mostly ignored and only 20.5° will be taken into account. That is what we want. That is the beauty of the formula with an alpha depending of the dt. So for me there is nothing to do the ema calculation. It should stay fix until the next measure and be calculated depending of the dt. On the contrary the slope need to be adjusted even if the sensor don't send new temp event, because we can consider that no event means a stable temperature. I will fix that in the algorithm of the slope calculation.

In my opinion it would be wrong to consider a fake measurement have been send just before a real measurement. In that case, dt with the previous will be very low, and the last real measurement will be not taken into account. That will introduce more delay for adjusting with the real temperature.

This is why I believe that it is necessary to use the latest measurement as input for the EMA in each cycle.

Given my explanation above, I think this is true for the slope (and missing as now) but especially not for the ema. You are referring to stock value which are coming at a constant rate, but this is not the case here and the alpha does the trick:

time_decay = (timestamp - self._last_timestamp).total_seconds()
alpha = 1 - math.exp(math.log(0.5) * time_decay / self._halflife)
new_ema = alpha * measurement + (1 - alpha) * self._current_ema

The increase to 20.7°C at 11:24:17 causes the EMA to increase sharply, because the last measurement is 19 minutes ago and the alpha used for calculation is huge. But 17 seconds later the temperature drops again to 20.6°C and this time the EMA is hardly reduced, as the alpha is tiny at 17 seconds. I'm not sure how to correctly solve the issue

"Increase sharply", we are talking about a gap from 20.62 to 20.68 (+0.06°). This is real not sharply inmy opinion. Remember that temperature has a resolution of 0.1°, so the gap is half under the resolution. Why do you want to cancel this +0.06° "pick" ? You will just introduce more latency in the system and too much reduce the dynamic. It is visible in the curve you gave:

12 min latency here:

I agree we can reduce the MAX_ALPHA (which was 0.9375), but 0.13 seems very, very too low (and so too smooth) for me. Maybe approx 0.5 is a good compromise. This makes an already strong filter (0.5 is the value I had before in the slope calculation).

Obviously there is a lag, but adjusting the half life to 2 minutes might help.

Reducing the halflife is just a way to augment the alpha that you wish to drastically reduce. Is that not contradictory ?

[jmcollin78]

done

[maia]

Currently the slope is not calculated considering the past. I just use two last points : the last known and the current event. And globally I make d emaTemp / dt. Because the ema takes care of the past, I think this is sufficient to have something well,

Agree, but I'm trying to look forward: For #208 we might need a 60 minute slope and/or a 10 minute slope, which will require more than these two points you mention.

In my opinion it would be wrong to consider a fake measurement have been send just before a real measurement.

There is no "fake" measurement. :) If a sensor isn't reporting a new temperature, it can only have 2 reasons: a) the temperature didn't change and b) the sensor is offline. But with a low alpha we don't need to add a "fake" measurement anyway.

"Increase sharply", we are talking about a gap from 20.62 to 20.68 (+0.06°). This is real not sharply in my opinion.

Not on an absolute basis. But the issue is that short spikes in temperature cause the EMA to be off for longer.

You will just introduce more latency in the system and too much reduce the dynamic. It is visible in the curve you gave

For window opening we just look at the slope and the slope turns to -0.05 in less than a minute after I opened the window. And also for future additional uses of the EMA I think we will mainly use the slope. Does the slope calculation already use the EMA values?

I agree we can reduce the MAX_ALPHA (which was 0.9375), but 0.13 seems very, very too low (and so too smooth) for me. Maybe approx 0.5 is a good compromise.

I don't like constants that are not rationally explainable. A MAX_ALPHA of 0.5 means that we assume that measurements that are older than 5 minutes were still correct 5 minutes ago – but that they were not correct less than 5 minutes ago:

alpha = 1 - math.exp(math.log(0.5) * time_decay / self._halflife)

1 - math.exp(math.log(0.5) * 1.0 / 5.0) = 0.13
1 - math.exp(math.log(0.5) * 2.0 / 5.0) = 0.24
1 - math.exp(math.log(0.5) * 3.0 / 5.0) = 0.34
1 - math.exp(math.log(0.5) * 4.0 / 5.0) = 0.43
1 - math.exp(math.log(0.5) * 5.0 / 5.0) = 0.50

Why would you think that a sensor does not report a changed temperature over 5 minutes?

Reducing the halflife is just a way to augment the alpha that you wish to drastically reduce.

The MAX_ALPHA only says how far ago the last measurement may be. In the screenshotted example with the open above the MAX_ALPHA would be irrelevant as the sensor reported a changed value multiple times per minute. So reducing the half life should reduce lag in such situations.

My new suggestion (which I'd still need to test) is:

• half life = 2.5 minutes
• max time duration of last measurement = 1.0 minute
• This results in: MAX_ALPHA = 0.24

[maia]

Here is 2.5min half life and MAX_ALPHA 0.24:

[maia]

Here is 2.5min half life and MAX_ALPHA 0.24 in a room that heated from 21 to 22°C:

[jmcollin78]

Maybe I'm wrong but there too much latency. Ok the curve is very smooth but the ema doesn't reflect the real value. This is a problem. There is 15 min latency in you example maybe more. We don't want the temperature 15 min ago but the temperature now smoothed. For me this is not the correct tuning. I wonder if we are aligned on the same goal.

1 - math.exp(math.log(0.5) * 5.0 / 5.0) = 0.50
Fine. A temperature 5 minutes ago is considered at half of its value and half to the real value. I think that is fine. And the more the point is near in time, the less its weight is.

half life = 2.5 minutes
max time duration of last measurement = 1.0 minute

Temperature don't change at this rate for me. The resolution (and I have a speed heating AC) is more than 5 minutes in horizontal resolution.
Look at the result:

I have never more than 5 minutes latency, 0 latency when temperature is stable, the pics are smoothed.
This parameters are:

• halflife 5 min (the size of the horizontal steps when temperature moves),
• MAX_ALPHA : 0.9375 (I think this can be reduce to maybe .5 to avoid your rapid pics due to sensor imprecision).

I will try that configuration. I hope you will give it a try with your configuration without modifications to the coefficient. I'm curious of what is wrong.

[jmcollin78]

https://github.com/jmcollin78/versatile_thermostat/releases/tag/4.2.0.alpha5

Please don't add comments in multiple page or code review. It is not easy to follow. Start here please.

[jmcollin78]

Here is the new curve I got with the previous settings, on a full heating phase:

We can clearly see the regulation and the ema following while smoothing the current temperature with a latency of 4 or 5 minutes.

[jmcollin78]

Here is a zoom of the slope vs ema:

This is not what I expect. There is some step at 0 which are not wanted. I guess this is because I add artificially points at each cycle which have a same value tha the preceeding just before. This lead to a 0 in the slope curve.

I should not add artificially point if real points added at each temperature change but only if no temperature change are received. I will have to think about that a little more.

EDIT : the MIN_DELTA_T parameter of the slope algorithm was wrong. Don't take this curve into consideration. I will have to redo one.

[maia]

Smoothing is always a tradeoff between decreasing timeliness and increasing interpretability. The question is usually how many measurements (or which time frame) one needs to allow a valid interpretation.

I believe many issues could be resolved if the default cycle for all these calculations wasn't 5 minutes but 1 minute. This would give us more data points on a curve to calculate e.g. a slope. Oh, and I still think it would be good to have a short term slope (5-10 minutes) and a long term slope (60 minutes). This could help with the adaptive regulation.

[maia]

Here are three rooms where I opened the windows this morning, using the latest alpha (with nothing changed). The slope calculation is off, I agree, we'd need to update the smoothed value every x minutes and use that for the slope.

If the smoothed temperature is calculated every minute, it should be possible to take the difference of the current smoothed temp to the smoothed temp 5 minutes ago for the slope (or 60 minutes for the "slow" slope).

Better are probably other algorithms than a simple subtraction, such as regressions, as these take into account everything that happens inbetween. It should be easy to store the values of the past 60 minutes in memory, no? (A list with key/value pairs for timestamp and temperature, each cycle the new data is added and all values that are too old are removed from the list)

EDIT: If the current temperature (the last measurement) is added to the ema calculation each cycle, there should nearly never be a situation where the previous ema equals the current ema. It would also reduce lag.

[jmcollin78]

With the alpha6 release I have got this as slope:

This look like promising.

[maia]

I've installed alpha6 2 hours ago. It's interesting how different our heating systems are: You can increase the temperature by 4°C within 1 hour. In many of my rooms I need 4 hours for 1°C, in some rooms it takes me 12-24 hours for a 1°C increase.

It's challenging to write algorithms that suit systems with all these different speeds.

[maia]

I had the window open for 20 minutes, the window detection would not have worked at a -0.1°C slope threshold, see below. I would have needed a -0.02 threshold in that case to have few open/close false signals in that time frame, but then I would have had false positives during the day (see the spike at 20:40).

Is it technically possible to run the ema calculation every minute, with a half life of 2.5 minutes? I imagine that could help, what do you think? Is it worth a try?

[jmcollin78]

how different our heating systems are

It is an example with a reversible air/air heatpump. And more over it is the more economical system I never used. Very speed to heat (with a fan integrated), very low consumption. But: regulation is bad, make some noise when its runs, need an external module.

And yes it is very hard to find suitable parameters for every situation like with the self-regulation.

I wonder if I will propose to adapt the parameters to situation. For the expert mode of the self-regulation, I implement loading parameters from configuration.yaml file. I should be easy to use this mecanism to be able to set max_alpha, halflife and various parameters like that.

Is it technically possible to run the ema calculation every minute, with a half life of 2.5 minutes? I imagine that could help, what do you think? Is it worth a try?

All is technically possible if you have time and money. It is no so easy, because I will need another event loop than the main cycle loop and I still continue to believe it is not necessary (but i'm so sure now when I see your curves). It will add another layer of complexity above something which already complex. So technically possible but no so simple and certainly not desirable.

Do you think you will be able to give me all points on the very bad last curve you have ? Only the EMA or brut temperature is needed. I will try do to some simulation on my side with a fixed set of data.

What is strange, is that the EMA seems not so bad. I really don't understand what happens à 20:40. Do you have an idea ?

Huum, when I look at my last curve I see something very bad around 22h00 :

22h00 is the time at which I change the target temperature. I don't see why changing target does this on the slope. There is still something wrong in the slope calculation. Ema on my curves seems very good.

[jmcollin78]

The temperature has fall from 0.2 ° at 21:50. It happens each time after a slope has been set to zero which is after a stable period. This is my fake point I add at each cycle when temperature event is not received during 10 min.

[maia]

It is an example with a reversible air/air heatpump. And more over it is the more economical system I never used. Very speed to heat (with a fan integrated), very low consumption. But: regulation is bad, make some noise when its runs, need an external module.

As I only rent this place, I cannot change the system, unfortunately.

I wonder if I will propose to adapt the parameters to situation.

Ideally that's not necessary. I believe good algorithms manage to adjust to the surrounding. I've not given up hope yet that we find a solution.

It is no so easy, because I will need another event loop than the main cycle loop and I still continue to believe it is not necessary

I understand, if it needs a second loop. But what if the default main loop would run every minute, not just the temperature related calculations? Does that have any disadvantage for the other parts of the code?

Do you think you will be able to give me all points on the very bad last curve you have ? Only the EMA or brut temperature is needed. I will try do to some simulation on my side with a fixed set of data.

Sure, if that's of any help I can provide you with data from different rooms (which use different temperature sensors too).
231123 kitchen.csv

Oh, and as I've not said it for a couple of days: Thanks for investing so much thought into this issue. :)

EDIT: I've mentioned it a few times already, but I'm slowly more and more convinced: I think users would benefit from ONE setting that defines the speed at which rooms heat or loose temperature. And internally many other parameters are derived from that (thresholds, half lifes, cycle speeds,…). This prevents people from configuring something wrong and also gives you much more possibility to adjust settings from one version to the other. As an example, if you'd ever want to suggest a different open window threshold, all current users would have to reconfigure. But ideally they would not need to care if it's -0.05 or -0.10 or -0.15 because that's too much detail for them.

[maia]

Here's another slope calculation gone wrong in the current alpha, causing a false open window alarm at -0,27°C/min at 22:02:21:

And here is the data for the actual measured temperature, if you want to try to recreate it:

2023-11-23 21:01:51,19.9
2023-11-23 21:02:48,19.8
2023-11-23 21:03:39,19.9
2023-11-23 21:04:02,19.8
2023-11-23 21:05:15,19.9
2023-11-23 21:06:42,19.8
2023-11-23 21:09:13,19.9
2023-11-23 21:09:35,19.8
2023-11-23 21:13:46,19.9
2023-11-23 21:14:08,19.8
2023-11-23 21:26:23,19.9
2023-11-23 21:28:21,19.8
2023-11-23 21:29:51,19.9
2023-11-23 21:30:42,19.8
2023-11-23 21:30:58,19.9
2023-11-23 21:31:04,19.8
2023-11-23 21:33:17,19.9
2023-11-23 21:33:55,19.8
2023-11-23 21:34:18,19.9
2023-11-23 21:43:19,19.8
2023-11-23 21:43:25,19.9
2023-11-23 21:45:17,19.8
2023-11-23 21:45:40,19.9
2023-11-23 21:48:20,19.8
2023-11-23 21:49:34,19.9
2023-11-23 21:49:56,19.8
2023-11-23 21:50:41,19.9
2023-11-23 21:51:03,19.8
2023-11-23 21:51:46,19.9
2023-11-23 21:52:08,19.8
2023-11-23 22:02:21,19.7
2023-11-23 22:03:12,19.8
2023-11-23 22:04:42,19.7
2023-11-23 22:05:33,19.8
2023-11-23 22:07:00,19.7
2023-11-23 22:08:30,19.8
2023-11-23 22:08:52,19.7