Derivative integration gives incorrect values when source does not change

[EDelsman]

The problem

The derivative of a constant value is 0:
https://owlcation.com/stem/The-Derivative-of-a-Constant-With-Examples

However, the derivative integration only updates if the source value changes. Therefore, the derivative integration is never 0, even if the source stays the same for a very long time. This makes it impossible to see when the source is unchanging. Instead the derivative indicates a constant rising or falling value, which isn't true.

In other words: the progressing of time, without changes to the source, should be considered as a possible change of the derivative too. The derivative should change towards zero while time progresses without changes to the source.

What version of Home Assistant Core has the issue?

2022.11.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

derivative

Link to integration documentation on our website

https://www.home-assistant.io/integrations/derivative/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant]

Hey there @afaucogney, mind taking a look at this issue as it has been labeled with an integration (derivative) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of derivative can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Change the title of the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign derivative Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

derivative documentation
derivative source
_{^{(message by IssueLinks)}}

[afaucogney]

Hello, thanks for posting your question.
Why a source could not update the value with the same numerical value ?

Please post a screenshot of the source and it's derivatives signal to better understand your issue. Thanks.

[EDelsman]

Sensors could post more values, but most battery ones usually don't when there's no change. A.f.a.i.k. template sensors also do not store any data if there's no change. I would not know how to force them to.

[EDelsman]

I've been thinking: a regular sensor and/or sensor graph assumes the value is unchanged if no new sensor update is known. Most sensors "expect" this behaviour: they send frequent updates when the value changes quickly, als slow (or no) updates when the value is stable.

For derivative,this assumption is currently the same: no new updates means derivative stays the same. But when the derivative is non-zero, that assumption clases with the above assumption.

For derivatives, if the real value is stable, the derivative changes toward 0. Basically, what it should periodically do is recalculate, assuming the value now() is equal to the last known value of the original sensor - the same as the assumption above. When you do, you'll see the derivative change.

This could be optimized because when the derivative is already 0, the change of time is irrelevant, because the derivative is in line with the assumption of a stable base sensor.

Trend sensors will likely suffer from the same wrong assumption, as they internally depend on a derivative.

[Petro31]

A proper way to handle long pauses within the HA paradigm is to schedule an update in x seconds. This update would only occur if isn't a natural update that occurs within those x seconds. Treat it as a fallback so that it does not impact sensors that have a periodic update frequency. The calculation would remain unchanged as the time window would still be honored, i.e. change/time.

This would most likely require a new optional field for the schema. The default would be none to keep backward compatibility. None would not add a next-scheduled-update. (1 through N) would add a next-scheduled-update in (1 though N) seconds from the last update. If no update was received between the last+update and next-scheduled-update, an update will fire at next-scheduled-update's time. Lastly, every update would reset next-scheduled-update.

Going this route would keep current functionality while adding granularity to the derivative calculations.

[EDelsman]

That sounds like a good way to solve it. But I am curious why you think the old behaviour would be beneficial to anyone. If sensors update regularly, the fix wouldn't change a thing. And if it does not, why would anyone want to keep the old behaviour? Because now the value of the sensor is determined by what happened prior to the long pause.

[Petro31]

That sounds like a good way to solve it. But I am curious why you think the old behaviour would be beneficial to anyone. If sensors update regularly, the fix wouldn't change a thing. And if it does not, why would anyone want to keep the old behaviour? Because now the value of the sensor is determined by what happened prior to the long pause.

Because we all have sensors that constantly update, I don't need this. My temperature and humidity update on a 30 second basis and the resolution is large enough that values are always different. It provides the correct result.

[ildar170975]

Could be related: #84088

[neil1111]

I’m having a similar problem. I setup a derivative to track temperature changes from an Aotec sensor. The temp is changing, but the derivative never does. See the following charts for both:

Here are the settings:

[finity69x2]

Every person who uses a derivative sensor should expect that the derivative sensor should react as a true derivative.

By definition a derivative is a rate of change.

If the source sensor never changes then the derivative by definition is 0.

The derivative sensor shouldn't need to see a change in the source sensor for it to re-calculate. if the source doesn't change then the derivative sensor should be 0.

I created a test setup to see what the true behavior of the sensor is.

Here are the results:

the source sensor stopped updating at 3:38 PM:

the derivative sensor stayed at the same value ever since:

the derivative sensor never goes to 0 as expected when the source stops changing.

Any user who actually understands how a derivative is supposed to work would be confused by the current behavior.

we shouldn't need to create a work around for the incorrect behavior of the derivative sensor.

[flo-wer]

In #67627 (comment) I wrote a detailed comment about both the derivative integration and the statistics integration that both provide the same derivative function. Both implementations have the same bug that a value of 0 is never reached but with different origin.

[sophof]

[snip]

we shouldn't need to create a work around for the incorrect behavior of the derivative sensor.

It's off topic but I still felt the need to react. I really don't like the condescending tone of your reaction, it reads as if you expect people did this almost out of spite or some absurd misunderstanding of what the definition of a derivative is. I'd suggest starting from the assumption that people not only tried their best but that any mistake is an honest mistake and not because they are idiots. Maybe they even know something you don't (like that it isn't trivial/automatic within HA to implement that a "sensor shouldn't need to see a change in the source sensor for it to re-calculate" unless you go full polling which you generally want to avoid).

Not just that, your reaction is in a issue where everyone including the code owner agrees it happens, that it shouldn't happen and people are actively trying to fix it in the best way possible.

You also seem to be overstating how easy it is to spot edge cases like this one. Most sensors will report with some frequency and most sensors will report values that trend to a zero derivative before being stable. At least this is the case for all my sensors where I use it for. Therefore the only way to notice - at least for me - is an issue like this one. It might have been realized earlier, each specific edge case on its own seems obvious in hindsight when you notice them, but take them all together and you'll always miss a few. This is also obvious from the fact that the statistics sensors shares this bug.

I don't understand why you reacted at all to be honest, all you added here was a sour taste for someone who contributed to this integration (speaking for myself) and a general condescending rant.

[neil1111]

I’m having a similar problem. I setup a derivative to track temperature changes from an Aotec sensor. The temp is changing, but the derivative never does. See the following charts for both:

Here are the settings:

Hi. I’m bumping the above post from last month, please. My derivative is always 0, even when the underlying sensor is changing. Can someone please help me with the configuration? Thanks very much!

[finity69x2]

I really don't like the condescending tone of your reaction

Sorry that you took my reply as condescending.

It was just meant to clarify and specify what the problem was.

it reads as if you expect people did this almost out of spite

not at all. I honestly fail to see where you could read that unless you are reading more into it in order to be upset about something.

Not just that, your reaction is in a issue where everyone including the code owner agrees it happens, that it shouldn't happen and people are actively trying to fix it in the best way possible.

Sorry again but I got none of that by the replies to this issue.

nowhere in this issue did anyone imply that the issue was actively being worked on. if they did then I missed it.

the only suggestion in the issue was that someone suggested a workaround for it and I'm pretty sure it wasn't the code owner. And the codeowner has only replied once in the issue and it was over a month ago now. So I'm not sure how anyone could know that it's being actively worked on.

I only commented that it shouldn't take a work around for this to work as expected.

(just to head this off...) I'm not saying that anyone should be required to fix anything, either.

You also seem to be overstating how easy it is to spot edge cases like this one.

No I'm not.

I literally never said anything of the sort.

Of course, the whole purpose of issues are likely due to edge cases. If it wasn't an edge case then the issue likely would have been fixed long before now.

I, again literally, just added a bit more information in the form of a couple of examples.

But as you said this is not helping the issue get resolved so at that I'll leave it alone and apologize again that you took what I said negatively. No offense was meant to you or anyone else.

[Petro31]

@neil1111 you need a time window. You have zero currently

[sophof]

@neil1111 you need a time window. You have zero currently

Unless it got changed without me noticing (which could be the case) you shouldn't need to set any window. Without a window it should simply always take the derivative based on the last two measurements. You could try checking by setting a 1 second time window, which effectively would have the same effect (the underlying assumption is linear, so the derivative is constant between two data points)

[issue-triage-workflows]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[ildar170975]

up

[sparkelende]

Having issues with this too. Derivative sensor keeps showing a change value even when change is 0.
Any workaround on this?

[mekaneck]

The workaround is to create a template sensor that forces the repeated values to be stored in the database, which then gives the derivative sensor data for it to actually calculate a zero slope. The way to enforce recording of unchanging states is to add an attribute to your template sensor that changes as often as you want to force an entry in the recorder database.

#67627 (comment)

When you create a derivative sensor based off this new template sensor you should no longer have an issue.

[sparkelende]

The workaround is to create a template sensor that forces the repeated values to be stored in the database, which then gives the derivative sensor data for it to actually calculate a zero slope. The way to enforce recording of unchanging states is to add an attribute to your template sensor that changes as often as you want to force an entry in the recorder database.

#67627 (comment)

When you create a derivative sensor based off this new template sensor you should no longer have an issue.

Not very experienced with HA coding, usually trial and error.. Tried for an hour or 2 but couldn't get a workaround working.
Could you please help me in the right direction, maybe even give me a litle bit of code I can use? :)

I have a Wemos d1 mini flashed with tasmota connected to a YF-B6 hall-effect sensor for measuring flow for a pump. Set it up as a counter in Tasmota.

So that gives me sensor.tasmota_systeempomp_counter_c1 in my HomeAssistant that counts the total of pulses.
But I need pulses per minute to calculate flow, so i created:

• platform: derivative
  name: systeempomp_puls_min
  unit_time: min
  source: "sensor.tasmota_systeempomp_counter_c1"
  time_window: "00:00:10"
  round: 1

This gives me the pulses per minute i need to calculate using 6.6Hz per l/min as a given spec for my YF-B6 sensor. So pulses per minute / (6.6Hz * 60 seconds) gives me the actual flow.

• platform: template
  sensors:
  systeempomp_liter_min:
  unit_of_measurement: "l/min"
  value_template: '{{ ((states("sensor.systeempomp_puls_min") |float) / 396) | round(1) }}'

So now I have the flow in liters per minute, which is pretty much is 99% accurate compared to the flowmeter on the Grundfos Alpha 2 pump itself.

The issue here is when the pump is turned off, flow drops from 15l/min to readings like 3 or 6 l/min, but never to 0 which messes up all sorts of calculations used for efficiency for my heatpump.
Noticed this issue day before yesterday ,explaining my lack of heatpump efficiency in recent weeks.

But it also looks totally stupid in the system overview i created.

Would really appreciate any help. Thanks in advance!

[sparkelende]

So I finally managed to get the workaround going. Ill share it for everyone experiencing similar issues.

First create the template sensor for forcing database entry of the counter. In my case for one of my pumps.

template:

• sensor:
  name: "warmtepomp_pulsen_totaal_templatesensor"
  state: >
  {{ (states("sensor.tasmota_warmtepomp_counter_c1")) }}
  attributes:
  minute_counter: "{{ now().minute }}"

After that, create the derivative sensor:

• platform: derivative
  name: wp_derivative_van_totaal_pulsen_templatesensor
  unit_time: min
  source: "sensor.warmtepomp_pulsen_totaal_templatesensor"
  time_window: "00:00:10"
  round: 2

Now you have the pulses per minute and you can create a different sensor with calculations to flow. Before, it didn't go to 0 when stopping the pump, now it does.

@mekaneck thank you for pointing me in the right direction.

[g1za]

Another solution/implementation, like the one initially suggested by @Petro31, is proposed here:
#88655

[edgarveersel]

For me this was working well until 2023.03 and only went wrong since 2023.04. I don't know the cause, but I assume the deduplication that was done in the database could be it...

Just for informational purposes: I created a workaround by using Node Red. It reads the incoming values, calculates the derivative and writes the result to a numeric helper. Here are the differences:

Derivative Helper

By Node Red

[jbsmi]

I have in the same problem with derivative that don't turn to 0, can you please share the Nod-Red code to create a helper with the right value?
I don't have this problem with mqtt value's, because of the "force_update: true" function with custom sensors, to update the value to HA even there is no change
But with modbus sensors this function is not supported in modbus.yaml

[edgarveersel]

Hi jbsmi,
I am currently on vacation, and I can't reach Node Red at the moment. But I can give you a functional description if that helps.
In my case, I read the source values from MQTT, of which I know come in every 180 seconds (Tasmota telemetry setting).
So I use an MQTT in node to read the value, then use a Change Node to calculate:

msg.derive = (msg.sourceValue- $flowContext('prev_sourceValue')) * (60/3)

Then I use a Change node to store the current value as a flow parameter flow.prev_sourceValue
Lastly, I store the derivative value back to a Number Helper in HA.

[karwosts]

I'm looking at porting over the max_sub_interval option which was recently added to Riemann here to Derivative. Seems to be somewhat working locally for me, just need some more time to polish the code and write tests. So hopefully something is coming before too long.

[Holgics]

I'm having same problem and tried to fix with a workaround to store value once per minute even there is no change.
But that doesn't work for me. Where is my mistake?

Here the config with the newly created sensor and the derivate sensor. Derivate allways stays at the same value and doesn't change:

- sensor:
      - name: "PV Solaredge update per minute"
        unique_id: "pv_solaredge_update_minute"
        state: >
           "{{ (states("sensor.tasmota_hichi_itron_ace3000_total_inz1")) }}"
        attributes: 
            minute_counter: "{{ now().minute }}"      
sensor:      
      - name: "PV Solaredge current power"
        platform: derivative
        source: sensor.PV_Solaredge_update_minute
        unit: kW
        round: 2

[erkr]

@Holgics
Your current template sensor will only be triggered when de state of the source sensor (sensor.tasmota_hichi_itron_ace3000_total_inz1) changes.
Make the sensor trigger based, with both a state trigger for the source sensor, and a time trigger of e.g every minute.

In that case the sensor will update the attribute even if the state doesn't change

[Holgics]

Thanks for your feedback. I'm not a HA expert, so could you please give me code for this sensor in my example?
Many thankjs

[1iverea9er]

@Holgics
In order for the update to be committed in HomeAssistant, the new sensor value must not be the same as the old one.
You can include some noise in the original sensor for this purpose

template:
  - trigger:
      - platform: time_pattern
        minutes: "/1"
      - platform: state
        entity_id: sensor.tasmota_hichi_itron_ace3000_total_inz1
        to: ~
    sensor:
      - name: "PV Solaredge update per minute" 
        state: "{{ (states('sensor.tasmota_hichi_itron_ace3000_total_inz1') | float + [0.01, -0.01] | random) | round(2) }}"

[erkr]

This is a working example for the template:

- trigger: 
   # work around 'source' sensor for the derive integrations
  - platform: time_pattern
    minutes: "/2" 
  - platform: state
    entity_id: 
     - sensor.total_gas_usage
    not_to:
      - "unknown"
      - "unavailable"
  sensor:    
  - unique_id: updating_total_gas_usage 
    name: updating_total_gas_usage
    availability: "{{ has_value('sensor.total_gas_usage') }}"
    state: "{{ states('sensor.total_gas_usage') }}"
    attributes:
      dummy: "{{ now().minute }}"
    unit_of_measurement: m³
    device_class: gas

[erkr]

Examples enough 🤓

[Holgics]

I tried random values now as I couldn't get run second example.

- sensor:
      - name: "PV Solaredge update per minute" 
        state: "{{ states('sensor.tasmota_hichi_itron_ace3000_strom_total_inz1') + [0.01, -0.01] | random) | round(2) }}"
 

But also here I receive an error message:

Invalid config for 'template' at configuration.yaml, line 81: invalid template (TemplateSyntaxError: unexpected ')') for dictionary value 'sensor->0->state', got "{{ states('sensor.tasmota_hichi_itron_ace3000_strom_total_inz1') + [0.01, -0.01] | random) | round(2) }}"

[Holgics]

ok, removed the ')' after random and could restart HA.
But value for derivate sensor still remains on old value, but I believe that I need to check tomorrow when value changes again?

[Holgics]

ok, removed the ')' after random and could restart HA.
But value for derivate sensor still remains on old value, but I believe that I need to check tomorrow when value changes again?

[Holgics]

ok, removed the ')' after random and could restart HA.
But value for derivate sensor still remains on old value, but I believe that I need to check tomorrow when value changes again?

[erkr]

You don't need to make the template more complex with random for state.
Derivative will trigger when only the attribute changes. Benefit it that the state is not slightly off

But did you create a trigger based template sensor?? Your example suggests not! Normal template sensors only update when the state changes, even when you use random

[1iverea9er]

@Holgics , sry, I didn't check template. Code edited, try.

[Holgics]

@1iverea9er : didn't work

@erkr :

I tried this, but also didn't work. I even cannot see a entity updating_PV_Solaredge
but config.yaml doesn't show an error.

[1iverea9er]

@Holgics
you're not indenting correctly.
trigger and sensor should be on the same level

see doc's attentively
https://www.home-assistant.io/integrations/template/

[Holgics]

thanks very much, now it works :-)

So current power is 0, this is correct.
Will see tomorrow whether this will work also if values changes.

[Holgics]

i'm running into the next problem now.
See below, the update frequence is higher than changes of the value. So derivate sensor delivers 0. I would like to change this to 15 min, so only all 15 min status of the signal should change. Which parameter needs to be changed?

[AleXSR700]

See below, the update frequence is higher than changes of the value. So derivate sensor delivers 0.

time_window if you know the update frequency and/or you don't want every little jump to alarm you

[Holgics]

I cannot get it working. I have set time windows to 15 min, but sensor thern shows same behaviour. The update frequence isn't known as it is no fix time fence - it depends on whether the counter is changing it's value. As this is a PV counter it changes during the day b ut not over night,.

[mark007]

Just to pitch in to say this is also affecting me, when my average temperature in my house isn't changing (which is common as its quite stable), the derivative sensor is showing sometimes very wrong values so ideally it should be fixed to not only trigger on the original sensor changing, but at some other regular period I would imagine.

[oywino]

I tried to establish a Trend Sensor Helper, and the result is this:

What does the red exclamation mark mean?

[Petro31]

I tried to establish a Trend Sensor Helper, and the result is this:

What does the red exclamation mark mean?

Viking, the forums are for support.

[oywino]

the forums are for support.

Oops, sorry - my mistake. Won't happen again 😶

[zoic21]

Hello,
Same issue for me derivation sensor don't return to 0....

[oywino]

So I ended up using the derivate sensor, and so far it does what I expected - including returning to zero:

[EDelsman]

It works fine if the sensor changes often enough. And you can see that for you that is the case. It will be more problematic for instance on solar production, which will drop to 0 and then not change for over 16 hours, during which the derivative may stay non zero.

[oywino]

Aha, I see.... Thanks for explaining.
Perhaps the HA team will find a creative solution for that 😊

[Holgics]

The problem is sill there. I have a solar sensor (total increasing) and based on this a derivate sensor to calculate current power.
I had to create a workaround sensor to fix that problem with staying derivate value:

I store the last value in an input variable . If the new value is the same I set my corrected sensor to 0 - otherwise I use the derivate value sensors's value:

Here is the automation to save last value of my hichi input sensor hat I need to compare against:

alias: PV last value update
description: ""
triggers:

• entity_id: sensor.tasmota_hichi_itron_ace3000_total_inz1
  trigger: state
  actions:
• target:
  entity_id: input_number.pv_last_value
  data:
  value: |
  {{ trigger.from_state.state | float }}
  action: input_number.set_value

And here corrected sensor

• sensor:
  • name: "PV with correction"
    unique_id: "PV_with_correction"
    unit_of_measurement: "kW"
    state: >
    {% set sourcesensor = states('sensor.tasmota_hichi_itron_ace3000_total_inz1') | float %}
    {% set last_val = states('input_number.pv_last_value') | float %}
    {% if sourcesensor == last_val %}
    0.00
    {% else %}
    {{ (states('sensor.PV_CurrentPower_as_derivate')) }}
    {% endif %}

So if there is no more a change my corrected value is 0 - originally it stayed on the last value