Sensor entity value forcasting / prediction

[frenck]

Context

For 2 types of services, I see a generic issue. Consider these services:

• Solar forecasting
• Weather forecasting

While it is cool we can create entities for them, it often boils down to solutions like this:

This results in a massive amount of entities, which isn't really user-friendly either. Buiendarar, in my case, throws in a whopping 108 entities:

Also, when automating on this, e.g., will the solar production go above 1kW today... is cumbersome, as it would require me to check multiple entities in my conditions.

Proposal

I would like to propose a forecast or prediction model to sensor entities. In which we allow an entity to provide forecasted sensor values.

Rough implementation idea:

    @property
    def native_value(self) -> StateType:
        """Return the value reported by the sensor."""
        return self._attr_native_value

    @property
    def native_forecasted_values(self) -> dict[date | datetime, StateType]:
        """Return the forecasted values reported by the sensor."""
        return self._attr_native_forecasted_values

In the UI, this would need some additional help/visualization to make this useful. I can imaging we show a graph, plotting the native value and also plotting the forecasted value in a dotted line (just like we do on the energy dashboard).

In (device) automations we could add support for allowing to check if the predicted/forecasted values so above or below a value within before or after a certain time. We could introduce a new trigger or condition for this, or maybe extend existing ones.

condition:
  - alias: Will it be above 15 degrees today?
    condition: numeric_state
    entity_id: sensor.temperature
    forecast: true
    above: 15

(PS Above condition is a rough idea to sketch the situation, but I really not happy with syntax 😉 )

Some additional notes:

• Forecasted values are not stored in the database state history and are not part of the state restore. That would be a lot of unneeded data to store (especially on state changes).

Consequences

• We get rid of a sensor or two 🤷

[BradleyFord]

When the value is returned will it be at the below: 1. At or before the requested timestamp 2. Next value after the requested timestamp 3. Interpolated based on the value before and after the requested timestamp I assume you would provide the service to also update or totally delete the forecasted value? Do we know if the various providers are also providing forecast confidence days? (I work for vendor if oil refinery data historians...)

…

On Thu, Nov 18, 2021, 4:11 PM Franck Nijhof ***@***.***> wrote: Context For 2 types of services, I see a generic issue. Consider these services: - Solar forecasting - Weather forecasting While it is cool we can create entities for them, it often boils down to solutions like this: [image: image] <https://user-images.githubusercontent.com/195327/142419868-ae3b2ffe-e52a-454a-a923-292438f758e2.png> [image: image] <https://user-images.githubusercontent.com/195327/142419901-bdf9a9b5-559a-4524-a9bc-122b90dfab68.png> This results in a massive amount of entities, which isn't really user-friendly either. Buiendarar, in my case, throws in a whopping 108 entities: [image: image] <https://user-images.githubusercontent.com/195327/142419989-6eeab8fc-fae4-4b96-a544-af06d0f1e104.png> Also, when automating on this, e.g., will the solar production go above 1kW today... is cumbersome, as it would require me to check multiple entities in my conditions. Proposal I would like to propose a forecast or prediction model to sensor entities. In which we allow an entity to provide forecasted sensor values. Rough implementation idea: @Property def native_value(self) -> StateType: """Return the value reported by the sensor.""" return self._attr_native_value @Property def native_forecasted_values(self) -> dict[date | datetime, StateType]: """Return the forecasted values reported by the sensor.""" return self._attr_native_forecasted_values In the UI, this would need some additional help/visualization to make this useful. I can imaging we show a graph, plotting the native value and also plotting the forecasted value in a dotted line (just like we do on the energy dashboard). In (device) automations we could add support for allowing to check if the predicted/forecasted values so above or below a value within before or after a certain time. We could introduce a new trigger or condition for this, or maybe extend existing ones. condition: - alias: Will it be above 15 degrees today? condition: numeric_state entity_id: sensor.temperature forecast: true above: 15 (PS Above condition is a rough idea to sketch the situation, but I really not happy with syntax 😉 ) Some additional notes: - Forecasted values are not stored in the database state history and are not part of the state restore. That would be a lot of unneeded data to store (especially on state changes). Consequences - We get rid of a sensor or two 🤷 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#667>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACOZ74LWNPGPQFUQBE5RV3UMT3OZANCNFSM5IJSO2WA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[sim-san]

I like the idea of forecast as entity.

There is energy company in Germany and Austria which provides an service which forecast the energy price for the next day.

This is a qoute from there website

"Our data feed provides the exchange prices for the next day via a defined web interface. This is the data of the EPEX Spot power exchange, which is updated every day at 14:00 for the next day."

I look fort to make automation based on the energy price of the next day.

[elupus]

Sometimes it is a forecast though. With a lock in when the day ends. Or the price "now" is always just a forecast, and the first non forecast value, is a day old value.

Would be nice to have a way to specify a time offset for entities, ie for when the value is applicable. So when graphed, they get shown offset related to other entities. Potentially, with forecast values graphed after the offset time.

[Danielhiversen]

The spot price in Norway and Sweden is known 12-36 hours ahead (and is not a forecast).

[elupus]

Yes in Sweden that is usually so, but i was thinking of: home-assistant/core#60673 (comment)

[Danielhiversen]

Ahh, understand 👍

[RicardP]

I am missing the "Future Aspect" base function, in History panel it should be visible to the right of present time So History Panel would change name to "Timeline" and inside that panel timeline should be named "History|Future forecast". The future "actually not happened yet" should always be considered a Forecast :)

[balloob]

The downside of making it a property is that in some cases the forecasting costs resources and so we would only want to do it on demand. For example if it's calculating based on recorder history or using some cloud service API.

If we make it a method, we can still support conditions and make it more dynamic:

    async def native_forecasted_value(self, point_in_time: datetime) -> StateType:
        """Return the forecasted values for a point in time."""

[purcell-lab]

In addition to the approach of multiple entities to hold forecasts, I also see a lot of cases of attributes being used to hold forecasts, but all in slightly different schema.

It would be useful to provide some guidance as to a standard schema for forecasts (with extensions for unforeseen elements).

[ianmcorvidae]

I'm not sure how valuable this insight is, but I've been thinking about this a little for my own ideal use and came to a fairly different sort of design. The main observation I had is that forecasting operates basically as bitemporal history and treating it as such could also make for options like correcting past data.

I was thinking that the way to do this would be to add concepts to sensors:
a.) allow a given reading to specify what time it is for as well as what time it's being entered (so a forecast would specify it's for a future time, while a correction of past data would specify a past time, with both specifying now as their entered time), and
b.) allow a given reading to specify its source -- such that if you were to, say, set up multiple solar forecasts, they could each be marked as forecasts of the same PV production entity, but tagged with the different integration names. Possibly also a
c.) to indicate if this is a forecast/correction/direct reading, in case of race conditions.

I think that data retention would need to be addressed, but might be as simple (by default, at least) as holding onto the most recently-entered value for a given time (and maybe throwing out forecasts some period after the actual forecasted time has passed).

[BradleyFord]

One thought on this topic, based on my past experience. Generally the future values are directly stored into the long-term database and are not put into memory.

In HA I think the equivalent would be writing the forecast data into the recorder, so they are not held in the state machine.

But then we have a choice, when the future data becomes approaches the current time:

1. Scrub forecast the data as it passes "now"
2. Leave the data

But a service would still need to be made available for the integration to scrub data X time into the future. For example if you have a forecast X time into the future, and it is updated hourly then you want to scrub the old forecast before writing in the new data.
(Remember the timestamp of the points in the forecast may vary)

[gjohansson-ST]

The suggestion would also be useful for transportation to replace multiple sensors for different departures etc.

[Quentame]

I would better think of: display the time the transportation is planned to arrive (for one specific transportation line), and a prediction to when it will arrive (sooner or later than it should be)

[frenck]

I am going to close this proposal. I'm no longer going to pursue this approach, considering upcoming changes to our services, which will be able to provide capabilities like this.

../Frenck

[larstobi]

That sounds fantastic, @frenck! :-)

I'm a bit lost as to which changes you are referring? Would you mind providing a clue or a link?

[jtwild]

Indeed sounds great! @frenck I am also curious, what upcoming changes are you talking about? Is this already implemented, or do you have a link?
I am working on an entity with a forecast myself currently, so would be very useful information.

[gjohansson-ST]

https://developers.home-assistant.io/docs/dev_101_services?_highlight=service#response-data

[jtwild]

For an example, see the weather.get_forecast service

[Berglund]

I created a PR to add future price-data to the Tibber integration and was directed at this discussion. Really happy to see this being discussed, as I'm also in favor of not having tons and tons of sensors.

In my implementation, I added the future electricity price (today, tomorrow) as attributes to the current price sensor. Is the recommended approach to use a service for providing future prices? Reading you right, I believe that is what you alluded to in your most recent response here, @frenck?

[gjohansson-ST]

Correct. It's recommended to use a service with a response instead of adding sensors or attributes.

[escoand]

@Berglund is there any progress in such a service implementation? would be interested in testing.

[rrooggiieerr]

Hi! I'm aware this discussion is closed, however I have been thinking about forecasts recently and I think some of my considerations might be of added value here.

The current consensus seems to be to use action response data, that leaves a lot of freedom to the implementation. While using action response data might be the right approach I think a more rigid definition of the expected action and response should be considered. Like agreeing that sensor.get_forecasts (like in the weather.get_forecasts action) would be the name of the action and a list of [timestamp, value] is the expected response. Then other places within HA could use this data, like a function that get the timestamp of lowest value from the response data, like Cheapest Energy Hours does.

Also I would like to see a way to show the forecast on the frontend, I'm not sure if action response data can make that possible.

To fuel the discussion I thought of the following areas where forecasts, including visualisation of frontend, could benefit.
Dynamic energy prices
Weather
Traffic
Energy consumption
Sun/moon/tides position
Upcoming song/Playlist