perf: improvements drive stats

[swiffer]

This brings several improvements to the Drive Stats Dashboard

• make use of Drives instead of Positions where possible (250ms vs 1ms here)
• reuse queries accross mutliple panels to reduce stress on database (3 queries no longer executed at all)

In addition some oddities have been fixed

• While Panels mentioned "Average" within the queries a "Median" has been calculated, change Panel titles
• Fix visualizations in first dashboard row by gapfilling data for days without drives
• Fix "Exclude Locations" filter (when empty locations containing spaces have been filtered out)

fixes #4199
fixes #4193

[netlify]

✅ Deploy Preview for teslamate ready!

Name	Link
🔨 Latest commit	9dc0f98
🔍 Latest deploy log	https://app.netlify.com/sites/teslamate/deploys/670d6900b4a16c0008fca54f
😎 Deploy Preview	https://deploy-preview-4258--teslamate.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[JakobLichterfeld]

Thanks for your efforts!

On my Raspberry Pi 3B+ this dashboard is now unusable. Loading ~ 46 seconds. Median per day not showing up at all.

2024-10-13.10-12-11.mov

[swiffer]

ok, that went totally in the wrong direction 🤔 will have a look...

[swiffer]

@JakobLichterfeld - can you run an explain analyze on this query please

explain analyze WITH since as (
	SELECT date at time zone 'UTC' as date FROM positions
	WHERE car_id = 1
	ORDER BY date ASC
	LIMIT 1
),

actual AS (
	SELECT
		date_trunc('day', start_date at time zone 'UTC') AS date,
		count(*) AS distance
	FROM drives
	WHERE car_id = 1
	GROUP BY 1
),

base_line AS (
	SELECT date from generate_series(date_trunc('day', (select date from since)), NOW(), '1 day'::interval) date
)

SELECT
  base_line.date as time,
	convert_km(COALESCE(actual.distance, 0)::numeric, 'km') as "distance_km"
FROM base_line
LEFT JOIN actual ON actual.date = base_line.date
WHERE base_line.date BETWEEN '2023-10-13T08:20:26.94Z' AND '2024-10-13T08:20:26.94Z'
order by base_line.date;

[swiffer]

never mind, found it!

[swiffer]

ready for retest

[JakobLichterfeld]

@JakobLichterfeld - can you run an explain analyze on this query please

-------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=208989.57..208989.71 rows=56 width=40) (actual time=36763.214..36769.724 rows=366 loops=1)
   Sort Key: date.date
   Sort Method: quicksort  Memory: 31kB
   InitPlan 1 (returns $0)
     ->  Limit  (cost=208795.94..208795.94 rows=1 width=8) (actual time=35885.387..35885.393 rows=1 loops=1)
           ->  Sort  (cost=208795.94..224757.68 rows=6384697 width=8) (actual time=35874.792..35874.796 rows=1 loops=1)
                 Sort Key: ((positions.date AT TIME ZONE 'UTC'::text))
                 Sort Method: top-N heapsort  Memory: 17kB
                 ->  Seq Scan on positions  (cost=0.00..176872.46 rows=6384697 width=8) (actual time=0.284..31167.745 rows=6383679 loops=1)
                       Filter: (car_id = 1)
   ->  Hash Right Join  (cost=127.22..192.01 rows=56 width=40) (actual time=36760.720..36762.525 rows=366 loops=1)
         Hash Cond: ((date_trunc('day'::text, (drives.start_date AT TIME ZONE 'UTC'::text))) = date.date)
         ->  HashAggregate  (cost=112.15..145.59 rows=2229 width=16) (actual time=80.388..81.257 rows=700 loops=1)
               Group Key: date_trunc('day'::text, (drives.start_date AT TIME ZONE 'UTC'::text))
               Batches: 1  Memory Usage: 145kB
               ->  Seq Scan on drives  (cost=0.00..101.01 rows=2229 width=8) (actual time=28.246..61.703 rows=2107 loops=1)
                     Filter: (car_id = 1)
         ->  Hash  (cost=15.01..15.01 rows=5 width=8) (actual time=36680.199..36680.201 rows=366 loops=1)
               Buckets: 1024  Batches: 1  Memory Usage: 16kB
               ->  Function Scan on generate_series date  (cost=0.01..15.01 rows=5 width=8) (actual time=36674.824..36675.131 rows=366 loops=1)
                     Filter: ((date >= '2023-10-13 08:20:26.94+00'::timestamp with time zone) AND (date <= '2024-10-13 08:20:26.94+00'::timestamp with time zone))
                     Rows Removed by Filter: 1471
 Planning Time: 24.053 ms
 JIT:
   Functions: 26
   Options: Inlining false, Optimization false, Expressions true, Deforming true
   Timing: Generation 11.140 ms, Inlining 0.000 ms, Optimization 20.642 ms, Emission 553.109 ms, Total 584.891 ms
 Execution Time: 38848.518 ms
(28 rows)

[JakobLichterfeld]

ready for retest

you are too fast for my rpi to take up :-)

[JakobLichterfeld]

never mind, found it!

I don't think so. It still takes ages. (45 seconds)

[swiffer]

Aufzeichnung.2024-10-13.104702.mp4

[swiffer]

you are using this one?

f2d13ea

[JakobLichterfeld]

I don't think so. It still takes ages. (45 seconds)

you are using this one?

f2d13ea

Nope. I was faster than my Rpi was able to pull the new image, so I thought I would use the latest one.

Now it runs at instant speed (two refreshes in attached video). Even with last 10 years it is instant. But Median per day is still empty.

2024-10-13.11-15-01.mov

[JakobLichterfeld]

Do you have a preferred merger order for #4253?

[JakobLichterfeld]

But Median per day is still empty.

Just to confirm: In v1.30.1 The average fields are populated correctly on my setup.

[swiffer]

But Median per day is still empty.

Just to confirm: In v1.30.1 The average fields are populated correctly on my setup.

using grafana 11.2.2 locally, will check in grafana 11.0.1

Do you have a preferred merger order for #4253?

i would prefer merging this first and add the additional panels afterwards.

[swiffer]

50% Percentile Calculations aren't available in Grafana 11.0.1 - reverted two panels to using it's own sql query instead of calculating with dataset from another panel. should work now.

[swiffer]

Nope. I was faster than my Rpi was able to pull the new image, so I thought I would use the latest one.

Now it runs at instant speed (two refreshes in attached video). Even with last 10 years it is instant. But Median per day is still empty.

great to hear - the trick is to avoid positions whenever possible / ensure proper index usage / exclude streaming data as described here #4253 (comment) for now...

[JakobLichterfeld]

reverted two panels to using it's own sql query instead of calculating with dataset from another panel. should work now.

With your latest push it works, values seems reasonable for last 7 days but not for last 2.
2 Drives with 20.8 km is correct. The median is a bit off :-)

[JakobLichterfeld]

For comparison with v1.30.1: It counts 4 drives which is in fact correct. There has been 2 ~800m drives as well.

22,3703 km driven in 4 drives means 5,592575 average innit? Ah it is average per day.
Happy to test more, but not today :-) Happy weekend.

[swiffer]

22,3703 km driven in 4 drives means 5,592575 average innit? Ah it is average per day. Happy to test more, but not today :-) Happy weekend.

in v1.30.1 it currently is percentile_disc(0,5) per day.... i actually vote for percentile_cont(0,5) per day instead in case of 2 days timeframe...

[swiffer]

Ok, ended up now basically beeing a rewrite of the dashboard... it could be that numbers are different from before but they should be better now, give it a try.

Some Notes:

• when selecting "Last 2 days" this is a relative timeframe and basically means: last 48 hours. the dashboard now has been adapted to show data from last 48 hours only.
• when selecting "Last 7 days" and you only had drives in day 1 and 2 of the week the estimated mileage most likely was way to high - this should have been corrected now.
• switched to percentile_cont (this should be noticeable on short timeframes only).

would be interesting to see v1.30.1 vs this PR in your instance again

[JakobLichterfeld]

that numbers are different from before but they should be better now

Yes, looks better now.

A median of zero does not make sense imo.

11-15.10.

v1.30.1

pr-4258

11-12.10.

v1.30.1

pr-4258

Last 1 year

v1.30.1

pr-4258

[swiffer]

depends - it looks like you are not driving enough 😆

median means 50 percentile - so if there are more than 50% of days where you are not driving at all then 0 km is correct.

should we switch to average calculations for the two panels ? (like the titles said before).

other approach would be still calculating a median but excluding all days without drives.

[JakobLichterfeld]

depends - it looks like you are not driving enough 😆

Yeah you are right, but don't miss the time on the road that much :-)

median means 50 percentile - so if there are more than 50% of days where you are not driving at all then 0 km is correct.

should we switch to average calculations for all 3 ? (like the titles said).

In this case I vote yes, as it is more intuitve, innit?

[swiffer]

rebased and back to dashboard queries & average calculations. should be ready!

[JakobLichterfeld]

perfect, ty!

[sdwalker]

Went from 1135 to 705 drives over the same 1 year timespan with average distance driven and average kWh used being wrong

Before:

After:

[swiffer]

@sdwalker thanks for reporting - see #4285 for a fix of drives count.

the average values are different but should be "better" now as these are averages per day (incl. drives without days) instead of median (over days with drives).

so different output expected here.

[sdwalker]

@sdwalker thanks for reporting - see #4285 for a fix of drives count.

Drive count is fixed

[sdwalker]

the average values are different but should be "better" now as these are averages per day (incl. drives without days) instead of median (over days with drives).

so different output expected here.

Count me in the group that preferred the median values