Review the 20 minutes filtering for Libre patched app data source

[psonnera]

AAPS will introduce a new filtering option in the next release. This 20 minutes filtering feature was introduced in xDrip+ when first adapting to the patched app and has the negative side effect to introduce a 5-10 minutes delay in readings. Since this data source is also used with Diabox and Juggluco, it slows AAPS reaction time to correct upwards and downwards trends. As Dexcom also changed strategy regarding this, could this filtering being made optional / modified or removed? https://www.diabettech.com/cgm/back-smoothing-or-not-back-smoothing-is-that-the-question/?utm_source=dlvr.it&utm_medium=facebook&utm_campaign=back-smoothing-or-not-back-smoothing-is-that-the-question

[Navid200]

I am not aware of a filter in xDrip other than one that only affects the graph.
So, would you be so kind for me to point me in the right direction in xDrip to find it and educate myself about it?

After I do that, I may have something useful to say related to what have mentioned here.

[psonnera]

@Navid200

xDrip/app/src/main/java/com/eveningoutpost/dexdrip/LibreReceiver.java

Line 140 in b58ca21

List<Libre2RawValue> smoothingValues = Libre2RawValue.last20Minutes();

[psonnera]

[Navid200]

@tzachi-dar This sounds reasonable to me.
Is there no setting to disable this? Can we add one?

[enpa23]

I have rebuilt the app just setting a different value here:

xDrip/app/src/main/java/com/eveningoutpost/dexdrip/Models/Libre2RawValue.java

Line 34 in b58ca21

double timestamp = (new Date().getTime()) - (60000 * 20);

I choose 8 instead of 20, in order to reduce the delay while keeping some smoothing. Setting: FSL3, juggluco +xDrip using "libre 2 patched app" as data source.
As expected, the delay is greatly reduced:

There is a tradeoff between smoothing - which is useful in case of compression lows and other random changes - and latency in shown values, in my experience with FSL3 a 20 minutes averaging is too much, maybe something around 10 could be the sweet spot.

As for patched app as data source, it would be useful to have also the possibility to affect both intercept and slope when adding values for calibration, just as it happens for OOP2.

I

[tzachi-dar]

Here are my 2c regarding this issue. Although this is a libre2 discussion I'm the owner of the oop2 code and not the patched app code, so not too familiar with this.

The smoothing that was introduced in this case is caused because patched-app algorithms use extrapolation for the data. This means that if measured bg in the last 5 minutes is 100, 101, 102, 103, 104 they will show a BG of ~115. Why is the 115? well based on this last 5 minutes data they estimate that in 10 minutes bg will be 115, and knowing that the CGM have delay (mainly because they measure BG in the interstitial fluid and not in the blood). Although this sounds reasonable, this also serves as an amplifier for noise.
The way that the patched app solved this issue was to use the smoothing, which tend to decrease noise.

If I understand correctly, now AAPS is also going to add some smoothing which also adds delay. Questions that I have regarding this are:

1. Will this be optional? Will this be the new default?
2. When looking at this change, did they think about libre, or only about dexcom?

Based on the post in the first message, it seems that the trigger to this is changes in the G7, and I wonder what about people still using G6.

In my opinion (and I don't know much about their change) we should simply not use it (Assuming it is optional). Removing our smoothing since they are using smoothing should probably have the same effect (on AAPS) as not doing both of it. That said, removing the smoothing will have an effect of a jitter graph in xDrip.

OOP2 also has an optional smoothing mechanism based on 5 minutes of the data, and that can also be used by people. I think that it is closer to what dexcom are doing since it does not have the extrapolation part.

HTH
Tzachi

[jbr7rr]

@tzachi-dar :

1. Smoothing is optional in AAPS
2. All sources where regarded AFAIK (It is source agnostic)

A few notes:

• The delay depends on which smoothing method is selected.
• There is still some discusion about this on the AAPS github see: For Discussion: SGV (glucose) Smoothing & AID nightscout/AndroidAPS#2354
• A recent change has lessened a delay that was always there due to recalculation of the value's to a 5 minute interval. See this PR: Fix recalculated data time sync nightscout/AndroidAPS#2303 (This issue was always there, but became more visible in dev due to some visual changes made for smoothing)

Personally I dont think a thing needs to be changed in xDrip+ as for most source smoothing is optional? Including for the patched app source? Also smoothing in AAPS is disabled by default. And it seems that it will stay that way.

Also I dislike the term filtering, it is smoothing, filtering can mean something else. Also smoothing is not limited to 20 minutes in a sense. But that will get way to techinical

[Navid200]

Also I dislike the term filtering, it is smoothing, filtering can mean something else.

That's exactly the outcome of a low-pass "filter". It smooths out the fluctuations.
A low-pass filter does that by attenuating the high-frequency components while leaving the low-frequency components alone.

[jbr7rr]

A filter usually removes values and thus removing noise. In your example low frequencies are removed. With smoothing/filtering BG values are replaced not removed. But well we mean the same thing. So what's in a name?

[Navid200]

A filter usually removes values and thus removing noise.

That's not correct. Only a low-pass filter removes high-frequency noise. A high-pass filter does not.

In your example low frequencies are removed.

Incorrect.
In my example, high frequencies are removed.

With smoothing/filtering BG values are replaced not removed.

Read "Fourier transform". Every signal, in time domain, can be broken down into a series of frequencies (frequency domain). Attenuating high frequency components is what a low-pass filter does. The result, in time domain, is that the points move to form a smoother curve.

When someone says "high frequencies are removes", he is talking about the frequency domain. He does not mean that the points in time are removed.

[jbr7rr]

Yes correct in the low-pass high-pass, read and wrote a bit too fast.. Low frequencies can also be considered noise, for example a 50hz ground buzz in an audio signal :)

Good explanation on the Fourier transform. Have not thought about it this way in this context. And can be applied to what we do with BG values. So ok, I dislike it a bit less now thanks :)

What confused me about the term filtering so far, is that what dexcom does with sensor errors is sometimes called filtering (they are removing faulty values).

[Navid200]

I see what you mean

[jamorham]

If the 20 minute smoothing is not currently option we can make it optional. There is graph smoothing now which doesn't affect broadcasted packets and I have a new smoothing framework coming soon that would allow for different algorithms and parameters etc.

My experience with AAPS was that smoothing helped as otherwise it would change decision a lot during any noise.