Indicate when status events are removed from the provider API

[billdirks]

Is your feature request related to a problem? Please describe.

When operators get a clearer picture of the real condition of their vehicles on the streets they will update the status events returned from the status_changes API endpoint. That is 2 calls to the same endpoint with the same parameters can result in different results. This was one motivation for the field publication_time, so one could audit when additional events were added.

Describe the solution you'd like

I am open to discuss any solution to solve this. Here is my proposal:

Adding an additional field called retraction_time which is null if the status change is valid. Otherwise it is the time the status_event has been removed.

Along with this change we should explicitly state that the status changes can't be deleted from the feed but can be retracted via this mechanism.

Is this a breaking change

A breaking change would require consumers or implementors of the API to modify their code for it to continue to function (ex: renaming of a required field or the change in data type of an existing field). A non-breaking change would allow existing code to continue to function (ex: addition of an optional field or the creation of a new optional endpoint).

• No, not breaking

Like publication time we can make this optional for now.

Provider or agency

For which API is this feature being requested:

• provider

Describe alternatives you've considered

Explicitly allowing deletions from the feed.
Adding additional retraction events.

It seems like adding a new field would be the easiest way to provide and consume this change. I'm open for other solutions though.

[marie-x]

I'd like to hear from the providers as to how often retraction/revision occurring in practice, and what the need is.

Agency treats status changes and telemetry as immutable, so, my first reaction is: status changes should be treated like a write-append log, not a mutable database. But that may be at variance with reality on the ground.

[marie-x]

This looks like it's closely related to #315

[alexdemisch]

As a follow-up to the discussion on the call today, I'm wondering if any providers think this may better help account for the stolen scooters problem. It seems like providers treat stolen/lost devices in Status Change in different ways (sometimes with certain event types or perhaps no event at all), and that it takes some time to investigate and make the determination that a device was lost. Being able to revise the status seems like a helpful use case.

[billdirks]

From today's meeting it seems like some users are constantly tailing the provider endpoints and don't look back in time so would not see the updated events. There were a few proposed solutions to this:

1. Making the events immutable and adding a new "delta event" which changes a previous event. The time associated with this delta event is the published time. One issue I see with this is the lack of time locality. That is, if one queries for status events (or trip ends) falling in a range, one may miss the delta event.

2. Add a new "updated_time" field in addition to the retraction field. This would be queriable by the API. That way, if one wants to get the most recent published status events and trips one can query for updated times (instead of status event/trip end times). If one wants to get all the for the events in a time window, one can query by status event time (or trip endpoint time).

I prefer the second proposal because "delta events" seem like a new type of object returned from the endpoints which would be harder to consume and produce and whose format would need to be agreed upon.

[thekaveman]

@billdirks we haven't discussed this issue or your proposal in quite some time. Querying by an updated_time field seems to go against our recent changes for static file datastores. Are you still interested in working on this?

[schnuerle]

I'm closing this since things have changed a lot since 0.4. If anyone would like to bring this up based on the latest MDS 2.0, please open a new issue.