Future kuksa.val API

Current APIs / protocols

SDV databroker has been merged into the KUKSA.val project. Because of this, there are currently two different APIs and protocols in use by components and downstream users.

KUKSA.val Protocol
This is implemented by kuksa-val-server.

It is a websocket based interface that is based on VISS v1 and VISS v2.

It deviates from the standards by using the "set" method to set "current values" of all signals. As a result of this, it also needed to introduce the concept of attributes and is using these in order to set "targetValue" for actuators.

It provides users with the ability to get / set and subscribe to values and target values of VSS signals. It also allows the user to get metadata of the VSS signals.

A python library / API (kuksa_client.KuksaThreadClient) is available, which provides an implementation of this protocol.
sdv.databroker.v1

This is implemented databroker.

It is a gRPC API / protocol that isn't based on a standard but instead tries to follow the semantics of VSS by providing a user facing API that allows users to get, set and subscribe to signals as described by the VSS model. Subscriptions are using a streaming RPC where user provide a SQL based query to select which signals to subscribe to and optionally apply conditions for when to receive notifications.

It uses separate methods for setting "current values" of signals, and providers can choose to either stream these or update them using unary calls.

In addition to these, there is a unused gRPC service implemented by kuksa-val-server.

~~kuksa_grpc_if~~

It is a gRPC API / protocol that resembles the KUKSA.val Protocol described above. It's not used actively by any downstream users (?).

New API

Design of `kuksa.val.v1`

The design is based on the previous gRPC API in kuksa-val-server.

The specific design goals of the API have not been clearly described as far as I know, but the Roadmap states that:

What: Trying to integrate the various KUKSA components (better) with databroker and future-proofing API

Why: Integrating some things we learned from the past years and take VSS developments into account. Enabling more of the existing KUKSA.val infrastructure to work with databroker

Design considerations

Here's the issue for the new API and the linked PR #320 where (some) of the design discussions took place.

Excerpts from the discussion:

Open questions:

Do we want the "Update" streaming also in base? It does seem useful for some feeders (even though can feeder is not using it atm)

We still need to think how/if at all we want to unify some of the metadata things. kuksa.databroker and kuksa.val-server differ a little bit here: val-server can do "everything"; but basically because it does not care at all, it just gives you verbatim JSON snippets of the underlying VSS model. databroker gives out structured data, but that of course then is limited to the field it supports directly. One important thing, that probably apps/feeders might want to know is at least querying the datatype of an entry. Also the "registering" of datapoints is almost but not quite entirely unlike the updateMetadata/tree in val-server and again, teaching the server new entries during runtime is a valid thing

Some rationale for decisions/changes

in the end decided to not use fieldmasks, for two reasons

I assume they would be needed often (I think most apps will want to restrict information they get/query specifically for what they need), and that is not the most efficient thing to do doing all thos handling of strings on the wire and in grpc libs

The approach now is defintely "less generic", and more tied to the VSS view of the world. I think that is good int his case

Modelled a Datapoint more completely following VSS

Used "oneof" pattern for optional fields

Did not add the "Streaming set" functions -> Currently we have literally almost no idea about our performance, or even the requirements, or if we are even the bottleneck. This can always be added later

Did not differentiate between "Single Set/Get" and "Batch Set/Get", I think the functions now cover everything, and we can always do the "single without needing to create a list" sugar-coating in the SDK

Moved the status (as in "server status", i.e. acces denied, path not found etc. out of the (VSS) values, because it is a different thing. Still retained the ability to give individual feedback to set/get paths

Since the current design is mostly based on the protocol used by kuksa-val-server, it is following the same principals. The design considerations and design choices of the databroker API have not been fully evaluated or taken into account (at least not observably). This can be partly explained by the lack of documentation pertaining to the design of the databroker API, i.e. what cannot be observed in it's current state of implementation cannot be evaluated by all involved.

This document is the start of an effort to change that as well as to document the design considerations of the new API. Some of the design considerations / choices are already reflected in the current databroker API & implementation, but missing in kuksa.val.v1. Others are reflected in the new API but not in databroker.

In order to ensure that the new API represents an improvement over current state of affairs, these should be taken into account as well.

VSS signals - Users vs providers

The Vehicle Signals Specification (VSS) and Vehicle Information Service Specification (VISS) describes the standardized signals available (or not) in a vehicle. Both standards also describe how users interact with these signals.

They can read and subscribe to actuators, sensors and attributes.
They can set actuators.) (and only actuators). For VISSv2 specifically, see (read, subscribe), update)).

VSS and VISS does not specify how signals are provided into the VSS server / model.

These two aspects of interacting with the signals can thus be divided into:

Provider
A provider is providing a signal (sensor, attribute or actuator) to the VSS tree.
A provider can also use signals as any other user.
User
A user (or client) is using the signals in the VSS tree (without providing any of them itself).

where the VSS and VISS* specifications only specify the User part.

When designing the databroker API, the method kuksa.val uses for providing signals was investigated. In short, kuksa-val-server takes the approach of trying to shoehorn the need of signal providers into the existing VISSv2 protocol. This is problematic for several reasons:

By reusing, but changing the meaning of the terminology used in VSS, there is a constant dissonance between the terms used. This is a recipe for confusion.
By deviating from the standard in this particular way, a standards compliant VISSv2 client cannot use it in any meaningful way.
It makes it harder to actually provide a standards compliant VISSv2 in the future .
By using the same methods for both signal providers and users of signals it's harder (or impossible) to customize them for their different requirements.

With this in mind, databroker chose to make a clear distinction between signal providers and signal users. It doesn't use this terminology though. It does this by splitting the interface into two separate services, which are customized for their different requirements / use cases. It doesn't need to be implemented in this way in order to achieve the same design goal, though.

Enable "easy to use" user facing API

This is meant to illustrate what type of user APIs that can be created depending on what the provider API looks like (assuming we have one).

Use case:

The user wants to lock a door and know when it's done / whether it worked.

The kuksa.val.v1 or "key-value store" way.

Something conceptually like this:

User calls set(Vehicle.Door.Locked, field=TARGET_VALUE, true)
User calls subscribe(Vehicle.Door.Locked, field=VALUE)
Provider (subscribing to TARGET_VALUE) receives the request and starts actuating, providing VALUE when it changes.
User is notified when VALUE turns to true, and concludes that the door has now been locked.

But what happens if the door fails to lock, e.g. the door is not close or the actuator is broken?

What should the user subscribe to for this information?
And how long should it wait before concluding that it failed?
And what happens if there is no provider of this actuator?

Another question, a bit convoluted for a quick actuator like this (but applicable for slower moving things), is happens if another user calls set(..., false) before the actuator change has taken place?

This can be solved by subscribing to both VALUE and TARGET_VALUE.

User calls set(Vehicle.Door.Locked, field=TARGET_VALUE, true)
User calls subscribe(Vehicle.Door.Locked, fields=[VALUE, TARGET_VALUE])
Provider (subscribing to TARGET_VALUE) receives the request and starts actuating, providing VALUE when it changes.
User is notified when VALUE turns true, and concludes that the door has now been locked, or the user is notified when TARGET_VALUE turns false, and knows that the operation was cancelled.

The user API + provider API way.

So what could this look like if we instead had an "easy to use" user API + a provider API and the server in between.

Something like this:

User calls set(Vehicle.Door.Locked, true)
Server receives the request and sends an ACTUATOR_TARGET value to the provider of this signal.
The provider receives it and starts actuating and provides VALUE back to the server when it changes.
The provider sends "actuator target reached" back to the server.
The server sends a (success) response back to the client.

Or in case of failure:

User calls set(Vehicle.Door.Locked, true)

Server receives the request and sends an ACTUATOR_TARGET value to the provider of this signal.
The provider receives it and starts actuating but notice that it fails, or that it's not allowed at the moment.
The provider sends "actuator failed" or something back.
The server sends a response "actuator failed" back to the client.

Or if there are no providers.

User calls set(Vehicle.Door.Locked, true).
The server knows that there are no providers, providing that signal.
The server sends a response "actuator failed" or "not available" back to the client

This latter approach would seem to represent an easier to use API for the user/library.

Note Doing it like this puts the requirement to know the details of the actuator on the actuator provider.

The actuator provider is better suited to know of reasonable timeouts etc in comparison to the users of signals (or the server). The user doesn't need to know how long to wait for something or to which error to subscribe. The server would only have to handle the presence detection which is a generic feature that doesn't require knowledge of sensor specifics.

Performance / runtime footprint

Providers, especially of sensor data, are often setting values in rapid succession over long periods of time. Using unary GRPC calls for Set operations, is less efficient in terms of throughput when compared to GRPC streams. It's also more CPU intensive.

The current design of kuksa.val.v1 only provides unary call to set values. This represents a pure regression when compared to the databroker API.

It's not a huge issue (in practice) if users avoid using kuksa_client.KuksaClientThread(). If they use that, I would say it's barely usable for e.g. CAN in it's current form.

Python performance setting signal values

Set only	type	throughput
kuksa_client ⁽¹⁾		~ 80 / s
kuksa.grpc ⁽²⁾	async	~ 2 500 / s
kuksa.val.v1 ⁽³⁾	async	~ 6 500 / s
kuksa.val.v1 ⁽³⁾	sync	~ 9 000 / s
databroker ⁽⁴⁾	sync	~ 26 000 / s

¹ kuksa_client is using kuksa_client.KuksaClientThread()

² kuksa.grpc is using kuksa_client.grpc without the legacy kuksa_client.KuksaClientThread() wrapping it

³ uses the generated val_pb2*.py python lib directly

⁴ uses the generated collector_pb2*.py python lib directly

Improvements:

Higher throughput.
Reduced CPU load.
Lower latency (probably, hasn't been measured)

What's needed:

Introduce a streaming (GRPC) interface for providing sensor data.

Provider control and provider capabilities

Open questions:

Should the "change type" of a sensor (i.e. CONTINUOUS vs ON_CHANGE) be decided by providers or in the VSS metadata? It only makes sense for consumers to request their preferred rate of updates when they are subscribing to a sensor of type CONTINUOUS. That would be an argument for providing this information as part of the VSS metadata, so that it doesn't vary between vehicles.

Control the rate of updates

Users of (continuous) sensor data can have different preferences with regard to how often they would like to receive updates. E.g. Vehicle.Speed is updated 100 times per second, but a consumer would only need it 1 time per second. This would introduce unnecessary processing requirements on the consumer (and provider).

Currently there is no way for databroker to know how often a provider should provide updates. There is also no way for clients to instruct databroker of what rate they want. Sensor data is just sent at the rate it is received and providers are just sending sensor data at the rate they themselves decide.

If a consumer can communicate this wish, there are several options for improvement.

Improvements:

Reduction in load for consumers by adapting the update rate based on their preferences.
Reduction in load for for providers by propagating needed update rate.
Reduction in load for databroker by disabling unneeded providers.

What's needed:

Introduce a way for clients to tell databroker of their preferred rate of updates.
Introduce a way for databroker to tell providers of highest needed frequency of sensor data to which they can then adapt. [probably needs] open stream databroker -> provider

Other considerations:

Setting the desired rate of update would only make sense for sensors of type CONTINUOUS. Sensors of type ON_CHANGE would always provide updates when the value changes. It could also make sense to introduce a way to request a momentary value from a provider, which would be used if a consumer only requests a momentary value (and doesn't subscribe).

Differentiate between different providers of the same VSS data

Different sensors can provide data that is mapped to the same VSS signal / entry. This data can be of different resolution and / or quality. For example, an accelerometer can be used to infer the current speed of a vehicle, but a speedometer would probably provide a higher quality measurement. In the same way that a consumer could instruct databroker of a preferred update rate, it could also instruct the databroker of what accuracy of a sensor it needs.

It's currently possible for multiple providers to set the same VSS entry, but there is no way for databroker to differentiate between them in any way.

It could make sense to introduce a way for providers to describe themselves in order to make it possible to differentiate between them with regard to update rates, power consumption, accuracy or quality of their sensor data.

This would give databroker the clients a way to to differentiate (and choose) different sources of data and make informed decisions based on that.

Improvements:

Choose between providers based on available update frequency.
Fallback when sensor information from one sensor isn't available.

What's needed:

Introduce a way for providers to describe their capabilities and properties of their provided sensor data.

Optional improvements:

Choose between providers based on needed quality / accuracy of sensor. [needs] control plane, i.e. an open stream databroker -> provider
Consumers can get extended sensor data information.

Optionally needed:

Introduce a way for consumers to tell databroker of their needed quality / accuracy of VSS signal.

Data Aliveness/Availability

The VSS signals / datapoints that are accessed through databroker can have a value and a timestamp. If they have never been set, they will have neither.

There is no way for databroker to know if a value is up to date or not, since it doesn't have any information with regard to how often it should be updated or a way to determine if a provider has stopped providing data.

For signals with a fixed update rate (sensors of type CONTINUOUS), it would theoretically be possible for either clients or the databroker to determine if a signal is up to date, by keeping track of the time since the last update.

The providers of sensor data would be better suited to know the rate of update, and if the databroker where provided this information, it could automatically determine if a sensor hasn't been update within it's expected time window.

For signals that only update based on events (i.e. a door opens), this isn't possible. Tracking the liveness of these signals would either require the providers to continuously send the same value even though it hasn't changed, or to have an open connection or another heartbeat mechanism between databroker and the provider to detect a missing provider.

If there was a way to determine the availability of providers, the databroker could automatically determine that a signal was stale if it's provider is no longer available.

Improvements:

Track availability / liveness of a VSS signals.

What's needed:

Introduce a way to signal that a signal is available / not available (in kuksa.val.v1).
Introduce a way for providers to tell databroker of a signals time to live (TTL).
Introduce a way for databroker to track availability of providers (and which VSS signals they are providing). [needs] an open stream provider -> databroker or databroker -> provider
Implement tracking of TTL in databroker to automatically set unavailable status to signals that have not been updated in time.

Other considerations: Attributes probably don't need to have aliveness functionality. They would be unavailable if they have never been set, but since they shouldn't update at runtime, once set they should be valid indefinitely.

Missing features from `sdv.databroker.v1` in `kuksa.val.v1`

Sort list: What features would be lost if removing sdv.databroker.v1 today

Registration of new datapoints
SQL queries
Streaming updates (i.e. worse performance)
Connectivity check (no streaming interface)

Example of a Provider™ API / method

Exploring a design of a bidirectional streaming API

This represent one way to design an interface that would enable most of the improvements listed above and provide a clear path forward for introducing them.

Design choices

In this design, a single bidirection stream is used to provide everything needed by providers: rpc Provide(stream ProviderRequest) returns (stream ProviderResponse);

This combines the control channel and the data channel into one. An alternative would be to split it into two bidirectional streams, one for control and the other for data. I'm not sure which makes the most sense.

Stream

By having an open stream (at all) between databroker and the provider, both ends can detect if the other goes away.

Furthermore:

A stream from provider -> databroker:

Enables higher throughput for sensor data updates and lower CPU usage.

A stream from databroker -> provider:

Provides a way for databroker to send control commands to this provider.
Provides a way for databroker to send errors to this provider.

Bidirectional stream

A bidirectional stream between Provider <-> databroker

Provides databroker with a way to associate information sent from the provider (e.g. capabilities, which actuators it provides, which sensors etc) with the stream it uses to control it.

Actuators

VSS defines three types of signals:

Attribute
Sensor
Actuator

An actuator acts as both something you can actuate and something providing values (a sensor). It's not even necessarily the same component providing the implementation of these separate concerns. With this in mind, a provider providing an VSS actuator would in this design provide an Actuator(path) to the server in order to receive ActuateCommands, and provide Sensor(path) and send SensorData when they are providing sensor data.

The alternative would be to duplicate everything in Sensor for Actuators.

Overview

The stream in each direction would consist of different "messages" implemented with oneof {...}.

In the direction of Provider -> Server, at least three types of "messages" would flow⁽²⁾:

SensorData containing data streamed by the provider.
Sensor containing sensor information from the provider.
Actuator containing actuator information from the provider.
Attribute containing attribute including the value. ⁽¹⁾

In the direction of Server -> Provider, at least three types of "messages" would flow:

ActuateCommand, tells actuator to actuate (triggered by someone setting actuator targets).
SensorCommand, controls behaviour of a sensor, i.e. "start", "stop", "try to use this update frequency" etc..
Error, an error occurred. One type at the moment. It would probably make sense to split it into errors that can occur at any time & errors that are directly caused by things the provider provided.

⁽¹⁾ It would probably make sense to introduce a separate Error in the direction of Provider -> Server. Currently, the only errors in that direction is ReadError as part of the sensor data.

⁽²⁾ It's possible that it makes mores sense to provide a separate unary RPC call for setting attributes, since attributes (probably) don't update frequently and (probably) wont need availability status etc..

service VAL {
  ...
  rpc Provide(stream ProviderRequest) returns (stream ProviderResponse);
}

message ProviderRequest {
  oneof provide {
    Sensor sensor = 1;
    Actuator actuator = 2;
    Attribute attribute = 3;
    SensorData sensor_data = 4;
  }
}

message ProviderResponse {
  oneof command {
    ActuateCommand actuator_command = 1;
    SensorCommand sensor_command = 2;
    Error error = 3;
  }
}

...

The full .proto specification for this proposal can be found here.

In the same branch there is a working prototype of a client and server showcasing its usage.

This includes:

Rate control
Sending Start / Stop instruction to the provider
Availability detection of providers
Availability detection of server

It also showcases that it's possible to wrap this GRPC interface in a simple library for users, i.e.

    provider = Provider()
    provider.start()
    for speed in read_current_speed():
        provider.set("Vehicle.Speed", speed)

    provider.stop()

Appendix

Provider API (expanded proto)

import "google/protobuf/timestamp.proto";
import "kuksa/val/v1/types.proto";

service VAL {
    // ... The messages and methods e.g. `Get`, `Set`, `Subscribe` that
    //     are also part of `VAL` are left out (for clarity)
    rpc Provide(stream ProviderRequest) returns (stream ProviderResponse);
}

message ProviderRequest {
  oneof provide {
    // Tell the server we are providing a sensor
    Sensor sensor = 1;

    // Tell the server we are providing an actuator
    Actuator actuator = 2;

    // Set an attribute
    Attribute attribute = 3;

    // Actuator
    // Provide sensor data.
    SensorData sensor_data = 4;
  }
}

message ProviderResponse {
  oneof command {
    // Actuator command - tell actuators to actuate
    ActuateCommand actuator_command = 1;

    // Sensor command - control the behaviour of a sensor
    // i.e. "start providing data", "stop",
    //      "use this update frequency" etc
    SensorCommand sensor_command = 2;

    // An error occurred
    Error error = 3;
  }
}

message Sensor {
  string path  = 1;
  string name  = 2;
  bool enabled = 3;

  // Specific for CONTINUOUS sensors
  // optional int32 current_update_hz = 4;
  // optional int32 min_update_hz     = 5;
  // optional int32 max_update_hz     = 6;
}

message Actuator {
  string path = 1;
  // bool enabled        = 3;
}

message Attribute {
  string path = 1;
  Value value = 2;
}

message SensorData {
  // Each `Reading` has a timestamp but can contain values for multiple paths
  // Readings are `repeated` to allow batching.
  repeated Reading readings = 10;
}

message ActuateCommand {
  // Timestamp
  google.protobuf.Timestamp timestamp = 1;

  // Targets
  // path, value
  map<string, Value> values = 2;
}

message SensorCommand {
  bool enable             = 1;
  int32 desired_update_hz = 2;
}

message Value {
  oneof value {
    string string_value      = 11;
    bool bool_value          = 12;
    sint32 int32_value       = 13;
    sint64 int64_value       = 14;
    uint32 uint32_value      = 15;
    uint64 uint64_value      = 16;
    float float_value        = 17;
    double double_value      = 18;
    StringArray string_array = 21;
    BoolArray bool_array     = 22;
    Int32Array int32_array   = 23;
    Int64Array int64_array   = 24;
    Uint32Array uint32_array = 25;
    Uint64Array uint64_array = 26;
    FloatArray float_array   = 27;
    DoubleArray double_array = 28;
  }
}

message Reading {
  // Timestamp
  google.protobuf.Timestamp timestamp = 1;

  // Values
  map<string, Value> values = 2;
}

message Error {
  string path          = 1;
  ErrorType error_type = 2;
}

enum ErrorType {
  ERROR_TYPE_UNSPECIFIED     = 0;
  ERROR_TYPE_ENTRY_NOT_FOUND = 1;
  ERROR_TYPE_INVALID_TYPE    = 2;
  ERROR_TYPE_OUT_OF_BOUNDS   = 4;
  ERROR_TYPE_ACCESS_DENIED   = 5;
}

enum SensorError {
  SENSOR_ERROR_UNSPECIFIED   = 0;
  SENSOR_ERROR_INVALID_VALUE = 1;
  SENSOR_ERROR_NOT_AVAILABLE = 2;
}

// ...

(old) kuksa-val-server GRPC API

// The connecting service definition.
service kuksa_grpc_if {
  rpc get (GetRequest) returns (GetResponse) {}
  rpc set (SetRequest) returns (SetResponse) {}
  rpc subscribe (stream SubscribeRequest) returns (stream SubscribeResponse) {}
  rpc authorize (AuthRequest) returns (AuthResponse) {}
}

message AuthRequest {
  string token = 1;
}

message GetRequest {
  RequestType type = 1;
  repeated string path = 2;
}

message GetResponse {
  repeated Value values = 1;
  Status status = 2;
}

message SetRequest {
  RequestType type = 1;
  repeated Value values = 2;
}

message SetResponse {
  Status status = 1;
}

message AuthResponse {
  string connectionId = 1;
  Status status = 2;
}

message SubscribeRequest {
  RequestType type = 1;
  string path = 2;
  bool start = 3;
}

message SubscribeResponse {
  Value values = 1;
  Status status = 2;
}

message Value {
  string path =  1;
  oneof val {
    uint32 valueUint32 = 2;
    int32 valueInt32 = 3;
    uint64 valueUint64 = 4;
    int64 valueInt64 = 5;
    bool valueBool = 6;
    float valueFloat = 7;
    double valueDouble = 8;
    string valueString = 9;
  }
  google.protobuf.Timestamp timestamp=10; 
}

message Status {
  uint32 statusCode = 1;
  string statusDescription = 2;
}

enum RequestType {
  UNKNOWN_TYPE = 0;
  CURRENT_VALUE = 1;
  TARGET_VALUE = 2;
  METADATA = 3;
};

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Future kuksa.val API

Current APIs / protocols

New API

Design of `kuksa.val.v1`

Design considerations

VSS signals - Users vs providers

Enable "easy to use" user facing API

Performance / runtime footprint

Provider control and provider capabilities

Control the rate of updates

Differentiate between different providers of the same VSS data

Data Aliveness/Availability

Missing features from `sdv.databroker.v1` in `kuksa.val.v1`

Example of a Provider™ API / method

Exploring a design of a bidirectional streaming API

Design choices

Stream

Bidirectional stream

Actuators

Overview

Appendix

Provider API (expanded proto)

(old) kuksa-val-server GRPC API

Clone this wiki locally

Future kuksa.val API

Current APIs / protocols

New API

Design of kuksa.val.v1

Design considerations

VSS signals - Users vs providers

Enable "easy to use" user facing API

Performance / runtime footprint

Provider control and provider capabilities

Control the rate of updates

Differentiate between different providers of the same VSS data

Data Aliveness/Availability

Missing features from sdv.databroker.v1 in kuksa.val.v1

Example of a Provider™ API / method

Exploring a design of a bidirectional streaming API

Design choices

Stream

Bidirectional stream

Actuators

Overview

Appendix

Provider API (expanded proto)

(old) kuksa-val-server GRPC API

Clone this wiki locally

Design of `kuksa.val.v1`

Missing features from `sdv.databroker.v1` in `kuksa.val.v1`