Support Objects in octet-stream codec

[derwehr]

I'd like to propose supporting objects in octet streams, for example, for describing a device's payload data that sends packets with bit sequences containing measurements starting at different offsets.
i.e., a device sends 2 bytes, where bit 1 & 2 are different flags and bit 3-16 are a temperature measurement.

Similiar to this:

"measurements": {
  // ...
  "type": "object",
  "properties": {
    "flag1": {
      "type": "boolean",
      "offset": 0,
      "bitLength": 1,
      // ...
    },
    "flag2": {
      "type": "boolean",
      "offset": 1,
      "bitLength": 1,
      // ...
    },
    "temperature": {
      "type": "number",
      "offset": 2,
      "bitLength": 14
      "scale": 2, // number of digits to the right of the decimal point
    // ...
    }
  }
}

[danielpeintner]

I hope I understand your proposal correctly.
You would like to describe the data exchange measurements with JSON but later one it should be transmitted more efficiently via an octect stream since you know each flag just requires one bit etc, right?

At the moment the default encoding is JSON and as such measurements is transmitted. If you want to do additional things you need to make sure that it is clear "what" the contenttype is and you can also implement your own codec on top of it (using also existing codecs)

see https://github.com/eclipse-thingweb/node-wot#mediatype-support

[derwehr]

I was thinking about the opposite way, where my device sends an octet stream and the JSON above is a possible description of the payload data in the device's TD.

More precisely, I would like to extend the octet stream to not throw an Error at https://github.com/eclipse-thingweb/node-wot/blob/master/packages/core/src/codecs/octetstream-codec.ts#L108, but call a new funtion valueToObject.

If that makes sense, I could try and provide a PR

[danielpeintner]

I think in any case it needs to be balanced. Having said that, you would need a solution for bytesToValue(...) but also for valueToBytes(...).
I wonder whether a generic solution for all different kinds of definitions can be established...

[derwehr]

I thought, that with the data type, bit length, and bit offset defined in the TD, one could re-use the de/encoding method for the respective data type. Thus, shouldn't it be as generalizable as the integer, float, and string operations of the codec?

To extend the example from above

"properties": {
  "measurements": {
    // ...
    "type": "object",
    "properties": {
      "flag1": {
        "type": "boolean",
        "offset": 0,
        "bitLength": 1,
        // ...
      },
      "flag2": {
        "type": "boolean",
        "offset": 1,
        "bitLength": 1,
        // ...
      },
      "temperature": {
        "type": "number",
        "offset": 2,
        "bitLength": 14
        "scale": 2, // number of digits to the right of the decimal point
      // ...
      },
      "unit": {
        "type": "string",
        "offset": 16,
        "bitLength": 24,
        // ...
      },
    }
  }
}

Now sending {flag1: 0, flag2: 0, temperature: 24.51, unit: "°C"} would simply call valueToBytes with each datatype and concatenate the resulting bits. Decoding 0000 1100 0011 0011 1100 0010 1011 0000 0100 0011 calls bytesToValue at each offset in the TD with the given bit length and the datatype.

Alternatively, if there are issues with supporting https://www.w3.org/TR/wot-thing-description11/#dfn-object that I did not see, maybe bit-offset and -length could be introduced anyway? With these parameters, I could model the above as separate properties.

[danielpeintner]

I am fine if we (you) want to implement this. Anyhow, I think there will be more to look at in the end

• what to do if one terms like offset is missing. I assume failing?
• what about nested definitions
  • shall we fail or support
  • will there be any padding
  • ...

@relu91 @JKRhb any opinions

[JKRhb]

In general, I think the idea makes a lot of sense for resource-constrained environments! I am wondering, though, if there are any pre-existing standards or serialization formats to build upon? (Because I haven't found any so far that go in the same direction).

Two more questions:

• Should this maybe become its own experimental Content-Type (since I think that application/octet-stream is meant to be opaque data)?
• Furthermore, I suppose the additional terms (i.e., offset, bitLength, etc.) should be prefixed (e.g., foo:offset) and imported via a @context extension (that could be a placeholder for now), right?

[derwehr]

In general, I think the idea makes a lot of sense for resource-constrained environments! I am wondering, though, if there are any pre-existing standards or serialization formats to build upon? (Because I haven't found any so far that go in the same direction).

The Bluetooth standard defines a set of services describing the data sent from devices. Like for example the Heart Rate Service, where bit 0, 1, 2 and 3 are flag bits indicating things like the measurement format and supported features, then theres a heart rate measurement field at a higher offset, which on some devices is 8 and on other devices 16 bits long, and additional information on the other offsets all sent in the same bit-stream.

Two more questions:

• Should this maybe become its own experimental Content-Type (since I think that application/octet-stream is meant to be opaque data)?

Since this codec fails with fully opaque data (no type defined), and it already provides a set of parameters, like byteorder, charset and signed, I assumed extending this codec would make sense. But I'd be happy to start on a new codec, too.

• Furthermore, I suppose the additional terms (i.e., offset, bitLength, etc.) should be prefixed (e.g., foo:offset) and imported via a @context extension (that could be a placeholder for now), right?

I guess they should be

[relu91]

@danielpeintner @derwehr do we consider this issue fixed by #1125 ?

[derwehr]

@danielpeintner @derwehr do we consider this issue fixed by #1125 ?

I do :)