Read-Only Directories

[RoboPhred]

I am building a device hub around TD. Currently, it communicates available TDs by simply sending an array of all known TDs to a predetermined url known by my specific client app, but I am interested in switching to the Discovery spec for greater compatibility.

However, looking through the spec, it seems the create / update / delete operations are not optional. Since my hub represents a source of real devices (zwave in this case), creation or modification of their spec is undesirable. The closest I could get is setting the hub into inclusion mode for a new zwave device, but the device capabilities (and the resulting TD) is not left up to the user.

After reviewing the options in the Discovery spec, I don't seem to have a clear path forward:

• A Directory (as I currently understand it) will not work, as I do not want to provide the capability of adding arbitrary TD or allowing for arbitrary updates on existing things.
• A well known url will not work, as I can only have one, and it can only return a single TD. I have multiple devices I need to return.
• mdns will not work as I cannot broadcast a url path alongside it, and all my things are under the hub's domain / server.

I suppose I could implement the CRUD actions and return FORBIDDEN for all of them regardless of operation, but that seems a bit cheaty...

My use case would be much better served if there was a way to advertise a single thing, and have that thing return additional TDs, just like a directory but without the contractual mutation support.

[relu91]

I would support lessening a little bit the spec to allows TDD to be just a queryable collection of TDs. It makes the mandatory implementation requirements smaller hence simpler to realize. Besides, I know that we are not aiming for cross-protocol compatibility, but a smaller set of functionalities is also easier to port.

An additional point would be that the webthings.io gateway would be easier to align. Maybe @benfrancis can expand on this.

[benfrancis]

Yes we have the same situation with WebThings Gateway.

1. Thing Descriptions are generated by the gateway to represent a real underlying device which may use another protocol like Zigbee or Z-Wave, it's not possible for a Consumer to just add an arbitrary Thing Description to the gateway's directory
2. It doesn't make sense for Consumers to be able to edit a Thing Description because they could for example add a property which doesn't exist in the underlying device, leaving the Web Thing in a broken state

+1 for allowing read-only directories.

[relu91]

Going back to this issue. In this PF (among the other projects), I was thinking to add a TDD interface to node-wot. But again the mandatory updates methods make no sense in that context, cause the WebThings can be only registered by node-wot itself.

Also, why can a TDD only be a collection of TDs? Why is JSONPath mandatory?

In node-wot, we also evaluated the well-known URL introduction mechanism, but it seems to not fit well as well. eclipse-thingweb/node-wot#411 . @danielpeintner I'm tagging you here if you want to add more feedback on this 😃 .

[danielpeintner]

In node-wot, we also evaluated the well-known URL introduction mechanism, but it seems to not fit well as well. eclipse/thingweb.node-wot#411 . @danielpeintner I'm tagging you here if you want to add more feedback on this 😃 .

The problem we are seeing is that /.well-known/wot-thing-description points to the root level of the host, not the thing itself.
Assuming a host disposes of several things it is unclear what /.well-known/wot-thing-description should return.

I am not sure if if miss anything but what makes most sense to me is that the host simply returns a list of links to things.
e.g., http://plugfest.thingweb.io:8083/

Moreover, each thing itself can self-describe itself under well-known,
e.g., plugfest.thingweb.io:8083/smart-coffee-machine/.well-known/wot-thing-description

Note: at the moment the TD is reported under http://plugfest.thingweb.io:8083/smart-coffee-machine directly

[k-toumura]

RFC8615 says:

Well-known URIs are rooted in the top of the path's hierarchy; they are not well-known by definition in other parts of the path. For example, "/.well-known/example" is a well-known URI, whereas"/foo/.well-known/example" is not.

So, we can not use this introduction mechanism to put a TD for each thing on a single host...

Or, we can update current specification, such as:

• Make it possible to place an array or map of TDs on /.well-known/wot-thing-description.
• Some indirection mechanism, e.g. put an array or map of TD URIs on /.well-known/wot-thing-description.
• etc..

[relu91]

Thank you for the feedback! I would say then we have several different options here:

1. Reduce constraints on TDDs: The minimal requirement is to have the things property. Then implementers can freely choose to support JSONPath, single retrieval (retriveTD action), and so on.
2. Reduce constraints on TDDs but keeping all the read-only operations as a minimal requirement. In other words, implementers must always support things property, JSONPath queries, and the retriveTD action.
3. Point read-only implementers to well-known introduction scheme. In this case, we have to fix it as suggested by @k-toumura .

Anyway, let's discuss more on today's F2F call!

[farshidtz]

The introduction mechanism is meant to point to a single thing. If a host has multiple of them and intends to act as a directory, the intro should resolve to the TD of that directory. If not, one way to address that is to have a virtual thing (hosted at the well-known address) with TD.links to actual things.

[mmccool]

• The .well-known problem should be created as a separate issue
• Being able to omit query implementation might also be useful for "small" TDDs, but we should also make a separate issue for that. We might still want some "basic" query mechanism be mandatory (e.g. keyword matching).
• We need to decide whether we want true read-only directories (e.g. pre-populated when the service starts, so it is populated out of band) or if it's just a permission thing (e.g. you need oauth authorization as an admin to register TDs)

Proposed Actions:

1. Make registration optional; a TDD can exist without them (in which case it is populated via some other, hidden mechanism). This may complicate the type/class system (do we want a different type for read-only TDD? Do we use @type for a list of capabilities? The latter would be able to deal with other optional features, e.g. queries). I think if a TDD does not support an interaction then the TD it returns should not include them.
2. Add standardized scopes to the forms. Also note that an implementation can apply different security definitions to different interactions to provide levels of access. In general a TDD will have to access controls for interactions, so a read-only TDD could simply refuse creation of new TD resources (for almost everyone, or everyone).

We might do 1 and 2, or just 2.

Proposed solution:

• Make modification capabilities of TDD optional (create, update, delete)
• Require that unimplemented interactions NOT be listed in the TD returned by TDD
• Recommend/explain that security be used (e.g. oauth) when what is really wanted is to limit a capability to a certain class of users (eg registration to a "discoverer" service)

No consensus yet:

• Define a set of capabilities for every optional group of (modification, query mechanisms...), have those in @type

Please continue in the comments below and state whether or you support the above proposal.

• Note that a "simple" TDD that only provides listing (no creation, no queries) can be used for a host supporting a small number (say, five) predefined Things.

[benfrancis]

Proposed solution:

• Make modification capabilities of TDD optional (create, update, delete)
• Require that unimplemented interactions NOT be listed in the TD returned by TDD
• Recommend/explain that security be used (e.g. oauth) when what is really wanted is to limit a capability to a certain class of users (eg registration to a "discoverer" service)

I agree.

Note that a "simple" TDD that only provides listing (no creation, no queries) can be used for a host supporting a small number (say, five) predefined Things.

+1

[benfrancis]

@farshidtz has confirmed in #267 that the registration API is mandatory for a ThingDirectory, and that the registration API must include create, update, read, delete and list operations.

I think this may make it impossible to make WebThings Gateway compliant with the Directory exploration mechanism part of the specification, because it has a bunch of assumptions which don't apply to gateways.

With WebThings Gateway, most devices are added out of band using some other non-WoT protocol (like Zigbee or Z-Wave). A software adapter generates a read-only Thing Description (and API) for the device which is then included in a listing of Things. Consumers can read the list of Things and interact with them using the API, but they can't arbitrarily create a Thing out of thin air or update an existing Thing, because that's just not how a gateway (at least not this gateway) works. A Consumer can't just add a new property to a Thing if that property doesn't actually exist in the physical device for example.

Unless these assertions change, and read-only directories are allowed, I don't think I'll be able to implement the Directory Service API.

Edit: Note that certain aspects of Thing Descriptions are actually editable (such as the title), but arbitrarily creating and updating Thing Descriptions is simply not feasible.

[benfrancis]

What does it even mean that write operations are mandatory? What should a Consumer do if it comes across a directory which doesn't allow creation of Things, or only allows editing certain members of Thing Descriptions? Would it prevent the Consumer from listing and reading Things? What should a directory do if a Consumer tries to edit a Thing Description it doesn't own? It's surely also true that directories may want different levels of authorisation for particular operations on particular Things?

[egekorkan]

I think that the reading and writing level of permissions will be indicated via the TD of the TDD, i.e. auth1 is needed for all operations and auth2 is needed for write level operations. Still, I think that these kinds of mechanisms are what is going to be annoying because they will force consumers to read the TD of the TDD. In a custom Consumer implementation, I have used all the CRUDL operations on WoT-Hive (from @AndreaCimminoArriaga) and then simply replaced the URL of the TDD with the Logilab one (from @wiresio). Everything worked fine (a nice win for interoperability) since both were using no security. However, once two TDDs do not implement the same list of CRUDL interfaces with the exact same security on all levels, such generic consumers cannot be written. One would need to adapt a TDD Consumer based on the TD of the TDD, which can be really annoying (but doable) for humans and pretty difficult to do it programmatically.

A nervous comment: Maybe profiles for TDDs?

[benfrancis]

@egekorkan wrote:

I think that the reading and writing level of permissions will be indicated via the TD of the TDD, i.e. auth1 is needed for all operations and auth2 is needed for write level operations.

The way that I always assumed permissions would work is that the Thing Description will only show interactions which the user who requested it is permitted to perform. I guess not everyone sees it that way since Thing Descriptions are considered static resources. Perhaps both approaches could be supported, since a Thing Description can either be dynamically generated or statically served like any other JSON resource.

A nervous comment: Maybe profiles for TDDs?

We have discussed the possibility of requiring support for the Directory Service API in the Core Profile, I'm not sure whether that's reasonable or not.

Getting back to the point, is there a technical reason why write operations need to be mandatory?

[farshidtz]

@benfrancis

is there a technical reason why write operations need to be mandatory?

In the gateway use case that you just explained, the directory does support write operations, but via means other than the HTTP registration API.

Note than in the current editor's draft, the HTTP registration API must implement CRUDL. But there is currently no assertion which says that the HTTP registration API is required in a directory, because the TM is yet to define what is required. The TD which got replacement by that TM was including the registration API so we have an understanding that it is mandatory, but it isn't written anywhere right now. I suggest you keep an eye on #86 and possible upcoming PRs and raise you arguments.

---

I think one possible solution for a directory that wants to expose a TD collection produced from static resources is to split listing (things properly) out of the registration API, and call it the Listing API. We could then make that Listing API mandatory and Registration API optional, along with currently optional notification and search APIs. The registration API, as it's name suggests, would cover all of CRUD (with necessary access control) but not Listing.

IMO, the registration API should not have that name and exclude the primitive "registration" functionality. The directory on the other hand, can exclude the registration API and still be a directory for listing and maybe search/filtering.

[benfrancis]

@farshidtz wrote:

I suggest you keep an eye on #86 and possible upcoming PRs and raise you arguments.

OK thanks, I will.

I think one possible solution for a directory that wants to expose a TD collection produced from static resources is to split listing (things properly) out of the registration API, and call it the Listing API. We could then make that Listing API mandatory and Registration API optional, along with currently optional notification and search APIs.

That makes a lot of sense. The registration API and listing API could continue to share the same things property though.

[relu91]

I think one possible solution for a directory that wants to expose a TD collection produced from static resources is to split listing (things properly) out of the registration API, and call it the Listing API. We could then make that Listing API mandatory and Registration API optional, along with currently optional notification and search APIs. The registration API, as it's name suggests, would cover all of CRUD (with necessary access control) but not Listing.

+1 for this direction.

[mmccool]

Actions:

• Make "listing" the only mandatory affordance
• Use "small" static directories to resolve Multiple Responses from Peer-to-Peer Discovery #249 (and not Thing Links)
• Update the Thing Model to indicate this clearly
• Restructure the section structure to move Listing out from under Registration to make it clearer
• To consider: moving "Retrieval" out as well... but still make it optional. For small devices a static list of a fixed number of TDs is fine so only Listing is really necessary
• This also limits the implementation burden of Consumers; in general, they only (need to) use "listing"

[mmccool]

Also:

• Introduction section still needs to be updated, Thing Link PR Permit use of TD Links for Self-Description of Multiple Endpoints #269 has that, but now needs be reworked (assign to @mmccool)
• In general (probably should be a new issue) we need a clearer description of what the consumer has to do to implement discovery.
• If THAT (consumer discovery) process needs normative statements... it will impact our schedule.

[farshidtz]

I've submitted a PR to resolve the concerns raised here. Two use cases are considered:

1. Static directories: these are truly read-only directories that don't allow service-level modifications. The data is bootstrapped from a file or an external resource and the directory serves the identical information during its runtime.
2. Directories with read-only HTTP API: write operations via other protocols.

Returning multiple TDs from introductions mechanism (well-known and dns-sd) is discussed in #249.