Web Share Target API

[mgiuca]

Hello TAG!

I'm requesting a TAG review of:

• Name: Web Share Target
• Specification URL: https://wicg.github.io/web-share-target/
• Explainer, Requirements Doc, or Example code: https://github.com/WICG/web-share-target/blob/master/docs/explainer.md (please ignore examples; there are updated examples in the spec)
• Tests: None
• Primary contacts: @mgiuca, @owencm

Further details (optional):

• I have reviewed the TAG's API Design Principles

You should also know that...

• This is a companion to the Web Share spec that was the subject of this previous TAG review. But the two specs are independent (other than sharing a common IDL data structure).
• This is highly coupled to the Web App Manifest spec, but I think it will remain as a separate spec, at least initially.
• There is a prototype implementation in Chrome on Windows, Linux, Chrome OS and Android.
• There is a demo site/app here.

Questions for the review panel:

• The biggest open issue is this: we should catch errors in the URL template at manifest parse time. This is tricky because it essentially means we need to resolve the URL template against the manifest URL at parse time, before substituting the parameters. This isn't well defined because the URL template is not a URL and so can't have the URL parser applied to it. Not sure how to proceed; my solution was to make a dummy parse by substituting the empty strings in, but it has issues noted in that pull request.
• Related, we have identified a problem if the template includes placeholders in path segments, like "/foo/{text}.", if the text supplied is ".." then it would be a path escape. Not sure how to resolve this; escaping doesn't help because the URL Standard explicitly says that "%2e%2e" also means parent directory. The easiest solution is to simply prevent placeholders from appearing before the '?' (so they have to be in the query or fragment).

We'd prefer the TAG provide feedback as (please select one):

• open issues in our Github repo for each point of feedback
• open a single issue in our Github repo for the entire review
• leave review feedback as a comment in this issue and @-notify @mgiuca

Thanks and feel free to post any follow-up discussion questions here.

[owencm]

FYI our intent to ship is only currently blocked on TAG review and we'd like to not bypass this step, so would really appreciate a review ASAP.

[dbaron]

So a few thoughts so far (in somewhat random order):

• I'm slightly disturbed that WebAppManifest uses underscores in the names of the members of the dictionary... but this is at least consistent with that, which is better than being inconsistent. (I mean... I actually like underscores in general, but they haven't been popular in web APIs elsewhere.)
• I've been wondering for a while about having something MIME-type specific, for example, to address things like the use case of calendar apps that want to receive text/calendar files (even those not coming from the Web Share API). See, for example, a recent discussion on mozilla.dev.platform about removal of navigator.registerContentHandler. That's perhaps broader than a general share/share target API, but I've thought it was something useful to solve. (Consider, for example, pages from those for WebEx conference calls to my dentist that offer ICS files (text/calendar) that I'd love to add to my google calendar from my browser.) It seems at least worth thinking about whether this API could be extended to handle that case, even if it doesn't happen right now.
• a dummy parse using empty values (or nonempty alphabetic values) seems reasonable to me; I can't think of anything better offhand, though maybe @annevk has ideas
• sites supporting this presumably need to be pretty careful even if the arbitrary data ends up in the query string. Limiting to being in the query string or the hash seems less than ideal, but allowing things like .. also seems less than great. (What are the rules on interpreting escaped /?) Imposing a particular URL structure seems less than ideal -- but on the other hand, you could think of it like form submission, which is where I suppose query strings came from in the first place. I suppose I can't make a particularly strong argument either way, but maybe others have thoughts.
• the spec probably should describe the escaping of data that implementations do when substituting into the placeholders in the url_template; it doesn't currently appear to describe this.

[dbaron]

Maybe a bigger issue here is thinking about what the path forward for better integration with both newer (mobile-focused) operating systems and older desktop-focused operating systems is. On desktop operating systems, there's often a mechanism for registration and handling of file MIME types, but registerContentHandler never took off. (On the other hand, registerProtocolHandler did, to match the parallel systems for protocol handling, which have sometimes been abused substantially to do things that aren't really protocols.) Is it the right mechanism, or as Anne suggests above, not? (Maybe it's right for native→web but doesn't work well for web→web?)

[dbaron]

Oh, and a final comment related to registries: it seems like there ought to be a master list somewhere of the extensions to WebAppManifest (and, for that matter, other uses of WebIDL partial) so that we don't end up with conflicting ones. (That's particularly interesting in this case given our discussion about preserving the ability for the origin policy and the web app manifest being the same file.) That's sort of the risk of partial... but eventually shouldn't these extensions get rolled back into the original spec so people can keep track of them?

[plinss]

There's also already a standard syntax for URL templates: https://tools.ietf.org/html/rfc6570 (with several npm module implementations) suggest we re-use that rather than reinventing the wheel there.

[mgiuca]

Hi David,

Thanks for the write-up. Sorry it's been awhile. I have been dealing with the URL escaping issue which is blocking the WST spec.

• I'm slightly disturbed that WebAppManifest uses underscores in the names of the members of the dictionary... but this is at least consistent with that, which is better than being inconsistent. (I mean... I actually like underscores in general, but they haven't been popular in web APIs elsewhere.)

I think this is out of scope here (as you say, WST is consistent with the rest of the Manifest API). FWIW, this has caused some trouble already (we've talked about APIs that share the IDL dictionaries in the Manifest spec, and the issue came up that we would need to modify the IDL spec to automatically convert between underscore and camel-case because underscores would be unacceptable in JavaScript APIs). Well that never eventuated, but it is annoying. I think it's too late to not add underscores to Manifest members.

• I've been wondering for a while about having something MIME-type specific, for example, to address things like the use case of calendar apps that want to receive text/calendar files (even those not coming from the Web Share API). See, for example, a recent discussion on mozilla.dev.platform about removal of navigator.registerContentHandler. That's perhaps broader than a general share/share target API, but I've thought it was something useful to solve. (Consider, for example, pages from those for WebEx conference calls to my dentist that offer ICS files (text/calendar) that I'd love to add to my google calendar from my browser.) It seems at least worth thinking about whether this API could be extended to handle that case, even if it doesn't happen right now.

Yes. I don't want this right now, but we need to make sure it can be extended into this. There's a proposal for Web Share to be able to share blobs. This has been a big feature request for WST also (mostly, receiving images). I think they would come in through the same mechanism; a Blob would be created in the document context containing the file, and then a new field would be added to the ShareData (and thus, come in as a URL parameter) referencing the Blob URL.

We can also filter by MIME type by adding new parameters to the share_target dictionary (which currently has just one field; it’s designed to be extended).

I think the existing WST proposal can be extended into this later.

• a dummy parse using empty values (or nonempty alphabetic values) seems reasonable to me; I can't think of anything better offhand, though maybe @annevk has ideas

So @annevk and I have been talking on w3c/web-share-target#31 about parsing the template. It looks like we want to pre-parse the template at manifest parse time (resolving the relative URL before substituting template parameters), and that would address the above issue. The downside of this is it appears we need to change the syntax from "{text}" to "%(text)". I'm hesitant about this because of @plinss comment (I'll address below).

• sites supporting this presumably need to be pretty careful even if the arbitrary data ends up in the query string. Limiting to being in the query string or the hash seems less than ideal, but allowing things like .. also seems less than great. (What are the rules on interpreting escaped /?) Imposing a particular URL structure seems less than ideal -- but on the other hand, you could think of it like form submission, which is where I suppose query strings came from in the first place. I suppose I can't make a particularly strong argument either way, but maybe others have thoughts.

The path is ... unfortunate, but it looks like we'll have to limit this to query and fragment only.

• The biggest concern is "." and ".." path segments, which would, at best, be lost or mis-interpreted, and at worst, represent a path escape.
• The second concern is an escaped '/'. We would automatically be escaping the '/' character as "%2F" (not possible for the sharer to inject a raw "/" into the path). So it depends on the URL processing. If a URL processor "properly" processes the path; splitting on raw '/'s and then decoding, then raw '/' characters in the share data will be correctly carried through to the target (not mis-interpreted as a path separator). But if the URL processor decodes the path before splitting on '/' (e.g., to look up in a real file system), then slashes in the share data will be interpreted as path separators. Basically, it depends on the share target implementation.

Processing a query string is pretty fool-proof. We will automatically escape '=' and '&' and any other reserved characters, so as long as the receiver processes the query by: 1. Splitting on '&', 2. Splitting those parameters on '=' to a key,value pair, 3. Decoding percent-escapes... then any arbitrary string can be handled correctly. As you say, this is the same as a form submission, so all web sites need to be able to handle this.

• the spec probably should describe the escaping of data that implementations do when substituting into the placeholders in the url_template; it doesn't currently appear to describe this.

It does: see Step 1.7 of the replace placeholders algorithm: "Replace each character of value, c, with the result of UTF-8 percent encoding c using the userinfo percent-encode set."

However, I've recently discovered that this is wrong. The userinfo percent-encode set doesn't encode nearly enough characters (notably '&' and '+'). I've been looking into exactly how to define this properly for the past week in whatwg/url#369. It's complicated, but I basically want to define it as "the complement of the unreserved set from RFC 2896". (That is, the same algorithm as encodeURIComponent.) We're just figuring out how to write that out in web spec language (since we use URL Standard, not RFCs).

Maybe a bigger issue here is thinking about what the path forward for better integration with both newer (mobile-focused) operating systems and older desktop-focused operating systems is. On desktop operating systems, there's often a mechanism for registration and handling of file MIME types, but registerContentHandler never took off. (On the other hand, registerProtocolHandler did, to match the parallel systems for protocol handling, which have sometimes been abused substantially to do things that aren't really protocols.) Is it the right mechanism, or as Anne suggests above, not? (Maybe it's right for native→web but doesn't work well for web→web?)

It’s harder on a system without a native intent system. There are a couple of answers (unfortunately some of these overlap with one another):

• Some desktop platforms do have a native share system. Both macOS and Windows do, to a certain degree. A user agent could potentially hook into those native share systems.
• Opening files (MIME types) doesn’t really make sense for the current standard. As described above, we could extend the share_target field, allowing a MIME type filter and file uploads. A UA could automatically register a share target accepting a MIME type as an operating system file handler. This would allow local files to be “opened” in the web app, but not edited and saved back to the disk. I’d love to tackle that at some point (perhaps as an extension to WST?) FYI this was the goal of the original research project that led to Web Share, Ballista. We’ve deliberately reduced the scope of that project to one-way sharing.
• Failing that, web to web sharing should work on any browser on any platform (with navigator.share). If we extend these specs with blob / MIME type sharing, you should still be able to do that web to web.

Oh, and a final comment related to registries: it seems like there ought to be a master list somewhere of the extensions to WebAppManifest (and, for that matter, other uses of WebIDL partial) so that we don't end up with conflicting ones. (That's particularly interesting in this case given our discussion about preserving the ability for the origin policy and the web app manifest being the same file.) That's sort of the risk of partial... but eventually shouldn't these extensions get rolled back into the original spec so people can keep track of them?

I think we can avoid this by simply standardizing this into the Web App Manifest spec. At the moment, it makes sense for it to be its own separate spec, but if we do put this on a standards track, I would expect it to be merged into that spec, so we don’t need to keep track of a list of extensions.

@plinss:
There's also already a standard syntax for URL templates: https://tools.ietf.org/html/rfc6570 (with several npm module implementations) suggest we re-use that rather than reinventing the wheel there.

Yikes! I wasn’t aware of this. It certainly looks relevant (just “Level 1” specifically). And coincidentally, it’s the exact syntax we settled on before the proposed syntax change in w3c/web-share-target#31. There’s just one problem: we reference the URL Standard, not RFC 3986, for our definition of a URL and URL processing. (This seems to be true of all new web standards.) I don’t see how to reconcile RFC 6570 with URL Standard.

Aside from URL Standard compatibility, there are a few other problems with this:

• The same problem we have with Should fail checks at manifest parse time, not launch time w3c/web-share-target#25 --- that we can’t resolve the URL template against the Manifest URL before substituting parameters. RFC 6570 doesn’t define an algorithm for resolving a URL template. In fact, it explicitly states: “A [relative] template is expanded before the resulting reference is resolved from relative to absolute form.” We want to be able to resolve the template at manifest parse time, before we have values for expansion.
• There doesn’t appear to be a way to limit expressions to just the query and fragment sections in this template syntax. For reasons discussed in Path escape if template parameter in path segment w3c/web-share-target#30, we want to limit these from appearing in the path.
• We only want “Level 1” of this standard. Obviously an implementation could just implement Level 1 without the rest, but it’s a little strange depending on such a complex spec when we only need a small sliver of its functionality.

Still, this gives me renewed interest in keeping the “{text}” syntax instead of changing to ”%(text)”, for compatibility with RFC 6570. I will discuss on w3c/web-share-target#31.

[slightlyoff]

Discussing today with @plinss at the W3C TAG meeting in London.

One discussion point regarding RFC 6570 and the timing of resolution, it seems that perhaps one way out of this pickle would be to drop the URL-only version and double-down on the share event in the Service Worker. In that solution, it's possible to pack whatever arguments and payload you like as part of the body of a Request that you can pass along with the event. That model naturally allows you to do the URL validation at registration/install time and not runtime.

Is there a compelling reason to continue down the URL-only path, particularly given that many (I hope most!) Share Target implementations will happen via PWA installability (which already requires a SW)?

/cc @mgiuca

[mgiuca]

Yes, if we went with the Service Worker approach, we side-step the entire issue of URL template syntax, since it's coming in through a JavaScript object (essentially it would be a ShareData object from Web Share). Initially, we were considering these two approaches in parallel, but we stopped considering the Service Worker approach a long time ago, for a number of fairly compelling reasons:

1. The URL template approach does not require a Service Worker at all. While it's a nice carrot to convince people to make a SW, it isn't really necessary, and this just slows adoption of WST.
2. The URL template approach can be implemented into existing share services with a single line in the manifest, and no code changes. For example, we just emailed Twitter a single line to add to the manifest and the next day, it was there and Twitter was a working Web Share Target (because they already had an endpoint URL that receives a share in the query parameters). Integrating into a SW is a more significant engineering investment.
3. While the SW allows you to do things like receive a share without opening a new window/tab, this is almost always something you want to do anyway.
4. The URL template approach fits neatly into the Service Worker navigate event proposal which I made last year. Instead of adding a new event to SW for WST, we can do the same thing by catching a launch event with the WST data in a URL.

Basically, simplicity trumps flexibility. I'd rather stick with the URL approach.

Also (and this shouldn't be a particularly strong argument when considering a web standard, but I'll say it), we have working implementations on Chrome for both mobile and desktop, as well as websites like Twitter that have implemented it.

[domenic]

While the SW allows you to do things like receive a share without opening a new window/tab, this is almost always something you want to do anyway.

Services like Pocket seem like a notable counterexample here, based on my experience using Android's share functionality.

Is there a plan for such services in the web share target world?

[mgiuca]

Services like Pocket seem like a notable counterexample here, based on my experience using Android's share functionality.

Yes, the "save for later" style of sharing is the one use case we've identified for receiving a service worker event rather than opening a page. But it is a soft requirement. You certainly could show a tab that says "I've saved this for later [Close]." Or "Do you want to save this for later? [Save] [Cancel]".

Is there a plan for such services in the web share target world?

Yes, the Service Worker navigate event. (Edit: Full site with explainer and demo.)

[plinss]

David and Alex will attempt to meet with mgiuca to discuss further offline

[mgiuca]

Sounds good. Let me know a time and I'll try to make it. 9am to 6pm weekdays Sydney time are best (if you are in the USA, this means afternoon/evening Sunday to Thursday). (Can email my GitHub username at chromium.org.)

[mgiuca]

@ewilligers: I think it's best to not post uncommitted versions of the spec to personal GitHub pages, since the versioning can get confusing (it's unstable since at any time, it reflects an unknown revision of an unknown branch).

The latest pull request is w3c/web-share-target#40, with a preview here including the big red disclaimer that this is a preview. It has gone through several rounds of code review since the document @dbaron is talking about (including addressing several of the concerns you raised). But it doesn't include the files part. Eric, perhaps you could create PRs for the "files" versions of these specs and then link to the Preview page instead of to your GitHub?

Note that we aren't planning to roll file sharing into the first version of Web Share Target. We're just thinking about it so we have a good model to extend into. The initial version of WST will more or less look like the above preview.

It would probably be useful to say what a share consists of.

The new version addresses this.

The definitions of the app manifest structure contain a bunch of normative "MUST" requirements that seem to be normative requirements on developers.

The new version addresses this. Specifically, the "MUST" requirement on the "action" member has been removed, since it was effectively an informative note. (Processing steps elsewhere say exactly what happens in this case.) I agree, that "MUST" should be reserved for conformance requirements on the processor (the user agent), not the content (the document); there isn't anything that content "MUST" do; rather, the standard describes how the processor handles all inputs.

I was curious why the canShare function exists given that what it does seems to be rather trivial -- or if it's intended to be extended to something more interesting in the future, whether it should return a Promise.

This was added in Eric's (not yet committed) file sharing version, as a way of allowing for feature detection of the files feature. See a long discussion on whatwg/webidl#107. Without it, you wouldn't be able to know whether you're going to be able to share a file without actually trying it. This is a very rough sketch at this stage. It probably should return a Promise.

I'm a little curious about the use of FileList given that the type's definition says that it's "at risk". Does that statement of being "at risk" need to be revisited?

This is something we're actively talking about with the Chrome storage team who are trying to build on those file specs, and considering whether to try to resuscitate them or build something new.

In summary, let's not get too caught up on the file sharing details of Eric's private branch. We are not launching those features any time soon (in Chrome) and do not plan to land them in the spec soon either. We just want to make sure there is a reasonable model to expand into file sharing territory later.

We want the TAG review to focus on the title/text/URL share target which is currently in preview but will hopefully soon land in the WICG draft.

[slightlyoff]

This was discussed a bit today and @dbaron will look more closely at the version you just posted in the last comment, @mgiuca. That said, I'm confused about the use of GET vs POST in this draft. Perhaps I'm reading it wrong, but it appears that the only form encoding allowed is application/x-www-form-urlencoded, which precluded sending files. Further, as these request may mutate state at the dispatch site, should they not be sent as POSTs?

[ewilligers]

We are proposing to initially ship only the title/text/URL share target, without files.

Further, as these request may mutate state at the dispatch site, should they not be sent as POSTs?

I hadn't considered this. The proposal was to exclusively support GET requests for the initial version without files, using encoding consistent with <form method="GET">: Mutate action URL.

The intention for POST would be to use multipart/form-data only, like FormData (MDN). Sharing files would only be supported with POST.

[mgiuca]

@slightlyoff This raises a really good point. We've been thinking about POST as being the vehicle through which we deliver form data, so our plan was to only allow GET in the base version, but later add file sharing using POST. What I realised after reading your message is that POST isn't just a way of delivering large binary files; it also has different semantics (state mutating & non-repeatable).

I don't think we should mandate POST or make it the default, since many web share targets will not mutate state without further input from the user (e.g. https://twitter.com/intent/tweet?text=I%20use%20Web%20Share); refreshing these pages is fine and they are designed to be navigated with a GET request. The original point of using this URL-based design is to hook into those existing share mechanisms.

However, I now think we should support POST in the base standard for WST (so sites can reliably assume any user agent that supports WST also supports "method": "POST"). Even if you don't receive files, you can still use "method": "POST" if your site immediately mutates state without asking the user to confirm, and we should encourage sites to use the semantically appropriate method.

[travisleithead]

Note: need to checkout the correct draft: http://wicg.github.io/web-share-target

[dbaron]

OK, I had a look at the correct version this time. I filed three pull requests (see above), and also have a few comments:

It seems like it might be worth a non-normative note explaining future expansion plans for images/files (and their being POST-only) if that part of the spec isn't going to be written soon, just so it doesn't get forgotten and so somebody else could pick it up later.

Two of the comments in #221 (comment) still seem to apply:

• It would probably be useful to say what a share consists of. I understand it to be a sort of arbitrary bundle of title, text, url, and in the future files. Right now that's stated in the example section but the relevant normative part of the spec seems to assume the reader knows what "the title", "the text", etc., are.

and (for both specs):

• I wonder if there should be some way for associating MIME types with the files.

Otherwise things look good to me, at least after one read-through.

[mgiuca]

It seems like it might be worth a non-normative note explaining future expansion plans for images/files (and their being POST-only) if that part of the spec isn't going to be written soon, just so it doesn't get forgotten and so somebody else could pick it up later.

I agree — we could have a section similar to my Extensibility of this API section in Web Share, but I got some negative feedback on this in w3c/web-share#61.

It would probably be useful to say what a share consists of. I understand it to be a sort of arbitrary bundle of title, text, url, and in the future files. Right now that's stated in the example section but the relevant normative part of the spec seems to assume the reader knows what "the title", "the text", etc., are.

Aren't the descriptions in ShareTargetParams and its members enough?

I wonder if there should be some way for associating MIME types with the files.

I think that will be relevant when we introduce file sharing, but not for the initial version of the spec which doesn't include file sharing.

[dbaron]

It would probably be useful to say what a share consists of. I understand it to be a sort of arbitrary bundle of title, text, url, and in the future files. Right now that's stated in the example section but the relevant normative part of the spec seems to assume the reader knows what "the title", "the text", etc., are.

Aren't the descriptions in ShareTargetParams and its members enough?

I think not quite -- since they give me the feeling that I'm missing some description elsewhere. I think it's useful to state that it's basically an arbitrary set of data possibly coming from an OS sharing system or possibly coming from Web Share.

[slightlyoff]

Hey all,

Discussed at today's F2F; we're excited by the direction this spec is taking, so are closing this issue and looking forward to hearing from y'all if/when there are other issues that arise that we can advise on.

As a parting note, it would be great to see expanded examples, particularly ones that demonstrate how to deal with files and the POST mechanism from Service Workers. These don't seem to be in either the spec or explainer yet.

Thanks again for working with us!

[mgiuca]

Thanks @slightlyoff and @dbaron! I'm filing issues about these things you raised:

• Add a non-normative section explaining future plans w3c/web-share-target#55 (future plans)
• Better descriptions of the semantics of each piece of ShareData w3c/web-share-target#56 (describe data fields)
• Give example with POST and state POST semantics w3c/web-share-target#57 (POST example)

[torgo]

Hi @mgiuca just following this up. I noticed this behaviour working in WebAPK based webapps recently. However it doesn't look like this has percolated back into the webapp manifest specification. Is there a plan to bring this back to the webapp manifest spec itself?