Add Remote Playback API to Shepherd

[tidoust]

See request to be able to autolink to Remote Playback API terms in w3c/remote-playback#137

The Remote Playback API uses Respec, and was last published to /TR/ in 2017, so current /TR/ document does not respect the definitions contract, and my understanding is that Shepherd cannot easily extract terms from Editor's Drafts that use Respec, so I don't know whether it's possible to add the spec for now.

[tidoust]

@tabatkins The Remote Playback API has been re-published since this issue was created (and the Editor's Draft on w3c.github.io is now the generated version of the document). Could the spec now be added to Shepherd for xref purpose, please?

Same question for the Presentation API while I'm at it :)

• Remote Playback API: https://www.w3.org/TR/remote-playback/
• Presentation API: https://www.w3.org/TR/presentation-api/

[tidoust]

Gentle ping @tabatkins @plinss. Could the specs be added to Shepherd, please?

[plinss]

Added both.

For future reference, Shepherd has a lot of heuristics to determine anchor types even without the definitions contract attributes (it was running before the contract was created).

Also, it's better for Shepherd if the drafts are generated into an Overview.html file rather than index.html to match /TR (it ensures that any links directly to the index page get reduced to a bare path). (It actually doesn't matter what you call, just be consistent between drafts and /TR.)

[tidoust]

Thanks @plinss.

The definitions contract does not talk about headings and it seems that Shepherd cannot extract headings from specs generated with ReSpec, e.g.: Presentation API, Payment Request. This makes it impossible to use section links such as [[PRESENTATION-API#establishing-a-presentation-connection]].

What is the markup that Shepherd expects for headings?

[plinss]

Shepherd parsed all the headings just fine. I confirmed all the heading archors are in the DB and were properly determined to be section headings. I don't know why they're not listed in the Bikeshed repo, maybe that's just taking time to update.

If you're running Bikeshed locally, try bikeshed --update --anchors --skip-manifest or using the Bikeshed API at: https://api.csswg.org/bikeshed

[plinss]

One thing to note, the anchor 'establishing-a-presentation-connection' isn't on a header element, it's on a section element, so if you're trying to link explicitly to a header, I can see how that might fail. 'x6-5-1-establishing-a-presentation-connection' is the anchor on the h4 element within that section. Shepherd has both anchors in its DB, the former is a 'section', the latter is a 'heading'

[markafoltz]

I updated my local copy of the BS anchor database per the command above and the following links still generate errors:

LINE ~785: Couldn't find section '#creating-a-receiving-browsing-context' in spec 'presentation-api-1':
[[PRESENTATION-API#creating-a-receiving-browsing-context]]

LINE ~787: Couldn't find section '#x6-6-1-creating-a-receiving-browsing-context' in spec 'presentation-api-1':
[[PRESENTATION-API#x6-6-1-creating-a-receiving-browsing-context]]

However the anchor is there in the generated ReSpec:

<div class="header-wrapper"><h4 id="x6-6-1-creating-a-receiving-browsing-context"><bdi class="secno">6.6.1 </bdi>
            Creating a receiving browsing context
          </h4><a class="self-link" href="#creating-a-receiving-browsing-context" aria-label="Permalink for Section 6.6.1"></a></div>

[tidoust]

Bikeshed retrieves the data it needs from bikeshed-data, so as long as the headings do not appear there, they won't show up locally either.

@tabatkins, sections and headings are in Shepherd (see e.g. the data returned by Shepherd for the Presentation API) but Bikeshed does not seem to like what it sees there, resulting in an empty headings file. What's missing?

[markafoltz]

I think this was the job that updated the bikeshed-data repo after Shepherd was updated.

[plinss]

FWIW, --skip-manifest causes bikeshed to go to the original sources rather than use the bikeshed-data repo.

[markafoltz]

--skip-manifest does not make a difference, I get the same error about missing headings when generating the spec.

[tabatkins]

Ahhhh, I see the problem.

So, I've always been confused by Shepherd's data model. ^_^ For reasons that I'm sure made sense at some point, I filter all the "anchors" for just those with type "dfn" or "heading", and then for heading-type ones, check that they have "section":true before adding them to the db. But none of your headings have that, since they're all nested within a <section>; the section shows up in the anchors as a "type":"other", "section":true, and the heading is a child anchor with "type":"heading" but no section key.

I'm pretty sure when I added this, I was getting too many things classified as "heading" for some reason, thus the extra check. It's just failing for this style of markup.

[tidoust]

That's what I'm proposing to change in #2358, preferring "section": true over "type": "heading" to get the headings because that gives better IDs in specs generated with ReSpec. I didn't spot extra headings being reported on the tests I made but I only tested that with a handful of specs.

[markafoltz]

It looks like your PR got merged, thank you so much for digging into this and finding the root cause and fix.

I'll do a local update of my bikeshed data and see if that resolves the linking issues I'm seeing.

[markafoltz]

After a full bikeshed update, still no luck downloading the headings; there's an empty presentation-api-1.json in my bikeshed installation.

[tabatkins]

...huh, that's extremely weird. https://github.com/tabatkins/bikeshed-data/blob/main/data/headings/headings-remote-playback-1.json is definitely empty, but when I run bikeshed update --skip-manifest --anchors locally that file fills up with all the heading info.

I'm gonna have to dig into wtf is going on, this is weird. It shouldn't be possible to be different like this, since the update process for that repo is basically just "run the skip-manifest update"!

[tabatkins]

Wait never mind, duh, bikeshed-data runs with the pip version of Bikeshed, which hasn't been updated yet. One sec.

[tabatkins]

Okay, pip updated to latest code, confirmed that bikeshed-data is installing it properly, but the Shepherd server is 500'ing now so no updates can run. ^_^ Well, this should fix itself as soon as the server unwedges.

[tabatkins]

Update: after the Shepherd server came back up on Friday, its data was missing a lot of specs, and they don't seem to have come back. I've temporarily halted bikeshed-data at a known-reasonable state; the Shepherd server is being migrated this afternoon, so I'll wait until tomorrow to try it again and see if things look reasonable.

[plinss]

To be clear, only the db server is being migrated today. I don’t expect any behavioral changes, just stability and performance.

I did a force reparse on all the specs yesterday and don’t see any reason data should be missing, but I’ll take a close look in a few.

[plinss]

I confirmed all the anchor data for remote-playback-1 is in the spec DB and is being exposed via the API, see: https://test.csswg.org/shepherd/api/spec/?spec=remote-playback-1&sections=1&draft=1&anchors=1

What specs do you see missing data for? The DB looks fine afaict.

[tabatkins]

When I tried a manual update earlier today (before the migration), a bunch of HTML definitions were still missing. (I specifically noticed a lot of element definitions, since I reference a bunch in the Bikeshed docs, which I was working on today.)

Earlier a bunch of CSS definitions were missing too, but I didn't verify they were still missing.

[tidoust]

Ahem, I'm afraid my fix for headings in #2358 unfortunately broke the import of cross-references because I actually restricted import to dfn types + sections, excluding all non dfn terms (IDL, css properties, etc.). On the plus side, it did fix headings... Really sorry about that! I'm preparing a fix. It may be wise to rollback the change in the meantime.

[tidoust]

Fix proposed in #2364.