update-bcd: various flaws with the design

[foolip]

At a high level, update-bcd takes a one or more test reports as input, and modifies a copy of BCD in place to match the test results.

The test results are potentially conflicting, for multiple reasons:

• There can be multiple tests for a single BCD entry, using different testing methodology (Generate both strict and lax tests, teach update script to interpret the results #523)
• Multiple browser+platform combinations can map to a single BCD browser release (for example, Chrome on macOS and Windows both map to chrome)
• Multiple browser releases can map to a single BCD release (see Revamp getBrowserAndVersion to handle Safari iOS and Samsung Internet #565)

The current design of the update-bcd script is:

• Build a "support matrix" mapping a BCD entry + browser release to a support status (true/false/null) and sometimes a prefix (this also resolves contradictory results for a single BCD entry)
• For every BCD entry (in api/**.json and css/**.json) for which we have data in the support matrix:
  • Infer a BCD-style support statement from the per-version data in the support matrix. Where we have incomplete information, ranged (≤) versions are used.
  • If the inferred support statement isn't simple (a single object) give up and do nothing.
  • If the BCD support statement had only prefix/alternative_name/flags entries, add our inferred statement
  • Otherwise, there was an existing simple statement, which we update using the inferred statement

There are a bunch of problems here:

• We don't correctly handle prefix and alternative_name, or at least not having multiple variants, as the support matrix squashes this information together too early.
• We do nothing if the inferred support statement is too complicated (so for example, we don't update BCD if we've found a add-remove-and-add-again situation, regardless of the existing state of BCD)
• We can never add a support statement if there's already a simple statement, only update it.

Not all of these are very serious, and I don't think any of them cause us to suggest incorrect updates. However, we're leaving some updates on the table by bailing in the situations.

[foolip]

One feature of an improved design that I'd like to see is that we have an explicit test for contradictions between our results for a specific browser release and a BCD-style support statement. Given that, we could apply that test before and after we've updated, and much more easily see what contradictions weren't resolved by the updates. Maybe we'll see that there aren't many and we'd be better off resolving those contradictions manually.

[foolip]

Hmm, I see that actually we can't deal with prefix or alternative_name at all, because we never look at the info object in the results where the prefix would be.

[foolip]

#1904 added another case where we do nothing, a dataIsOlder early exit. That most likely leads to additional cases where we fail to update the data even though there are contradictions.

[jugglinmike]

#1904 added another case where we do nothing, a dataIsOlder early exit. That most likely leads to additional cases where we fail to update the data even though there are contradictions.

@foolip was this addressed by the refectoring in gh-2629?

[foolip]

@jugglinmike indeed it was!