Add validate-consistency script to list contradictions between results and BCD

[foolip]

Splitting out #571 (comment) into a new issue after discussion with @vinyldarkscratch:

One feature of an improved design that I'd like to see is that we have an explicit test for contradictions between our results for a specific browser release and a BCD-style support statement. Given that, we could apply that test before and after we've updated, and much more easily see what contradictions weren't resolved by the updates. Maybe we'll see that there aren't many and we'd be better off resolving those contradictions manually.

[jugglinmike]

Hi @foolip. @mzgoddard and I have a few questions about this:

• Do you want this consistency check to be built into the “update-bcd” script? If not, do you want it to invoke “update-bcd”, or would you prefer the operator to invoke it themself?
• Should “validate-consistency” use a single results file from the collector? Or should it operate on all the results files in the “mdn-bcd-results” directory (just like “update-bcd” operates today)?
• We believe the intent of the “validate-consistency” script is to allow maintainers to understand how changes to “update-bcd” impact its effect on BCD when given the same data. Is that right?

[foolip]

Do you want this consistency check to be built into the “update-bcd” script? If not, do you want it to invoke “update-bcd”, or would you prefer the operator to invoke it themself?

I suspect things will turn out best if it's a mode of the update script, or at least if the two scripts use the same shared utility code. Otherwise, if they ever disagree we'll have one script suggesting changes that the other then complains about.

Should “validate-consistency” use a single results file from the collector? Or should it operate on all the results files in the “mdn-bcd-results” directory (just like “update-bcd” operates today)?

A single file is insufficient when there are OS differences, where using the macOS results would undo changes based on Windows results and vice versa. I think a BCD release is the right granularity, and batching beyond that is for ergonomics and convenience.

An important case of convenience is when an issue spans releases. A warning like "BCD says this thing is supported since Chrome 45, but collector results indicate support since Chrome 30" is better than 15 individual warnings from 30-44.

For an integrated approach with a single script, I'm imagining something like this:

• Combine results for each browser release, resolving things like different results on macOS and Windows. The combined result could be null ("don't know") but it's one result. Maybe the strategy should be configurable.
• For each per-browser-release result, look it up in BCD and see if there's a contradiction. Collect all confirmed and contradicted data points.
• For each BCD entry with contradictions, determine the simplest possible edit that would resolve the contradiction without introducing new ones. I'm not 100% sure what the best strategy is, but it comes down to adding/merging/splitting ranges.
• For any contradiction we don't know how to resolve, log a warning. Depending on the above, perhaps we can actually make all edits, but if not we should tell a human to intervene.

We believe the intent of the “validate-consistency” script is to allow maintainers to understand how changes to “update-bcd” impact its effect on BCD when given the same data. Is that right?

Not quite actually. I view it more as a way to form a burndown list of BCD errors, not limited to ones that we know how to resolve automatically. I hope my integrated approach above makes the intent more clear.

[foolip]

In the approach I've sketched, I was particular about when to transform a set of results for individual releases into a support statement for ranges of releases.

When to make this transformation will affect the structure of the code, but the most important point is actually not when but whether it's a lossless transformation.

As long as it represents exactly the same thing, working with ranges would be fine.

What's important to avoid is filling in gaps (missing or null results) in the results when making the transformation and then warning about contractions in that gap, where we actually don't have any data.

I hope this might clarify the reason for suggesting per-release validation, but anything that is equivalent would be fine too.