Regenerate tables for Unicode 13.0.0

[cuviper]

No description provided.

[rust-highfive]

r? @kennytm

(rust_highfive has picked a reviewer for you, use r? to override)

[Mark-Simulacrum]

Would you mind providing diffs of sizes against 12.1 (might require code modifications, I forget if the various scripts support it)?

Mostly just want to make sure none of the algorithms we have are breaking horribly by accident -- though it doesn't look like it based on the diffs.

[cuviper]

OK, I hacked the URL_PREFIX to run with 12.1.0 for comparison.

12.1.0:

Alphabetic     : 2982 bytes, 127257 codepoints
Case_Ignorable : 2112 bytes, 2397 codepoints
Cased          : 934 bytes, 4280 codepoints
Cc             : 43 bytes, 66 codepoints
Grapheme_Extend: 1734 bytes, 1966 codepoints
Lowercase      : 985 bytes, 2341 codepoints
N              : 1239 bytes, 1755 codepoints
Uppercase      : 934 bytes, 1909 codepoints
White_Space    : 140 bytes, 25 codepoints
Total table sizes: 11103 bytes

13.0.0:

Alphabetic     : 3055 bytes, 132876 codepoints
Case_Ignorable : 2136 bytes, 2414 codepoints
Cased          : 934 bytes, 4287 codepoints
Cc             : 43 bytes, 66 codepoints
Grapheme_Extend: 1774 bytes, 1980 codepoints
Lowercase      : 985 bytes, 2345 codepoints
N              : 1266 bytes, 1782 codepoints
Uppercase      : 934 bytes, 1912 codepoints
White_Space    : 140 bytes, 25 codepoints
Total table sizes: 11267 bytes

All but Cc and White_Space increased in codepoints, but Cased, LowerCase, and UpperCase still kept the same number of bytes.

[Mark-Simulacrum]

Looks great, thanks! Let's r? @SimonSapin for approval on the version bump (code changes look correct though I didn't run the script locally).

[SimonSapin]

r? @Manishearth

[Manishearth]

Version bump seems fine to me. I haven't run it locally either though

[Mark-Simulacrum]

I think we're running into "too many approvals" at this rate so I'm just going to go ahead and approve. I don't think we need libs FCP.

@bors r+

@rustbot label add relnotes

(let's see if I, the author, managed to remember syntax...)

[bors]

📌 Commit 543832b has been approved by Mark-Simulacrum

[RalfJung]

#69996 (comment) failed in "xsv" cargotest complaining that strings involving unicode differ, I am going to guess this PR is at fault. @bors r-

2020-03-14T13:27:01.2477617Z thread 'test_cat::prop_cat_cols' panicked at 'assertion failed: `(left == right)`
2020-03-14T13:27:01.2490914Z   left: `[["\u{ffffd}2誢", "\u{e}", "-9", "\u{6dd}\u{90}隙", "O?", "\u{202b}5K⁑"], ["-", "", "", " -鈛", "", "\u{16}Cⶽ"], ["\u{fff6}", "\u{e0001}\u{a96d0}", "\u{14}叐;", "\u{14}¤3\u{10fffe}", "\n\u{206f}", "\u{83}"], ["ꑚo", "<䈰=\u{a33e7}", "\u{8dafc}\u{192d}+", "\u{fff8}\u{8a}/s", "", ""], ["@ \u{dcf}", "(\u{f55a7}", "Iv\"", "0 e", "㦬\u{e0001}@", "‟\u{f133}"], ["_", "", "M\u{86}", "", "n\u{18}4", "\u{15},"], ["", "L1\u{603}{", "ꓝ\u{8a}\r\u{49b91}", "\'[M<", "쳌;", "\u{a4cb}#\u{fff3}"], ["\u{85}  ", "\u{ecac}\u{9d}\u{6}웟", "", "\u{95}-:", "\u{88},", "\u{202f}"], ["$)\'", "B #3", "*\u{8}", "", "\'§", "{M"], ["\u{f}\u{e}\u{16}!", "\u{df0}", "`t}", "-]Z", "\u{1b}\u{1a}", "\u{df99d}4X"], ["", "\u{e4bf}", "9", "ㄸH:", "떟}\u{2028}!", "¨\u{6b5b6}\u{1b}"], ["\u{11} ", "®x", "B\u{e1b0}\u{6dc78}\u{88}", "", " ‑`", "`"], ["!㞾", "", "", "fI&\u{4}", "", "\u{e1c0f}}\u{9b}7"], ["", "\"", "\u{202a}—`^", "搄\u{2f08e}", "\u{15}Y~\u{13}", ""], ["\u{87}\u{85}!", "븞\u{90}\u{61c}", "y 6", "\u{fff0}UZU", "c", "\u{84}"], ["\u{16}\u{d3564}\u{1d}~", "\u{82}&\u{100a24}k", "[", "6", "", "C얣\u{fff2}"], ["\u{fff4}", "\u{3}", "1\u{ae8b9}c&", "u8", "jD", ">"], ["§\t", "\u{110bd}!s`", "^\u{8c}\u{14e2b}⁞", "\u{3}<P\u{f0000}", "", "¯¯쏍!"], ["釂¥0,", "-‧", "뎮", "‚湛w¤", "腽\u{f22d5}C\u{b}", "\u{9c}\u{87}"], ["3\u{18}𐨓9", "~-n!", "@¥_", "6", "«", "鐛<"], ["\u{faf5}\r)\u{17}", "¡,I\u{5}", "?:\u{10}b", "pS\u{c7fe4}‹", "!-?y", "\u{e0001}){{"], ["_\u{b9191}\u{2063}ꌨ", ";붌:-", "g", "", "휀_\u{ffffd}", "ꙕ"], ["\\\u{6b40b}", "\u{8}|_", "J", "\u{0}2矖", "\u{9f}\"", "$ꈂ"], ["£+", "t‾,\u{2063}", "`\u{ad}`", "\u{970aa}", "", ""], ["[", "", "@<\u{eb35}", "勶d\u{92}\u{76f51}", "\u{fff4}N깭\\", "\'"], ["\u{8}&2鄧", "", "\u{99}Y⁁", "", "", ""], ["\u{12}_ᢪ ", ";\n\u{96}꺅", "<", "\u{42609}\u{202d}", "\u{92}\u{12}", "\"h\u{d7701}"], ["", "H\u{1680}\u{5}(", "\u{2002}k$", "\u{d3c63}:b", "", "\u{ffffd}\t"], ["鵀53\u{14}", "<", "", "#¦£\\", "", ""], ["", "\u{fff8}", "&1�", "6\"섲\u{e0020}", "", ""], ["| ", "%", "7\u{7}", "", "\u{10fffd}", "f8;"], ["\u{e6406}<⁈", " \u{5dca1}}", "|", "", ")", ""], ["礔豿7", "*#", "&⒕[1", "4E!\u{85}", "22", ",\u{fff6}"], ["", "\u{87}[\'@", "&|―\u{ed51}", "o", "-", "\\"], ["C.", "", "£‥ª0", "];\u{16}", "\u{f0000}", "\\\u{12}U嗨"], ["}1!", "¯\u{10}\"\u{16}", "⁍", "+i", "\u{9f}", "⁈¦\n"], ["C\u{d55d1}W", "\u{97}$\u{18}", "", "\u{97}㜪", "䍧‖4", ""], ["\u{e0020}0[{", "H¬", "E/", "\u{14}x¨|", "W", "\u{7}\u{f}"], ["", "", "", "숆&㵲\u{ffffe}", "", "X=§⁂"], ["", "孄\u{5}\u{87}", "", "瘒\u{95}&\u{dade3}", "T", "\u{fb10}›9"], [",\u{1d}", "m", "", "\u{953}\u{0}$", "0", ""], ["⇨,", "\u{9c}\u{2028}\u{d5a1e} ", "‷", "", "6*l嗃", "\u{9b}\u{17}\u{4}ꋯ"], ["]-", "5\u{603}", "\u{602}\u{3}", "", "u䯝4`", ""], ["", "\u{5586a}(\u{c}\u{12}", "\u{cf3ca}", "\u{1f}{", ")_9C", "4\u{16}\u{9b}\u{9e}"], ["", "`.", "", "~", "*\u{14}\u{83}]", ""], ["\u{b3262}\u{88}⬳", "/鿵\\⁍", "", "4‴䦋", "\u{87}&�\u{81}", ""], ["[㐉⁅;", "皟\u{fffa}", "", "懍}h", "膳\u{100000}0", ""], ["\u{e000}£~", "‐`% ", "캯\u{6f85d}s", "ꍜ⁅^", "@\u{e846}\u{5}", "\u{1d173}𢕄‿\""], ["`", "\u{f14d1}`", "\u{9c}0\u{206f}\u{81}", ":\u{16}\u{fffb}", "", "朅\u{0}"], ["u", "3p\u{12}", "I(>P", "\u{e056}", "3ᚣ/", ",ªj"], ["\"", "", "a\u{80}", "諐`6¡", "⁘~", "J‼!\u{9c}"], ["\u{9d}1", "5«", "", "+\u{6}J", "9\u{2}", "=㴼3"], ["4\u{67e86}3", "@\u{f088}- ", "", "\'`-", "\u{1a}", "V"], ["䧂⁔", "]9\'㝴", "\'£\u{84b19}", "", "d\u{f}?", "\u{8b}\u{3ab24}"]]`,
2020-03-14T13:27:01.2512712Z  right: `[["\u{ffffd}2誢", "\u{e}", "\u{feff}-9", "\u{6dd}\u{90}隙", "O?", "\u{202b}5K⁑"], ["-", "", "", " -鈛", "", "\u{16}Cⶽ"], ["\u{fff6}", "\u{e0001}\u{a96d0}", "\u{14}叐;", "\u{14}¤3\u{10fffe}", "\n\u{206f}", "\u{83}"], ["ꑚo", "<䈰=\u{a33e7}", "\u{8dafc}\u{192d}+", "\u{fff8}\u{8a}/s", "", ""], ["@ \u{dcf}", "(\u{f55a7}", "Iv\"", "0 e", "㦬\u{e0001}@", "‟\u{f133}"], ["_", "", "M\u{86}", "", "n\u{18}4", "\u{15},"], ["", "L1\u{603}{", "ꓝ\u{8a}\r\u{49b91}", "\'[M<", "쳌;", "\u{a4cb}#\u{fff3}"], ["\u{85}  ", "\u{ecac}\u{9d}\u{6}웟", "", "\u{95}-:", "\u{88},", "\u{202f}"], ["$)\'", "B #3", "*\u{8}", "", "\'§", "{M"], ["\u{f}\u{e}\u{16}!", "\u{df0}", "`t}", "-]Z", "\u{1b}\u{1a}", "\u{df99d}4X"], ["", "\u{e4bf}", "9", "ㄸH:", "떟}\u{2028}!", "¨\u{6b5b6}\u{1b}"], ["\u{11} ", "®x", "B\u{e1b0}\u{6dc78}\u{88}", "", " ‑`", "`"], ["!㞾", "", "", "fI&\u{4}", "", "\u{e1c0f}}\u{9b}7"], ["", "\"", "\u{202a}—`^", "搄\u{2f08e}", "\u{15}Y~\u{13}", ""], ["\u{87}\u{85}!", "븞\u{90}\u{61c}", "y 6", "\u{fff0}UZU", "c", "\u{84}"], ["\u{16}\u{d3564}\u{1d}~", "\u{82}&\u{100a24}k", "[", "6", "", "C얣\u{fff2}"], ["\u{fff4}", "\u{3}", "1\u{ae8b9}c&", "u8", "jD", ">"], ["§\t", "\u{110bd}!s`", "^\u{8c}\u{14e2b}⁞", "\u{3}<P\u{f0000}", "", "¯¯쏍!"], ["釂¥0,", "-‧", "뎮", "‚湛w¤", "腽\u{f22d5}C\u{b}", "\u{9c}\u{87}"], ["3\u{18}𐨓9", "~-n!", "@¥_", "6", "«", "鐛<"], ["\u{faf5}\r)\u{17}", "¡,I\u{5}", "?:\u{10}b", "pS\u{c7fe4}‹", "!-?y", "\u{e0001}){{"], ["_\u{b9191}\u{2063}ꌨ", ";붌:-", "g", "", "휀_\u{ffffd}", "ꙕ"], ["\\\u{6b40b}", "\u{8}|_", "J", "\u{0}2矖", "\u{9f}\"", "$ꈂ"], ["£+", "t‾,\u{2063}", "`\u{ad}`", "\u{970aa}", "", ""], ["[", "", "@<\u{eb35}", "勶d\u{92}\u{76f51}", "\u{fff4}N깭\\", "\'"], ["\u{8}&2鄧", "", "\u{99}Y⁁", "", "", ""], ["\u{12}_ᢪ ", ";\n\u{96}꺅", "<", "\u{42609}\u{202d}", "\u{92}\u{12}", "\"h\u{d7701}"], ["", "H\u{1680}\u{5}(", "\u{2002}k$", "\u{d3c63}:b", "", "\u{ffffd}\t"], ["鵀53\u{14}", "<", "", "#¦£\\", "", ""], ["", "\u{fff8}", "&1�", "6\"섲\u{e0020}", "", ""], ["| ", "%", "7\u{7}", "", "\u{10fffd}", "f8;"], ["\u{e6406}<⁈", " \u{5dca1}}", "|", "", ")", ""], ["礔豿7", "*#", "&⒕[1", "4E!\u{85}", "22", ",\u{fff6}"], ["", "\u{87}[\'@", "&|―\u{ed51}", "o", "-", "\\"], ["C.", "", "£‥ª0", "];\u{16}", "\u{f0000}", "\\\u{12}U嗨"], ["}1!", "¯\u{10}\"\u{16}", "⁍", "+i", "\u{9f}", "⁈¦\n"], ["C\u{d55d1}W", "\u{97}$\u{18}", "", "\u{97}㜪", "䍧‖4", ""], ["\u{e0020}0[{", "H¬", "E/", "\u{14}x¨|", "W", "\u{7}\u{f}"], ["", "", "", "숆&㵲\u{ffffe}", "", "X=§⁂"], ["", "孄\u{5}\u{87}", "", "瘒\u{95}&\u{dade3}", "T", "\u{fb10}›9"], [",\u{1d}", "m", "", "\u{953}\u{0}$", "0", ""], ["⇨,", "\u{9c}\u{2028}\u{d5a1e} ", "‷", "", "6*l嗃", "\u{9b}\u{17}\u{4}ꋯ"], ["]-", "5\u{603}", "\u{602}\u{3}", "", "u䯝4`", ""], ["", "\u{5586a}(\u{c}\u{12}", "\u{cf3ca}", "\u{1f}{", ")_9C", "4\u{16}\u{9b}\u{9e}"], ["", "`.", "", "~", "*\u{14}\u{83}]", ""], ["\u{b3262}\u{88}⬳", "/鿵\\⁍", "", "4‴䦋", "\u{87}&�\u{81}", ""], ["[㐉⁅;", "皟\u{fffa}", "", "懍}h", "膳\u{100000}0", ""], ["\u{e000}£~", "‐`% ", "캯\u{6f85d}s", "ꍜ⁅^", "@\u{e846}\u{5}", "\u{1d173}𢕄‿\""], ["`", "\u{f14d1}`", "\u{9c}0\u{206f}\u{81}", ":\u{16}\u{fffb}", "", "朅\u{0}"], ["u", "3p\u{12}", "I(>P", "\u{e056}", "3ᚣ/", ",ªj"], ["\"", "", "a\u{80}", "諐`6¡", "⁘~", "J‼!\u{9c}"], ["\u{9d}1", "5«", "", "+\u{6}J", "9\u{2}", "=㴼3"], ["4\u{67e86}3", "@\u{f088}- ", "", "\'`-", "\u{1a}", "V"], ["䧂⁔", "]9\'㝴", "\'£\u{84b19}", "", "d\u{f}?", "\u{8b}\u{3ab24}"]]`', tests/test_cat.rs:86:9

(and more)

[Mark-Simulacrum]

cc @BurntSushi -- I suspect that's just the test needing to be updated to a new unicode version? Not sure though.

[BurntSushi]

xsv shouldn't depend on any specific Unicode version for its tests to pass. I believe that test failure is spurious. Either way, I won't be able to really dig into it for quite some time.

[Mark-Simulacrum]

No worries. @cuviper -- could you try to run the xsv tests locally perhaps? If it is spurious we can likely just retry. I personally suspect it might be a bug in the code generation scripts (e.g., some limit is hit and we overwrite existing data with new data).

[ollie27]

xsv is using random numbers to generate test cases: https://github.com/BurntSushi/xsv/blob/66956b6bfd62d6ac767a6b6499c982eae20a2c9f/tests/tests.rs#L56. This means that xsv needs to be removed from cargotest because otherwise we're just asking for spurious test failures.

My best guess as to what's happened is through random chance a test case with a BOM ('\u{feff}') was generated and that breaks the tests. It looks like this has happened many times before: #45348 (comment) #45380 (comment) #47195 (comment) #49394 (comment) #58224 (comment) #65342 (comment) #67015 (comment) etc...

In other words I don't think this PR had anything to do with the test failure.

[BurntSushi]

Right, as I said, I'm pretty sure it's spurious. And "random numbers" aren't the problem. The test itself is likely broken.

[cuviper]

I ran xsv's testsuite in a loop with stable-x86_64-unknown-linux-gnu, rustc 1.42.0 (b8cedc004 2020-03-09), and on the 13th try got this failure:

thread 'test_frequency::prop_frequency_indexed' panicked at '[quickcheck] TEST FAILED (runtime error). Arguments: (CsvData { data: [[[]], [[]], [[239, 187, 191]], [[]], [[]]] })
Error: "assertion failed: `(left == right)`\n  left: `[(\"(NULL)\", 4)]`,\n right: `[(\"(NULL)\", 3), (\"\\u{feff}\", 1)]`"', <::std::macros::panic macros>:2:4

That's not the same test that #69996 hit, but it does appear there are lingering BOM issues.

[Mark-Simulacrum]

Sounds good, thanks for investigating. I think that means that hopefully a @bors retry will pass.

It sounds like we'll plausibly want to either ignore that test inside the cargotest suite or perhaps drop xsv for now (but I'm not filing an issue since we have some further discussion scheduled on that topic in the infra team, IIRC).

[cuviper]

The queue doesn't show approval, despite the label change, so let's try again...

@bors r=Mark-Simulacrum

[bors]

📌 Commit 543832b has been approved by Mark-Simulacrum