tidb: add description about GB18030 #18662

CbcWestwolf · 2024-09-19T09:44:40Z

What is changed, added or deleted? (Required)

Add description about GB18030

Which TiDB version(s) do your changes apply to? (Required)

Tips for choosing the affected version(s):

By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.

For details, see tips for choosing the affected versions (in Chinese).

What is the related PR or file link(s)?

This PR is translated from:
Other reference link(s): Do NOT merge this PR until charset,collation: implement GB18030 and 2 collations tikv/tikv#17467 and charset,collation: implement GB18030 and 2 collations | tidb-test=pr/2087 tidb#55792 are merged

Do your changes match any of the following descriptions?

Delete files
Change aliases
Need modification after applied to another branch
Might cause conflicts after applied to another branch

character-set-gb18030.md

dm/dm-best-practices.md

ti-chi-bot · 2024-09-23T13:32:45Z

[LGTM Timeline notifier]

Timeline:

2024-09-23 13:32:44.88699319 +0000 UTC m=+1486434.627417141: ☑️ agreed by hfxsd.

Co-authored-by: xixirangrang <hfxsd@hotmail.com>

character-set-gb18030.md

lilin90 · 2024-09-24T10:33:32Z

sql-statements/sql-statement-show-collation.md

 ```

-若开启了新排序规则框架，则在二进制排序规则之外，额外支持 `utf8_general_ci` 和 `utf8mb4_general_ci` 两种大小写和口音不敏感的排序规则：
+若开启了新排序规则框架，则在二进制排序规则之外，额外支持若干种大小写和口音不敏感的排序规则：


这里的“若干”指代能否更具体一些？

Fixed by 63942e5

Co-authored-by: Lilian Lee <lilin@pingcap.com>

shawn0915 · 2024-10-30T01:55:12Z

@CbcWestwolf 请问这里的GB18030是指 GB 18030-2022 么？

CbcWestwolf · 2024-10-30T09:20:33Z

请问这里的GB18030是指 GB 18030-2022 么？

是的 @shawn0915

shawn0915 · 2024-11-20T09:45:31Z

请问这里的GB18030是指 GB 18030-2022 么？

是的 @shawn0915

这是之前企业版支持的特性？要不要明确说明一下是 GB 18030-2022 毕竟跟老国标是有区别的。

lilin90 · 2024-12-30T09:39:43Z

@CbcWestwolf Would you please update the corresponding descriptions in this PR from v8.4 to v9.0, and add description about #18662 (comment)?

CbcWestwolf · 2024-12-31T08:20:39Z

@lilin90 Sure, let me update it after merging the code.

ti-chi-bot · 2025-01-06T03:28:16Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from hfxsd, ensuring that each of them provides their approval before proceeding. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hfxsd · 2025-01-10T01:40:59Z

character-set-and-collation.md


-使用 `utf8_general_ci`、`utf8mb4_general_ci`、`utf8_unicode_ci`、`utf8mb4_unicode_ci`、`utf8mb4_0900_ai_ci` 和 `gbk_chinese_ci` 中任一种时，字符串之间的比较是大小写不敏感 (case-insensitive) 和口音不敏感 (accent-insensitive) 的。同时，TiDB 还修正了排序规则的 `PADDING` 行为：
+使用 `utf8_general_ci`、`utf8mb4_general_ci`、`utf8_unicode_ci`、`utf8mb4_unicode_ci`、`utf8mb4_0900_ai_ci`、`gbk_chinese_ci` 和 `gb18030_chinese_ci` 中任一种时，字符串之间的比较是大小写不敏感 (case-insensitive) 和口音不敏感 (accent-insensitive) 的。同时，TiDB 还修正了排序规则的 `PADDING` 行为：


L580 优先级排序需要加上 gbk_chinese_ci 和 gb18030_chinese_ci 吗？

character-set-gb18030.md

hfxsd · 2025-01-10T01:52:26Z

character-set-gb18030.md

+* 在系统变量 [`character_set_client`](/system-variables.md#character_set_client) 和 [`character_set_connection`](/system-variables.md#character_set_connection) 没有同时设置为 `gb18030` 的情况下，TiDB 处理非法字符的方式与 MySQL 一致。
+* 在 `character_set_client` 和 `character_set_connection` 同时设置为 `gb18030` 的情况下，TiDB 处理非法字符的方式与 MySQL 有如下区别：
+
+    - MySQL 处理非法 GB18030 字符集时，对读和写操作的处理方式不同。


需要介绍分别如何处理的吗？或者给个 MySQL 相关内容的链接。

sql-statements/sql-statement-import-into.md

Co-authored-by: xixirangrang <hfxsd@hotmail.com>

CbcWestwolf · 2025-01-13T03:39:55Z

character-set-gb18030.md

+
+TiDB 和 MySQL 在 `gb18030` 字符集的默认排序规则上存在差异，具体如下：
+
+- TiDB 的 `gb18030` 字符集的默认排序规则为 `gb18030_bin`。


TODO: 更改默认值描述

Frank945946 · 2025-01-13T03:25:07Z

character-set-gb18030.md

+| Collation   | Charset | Id  | Default | Compiled | Sortlen | Pad_attribute |
+-------------+---------+-----+---------+----------+---------+---------------+
+| gb18030_bin | gb18030 | 249 | Yes     | Yes      |       1 | PAD SPACE     |
+-------------+---------+-----+---------+----------+---------+---------------+


@CbcWestwolf 这里的结果是不是漏了 ”gb18030_chinese_ci“ 这个 collation ？

Frank945946 · 2025-01-13T03:32:59Z

character-set-gb18030.md

+- MySQL 的 `gb18030` 字符集的默认排序规则为 `gb18030_chinese_ci`。
+- TiDB 支持的 `gb18030_bin` 与 MySQL 支持的 `gb18030_bin` 排序规则实现不同，TiDB 通过将 `gb18030` 字符集转换为 `utf8mb4` 然后进行二进制排序。
+
+如果要使 TiDB 兼容 MySQL GB18030 字符集的排序规则，你需要在首次初始化 TiDB 集群时，将 TiDB 配置项 [`new_collations_enabled_on_first_bootstrap`](/tidb-configuration-file.md#new_collations_enabled_on_first_bootstrap) 设置为 `true` 来开启[新的排序规则框架](/character-set-and-collation.md#新框架下的排序规则支持)。开启新的排序规则框架后，查看 GB18030 字符集对应的排序规则，可以看到 TiDB GB18030 默认排序规则已经切换为 `gb18030_chinese_ci`。


@CbcWestwolf @hfxsd 我看默认值就是 ture，对于新集群就不会有问题，是不是只需要强调低版本如果设置为了 false，升级到高版本需要手动设置这个配置项为 ture？

Frank945946 · 2025-01-13T03:35:09Z

character-set-gb18030.md

+- TiDB 的 `gb18030` 字符集的默认排序规则为 `gb18030_bin`。
+- MySQL 的 `gb18030` 字符集的默认排序规则为 `gb18030_chinese_ci`。


@CbcWestwolf 这里为何不与 MySQL 保持一致？如果用户 MySQL 默认字符集是 gb18030 没有显式指定 collation，那迁移到 TiDB 就会变成 gb18030_bin，会不会有问题？能否保持一致的行为，来避免类似的问题？

Frank945946 · 2025-01-13T03:38:13Z

character-set-gbk.md

+-----------+---------+----+---------+----------+---------+---------------+
+| Collation | Charset | Id | Default | Compiled | Sortlen | Pad_attribute |
+-----------+---------+----+---------+----------+---------+---------------+
+| gbk_bin   | gbk     | 87 | Yes     | Yes      |       1 | PAD SPACE     |
+-----------+---------+----+---------+----------+---------+---------------+


这里是不是也漏了返回 gbk_chinese_ci 这个collation 结果？

tidb: add description about GB18030

edf6730

ti-chi-bot bot added missing-translation-status This PR does not have translation status info. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 19, 2024

lilin90 self-assigned this Sep 20, 2024

lilin90 added v8.4 This PR/issue applies to TiDB v8.4. translation/doing This PR’s assignee is translating this PR. labels Sep 20, 2024

ti-chi-bot bot removed the missing-translation-status This PR does not have translation status info. label Sep 20, 2024

lilin90 requested review from lilin90 and hfxsd September 23, 2024 03:52

lilin90 added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 23, 2024

update

794e35f

hfxsd reviewed Sep 23, 2024

View reviewed changes

character-set-gb18030.md Outdated Show resolved Hide resolved

hfxsd reviewed Sep 23, 2024

View reviewed changes

character-set-gb18030.md Outdated Show resolved Hide resolved

hfxsd reviewed Sep 23, 2024

View reviewed changes

character-set-gb18030.md Outdated Show resolved Hide resolved

hfxsd reviewed Sep 23, 2024

View reviewed changes

dm/dm-best-practices.md Outdated Show resolved Hide resolved

hfxsd approved these changes Sep 23, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Sep 23, 2024

Apply suggestions from code review

5bf0a04

Co-authored-by: xixirangrang <hfxsd@hotmail.com>

lilin90 reviewed Sep 24, 2024

View reviewed changes

character-set-gb18030.md Outdated Show resolved Hide resolved

lilin90 reviewed Sep 24, 2024

View reviewed changes

character-set-gb18030.md Outdated Show resolved Hide resolved

character-set-gb18030.md Outdated Show resolved Hide resolved

character-set-gb18030.md Outdated Show resolved Hide resolved

character-set-gb18030.md Outdated Show resolved Hide resolved

Update wording and punctuation

0241804

lilin90 reviewed Sep 24, 2024

View reviewed changes

lilin90 mentioned this pull request Sep 24, 2024

releases: add release notes for v8.4.0 #18556

Merged

16 tasks

CbcWestwolf and others added 2 commits September 25, 2024 10:31

Apply suggestions from code review

1ad75f9

Co-authored-by: Lilian Lee <lilin@pingcap.com>

update

63942e5

lilin90 removed the v8.4 This PR/issue applies to TiDB v8.4. label Oct 8, 2024

lilin90 added the v9.0 label Dec 30, 2024

lilin90 requested a review from Benjamin2037 December 30, 2024 06:46

lilin90 requested a review from hfxsd December 30, 2024 09:40

lilin90 assigned hfxsd and unassigned lilin90 Jan 6, 2025

update version to 9.0; specify the gb18030-2022 charset version

b3c949f

hfxsd added 2 commits January 10, 2025 09:37

Update character-set-gb18030.md

4c45bac

Update character-set-gb18030.md

c8d8590

hfxsd reviewed Jan 10, 2025

View reviewed changes

Apply suggestions from code review

4766193

Co-authored-by: xixirangrang <hfxsd@hotmail.com>

hfxsd requested a review from Frank945946 January 13, 2025 01:54

CbcWestwolf commented Jan 13, 2025

View reviewed changes

Frank945946 reviewed Jan 13, 2025

View reviewed changes

dveeden mentioned this pull request Jan 13, 2025

Character set and collation updates pingcap/docs#19976

Merged

15 tasks

ti-chi-bot mentioned this pull request Jan 14, 2025

Character set and collation updates (#19976) pingcap/docs#19986

Merged

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tidb: add description about GB18030 #18662

tidb: add description about GB18030 #18662

CbcWestwolf commented Sep 19, 2024 •

edited by lilin90

Loading

ti-chi-bot bot commented Sep 23, 2024

lilin90 Sep 24, 2024

CbcWestwolf Sep 25, 2024

shawn0915 commented Oct 30, 2024

CbcWestwolf commented Oct 30, 2024 •

edited

Loading

shawn0915 commented Nov 20, 2024

lilin90 commented Dec 30, 2024

CbcWestwolf commented Dec 31, 2024

ti-chi-bot bot commented Jan 6, 2025

hfxsd Jan 10, 2025

hfxsd Jan 10, 2025 •

edited

Loading

CbcWestwolf Jan 13, 2025

Frank945946 Jan 13, 2025

Frank945946 Jan 13, 2025

Frank945946 Jan 13, 2025

Frank945946 Jan 13, 2025


		使用 `utf8_general_ci`、`utf8mb4_general_ci`、`utf8_unicode_ci`、`utf8mb4_unicode_ci`、`utf8mb4_0900_ai_ci` 和 `gbk_chinese_ci` 中任一种时，字符串之间的比较是大小写不敏感 (case-insensitive) 和口音不敏感 (accent-insensitive) 的。同时，TiDB 还修正了排序规则的 `PADDING` 行为：
		使用 `utf8_general_ci`、`utf8mb4_general_ci`、`utf8_unicode_ci`、`utf8mb4_unicode_ci`、`utf8mb4_0900_ai_ci`、`gbk_chinese_ci` 和 `gb18030_chinese_ci` 中任一种时，字符串之间的比较是大小写不敏感 (case-insensitive) 和口音不敏感 (accent-insensitive) 的。同时，TiDB 还修正了排序规则的 `PADDING` 行为：


		TiDB 和 MySQL 在 `gb18030` 字符集的默认排序规则上存在差异，具体如下：

		- TiDB 的 `gb18030` 字符集的默认排序规则为 `gb18030_bin`。

		- TiDB 的 `gb18030` 字符集的默认排序规则为 `gb18030_bin`。
		- MySQL 的 `gb18030` 字符集的默认排序规则为 `gb18030_chinese_ci`。

tidb: add description about GB18030 #18662

Are you sure you want to change the base?

tidb: add description about GB18030 #18662

Conversation

CbcWestwolf commented Sep 19, 2024 • edited by lilin90 Loading

What is changed, added or deleted? (Required)

Which TiDB version(s) do your changes apply to? (Required)

What is the related PR or file link(s)?

Do your changes match any of the following descriptions?

ti-chi-bot bot commented Sep 23, 2024

[LGTM Timeline notifier]

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawn0915 commented Oct 30, 2024

CbcWestwolf commented Oct 30, 2024 • edited Loading

shawn0915 commented Nov 20, 2024

lilin90 commented Dec 30, 2024

CbcWestwolf commented Dec 31, 2024

ti-chi-bot bot commented Jan 6, 2025

Choose a reason for hiding this comment

hfxsd Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CbcWestwolf commented Sep 19, 2024 •

edited by lilin90

Loading

CbcWestwolf commented Oct 30, 2024 •

edited

Loading

hfxsd Jan 10, 2025 •

edited

Loading