Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tidb: add description about GB18030 #18662

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
+198 −95
Open

tidb: add description about GB18030 #18662

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
edf6730
tidb: add description about GB18030
CbcWestwolf Sep 14, 2024
794e35f
update
CbcWestwolf Sep 23, 2024
5bf0a04
Apply suggestions from code review
CbcWestwolf Sep 24, 2024
0241804
Update wording and punctuation
lilin90 Sep 24, 2024
1ad75f9
Apply suggestions from code review
CbcWestwolf Sep 25, 2024
63942e5
update
CbcWestwolf Sep 25, 2024
b3c949f
update version to 9.0; specify the gb18030-2022 charset version
CbcWestwolf Jan 6, 2025
4c45bac
Update character-set-gb18030.md
hfxsd Jan 10, 2025
c8d8590
Update character-set-gb18030.md
hfxsd Jan 10, 2025
4766193
Apply suggestions from code review
CbcWestwolf Jan 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions TOC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -951,6 +951,7 @@
- 字符集和排序规则
- [概述](/character-set-and-collation.md)
- [GBK](/character-set-gbk.md)
- [GB18030](/character-set-gb18030.md)
- [Placement Rules in SQL](/placement-rules-in-sql.md)
- 系统表
- `mysql` Schema
Expand Down
1 change: 1 addition & 0 deletions br/backup-and-restore-overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ TiDB 支持将数据备份到 Amazon S3、Google Cloud Storage (GCS)、Azure Blo
| 功能 | 相关 issue | 解决方式 |
| ---- | ---- | ----- |
|GBK charset|| BR 在 v5.4.0 之前不支持恢复 `charset=GBK` 的表。并且，任何版本的 BR 都不支持恢复 `charset=GBK` 的表到 v5.4.0 之前的 TiDB 集群。|
|GB18030 charset|| BR 在 v9.0.0 之前不支持恢复 `charset=GB18030` 的表。并且，任何版本的 BR 都不支持恢复 `charset=GB18030` 的表到 v9.0.0 之前的 TiDB 集群。|
| 聚簇索引 | [#565](https://github.com/pingcap/br/issues/565) | 确保恢复时集群的 `tidb_enable_clustered_index` 全局变量和备份时一致，否则会导致数据不一致的问题，例如 `default not found` 和数据索引不一致。 |
| New collation | [#352](https://github.com/pingcap/br/issues/352) | 确保恢复时集群的 `mysql.tidb` 表中 `new_collation_enabled` 变量值和备份时的一致，否则会导致数据索引不一致和 checksum 通不过。更多信息，请参考 [FAQ - BR 为什么会报 `new_collations_enabled_on_first_bootstrap` 不匹配？](/faq/backup-and-restore-faq.md#恢复时为什么会报-new_collation_enabled-不匹配)。 |
| 全局临时表 | | 确保使用 BR v5.3.0 及以上版本进行备份和恢复，否则会导致全局临时表的表定义错误。 |
Expand Down
69 changes: 36 additions & 33 deletions character-set-and-collation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: 字符集和排序规则
aliases: ['/docs-cn/dev/character-set-and-collation/','/docs-cn/dev/reference/sql/characterset-and-collation/','/docs-cn/dev/reference/sql/character-set/']
summary: TiDB 支持的字符集包括 ascii、binary、gbk、latin1、utf8 和 utf8mb4。排序规则包括 ascii_bin、binary、gbk_bin、gbk_chinese_ci、latin1_bin、utf8_bin、utf8_general_ci、utf8_unicode_ci、utf8mb4_0900_ai_ci、utf8mb4_0900_bin、utf8mb4_bin、utf8mb4_general_ci 和 utf8mb4_unicode_ci。TiDB 强烈建议使用 utf8mb4 字符集，因为它支持更多字符。在 TiDB 中，默认的排序规则受到客户端的连接排序规则设置的影响。如果客户端使用 utf8mb4_0900_ai_ci 作为连接排序规则，TiDB 将遵循客户端的配置。TiDB 还支持新的排序规则框架，用于在语义上支持不同的排序规则。
summary: TiDB 支持的字符集包括 ascii、binary、gbk、gb18030、latin1、utf8 和 utf8mb4。排序规则包括 ascii_bin、binary、gbk_bin、gbk_chinese_ci、gb18030_bin、gb18030_chinese_ci、latin1_bin、utf8_bin、utf8_general_ci、utf8_unicode_ci、utf8mb4_0900_ai_ci、utf8mb4_0900_bin、utf8mb4_bin、utf8mb4_general_ci 和 utf8mb4_unicode_ci。TiDB 强烈建议使用 utf8mb4 字符集，因为它支持更多字符。在 TiDB 中，默认的排序规则受到客户端的连接排序规则设置的影响。如果客户端使用 utf8mb4_0900_ai_ci 作为连接排序规则，TiDB 将遵循客户端的配置。TiDB 还支持新的排序规则框架，用于在语义上支持不同的排序规则。
---

# 字符集和排序规则
Expand Down Expand Up @@ -68,17 +68,18 @@ SHOW CHARACTER SET;
```

```sql
+---------+-------------------------------------+-------------------+--------+
| Charset | Description | Default collation | Maxlen |
+---------+-------------------------------------+-------------------+--------+
| ascii | US ASCII | ascii_bin | 1 |
| binary | binary | binary | 1 |
| gbk | Chinese Internal Code Specification | gbk_bin | 2 |
| latin1 | Latin1 | latin1_bin | 1 |
| utf8 | UTF-8 Unicode | utf8_bin | 3 |
| utf8mb4 | UTF-8 Unicode | utf8mb4_bin | 4 |
+---------+-------------------------------------+-------------------+--------+
6 rows in set (0.00 sec)
+---------+-------------------------------------+--------------------+--------+
| Charset | Description | Default collation | Maxlen |
+---------+-------------------------------------+--------------------+--------+
| ascii | US ASCII | ascii_bin | 1 |
| binary | binary | binary | 1 |
| gb18030 | China National Standard GB18030 | gb18030_chinese_ci | 4 |
| gbk | Chinese Internal Code Specification | gbk_chinese_ci | 2 |
| latin1 | Latin1 | latin1_bin | 1 |
| utf8 | UTF-8 Unicode | utf8_bin | 3 |
| utf8mb4 | UTF-8 Unicode | utf8mb4_bin | 4 |
+---------+-------------------------------------+--------------------+--------+
7 rows in set (0.00 sec)
```

TiDB 支持以下排序规则：
Expand All @@ -88,24 +89,26 @@ SHOW COLLATION;
```

```sql
+--------------------+---------+------+---------+----------+---------+
| Collation | Charset | Id | Default | Compiled | Sortlen |
+--------------------+---------+------+---------+----------+---------+
| ascii_bin | ascii | 65 | Yes | Yes | 1 |
| binary | binary | 63 | Yes | Yes | 1 |
| gbk_bin | gbk | 87 | | Yes | 1 |
| gbk_chinese_ci | gbk | 28 | Yes | Yes | 1 |
| latin1_bin | latin1 | 47 | Yes | Yes | 1 |
| utf8_bin | utf8 | 83 | Yes | Yes | 1 |
| utf8_general_ci | utf8 | 33 | | Yes | 1 |
| utf8_unicode_ci | utf8 | 192 | | Yes | 1 |
| utf8mb4_0900_ai_ci | utf8mb4 | 255 | | Yes | 1 |
| utf8mb4_0900_bin | utf8mb4 | 309 | | Yes | 1 |
| utf8mb4_bin | utf8mb4 | 46 | Yes | Yes | 1 |
| utf8mb4_general_ci | utf8mb4 | 45 | | Yes | 1 |
| utf8mb4_unicode_ci | utf8mb4 | 224 | | Yes | 1 |
+--------------------+---------+------+---------+----------+---------+
13 rows in set (0.00 sec)
+--------------------+---------+-----+---------+----------+---------+---------------+
| Collation | Charset | Id | Default | Compiled | Sortlen | Pad_attribute |
+--------------------+---------+-----+---------+----------+---------+---------------+
| ascii_bin | ascii | 65 | Yes | Yes | 1 | PAD SPACE |
| binary | binary | 63 | Yes | Yes | 1 | NO PAD |
| gb18030_bin | gb18030 | 249 | | Yes | 1 | PAD SPACE |
| gb18030_chinese_ci | gb18030 | 248 | Yes | Yes | 1 | PAD SPACE |
| gbk_bin | gbk | 87 | | Yes | 1 | PAD SPACE |
| gbk_chinese_ci | gbk | 28 | Yes | Yes | 1 | PAD SPACE |
| latin1_bin | latin1 | 47 | Yes | Yes | 1 | PAD SPACE |
| utf8_bin | utf8 | 83 | Yes | Yes | 1 | PAD SPACE |
| utf8_general_ci | utf8 | 33 | | Yes | 1 | PAD SPACE |
| utf8_unicode_ci | utf8 | 192 | | Yes | 8 | PAD SPACE |
| utf8mb4_0900_ai_ci | utf8mb4 | 255 | | Yes | 0 | NO PAD |
| utf8mb4_0900_bin | utf8mb4 | 309 | | Yes | 1 | NO PAD |
| utf8mb4_bin | utf8mb4 | 46 | Yes | Yes | 1 | PAD SPACE |
| utf8mb4_general_ci | utf8mb4 | 45 | | Yes | 1 | PAD SPACE |
| utf8mb4_unicode_ci | utf8mb4 | 224 | | Yes | 8 | PAD SPACE |
+--------------------+---------+-----+---------+----------+---------+---------------+
15 rows in set (0.02 sec)
```

> **警告：**
Expand Down Expand Up @@ -141,7 +144,7 @@ SHOW COLLATION WHERE Charset = 'utf8mb4';
5 rows in set (0.00 sec)
```

TiDB 对 GBK 字符集的支持详情见 [GBK](/character-set-gbk.md)。
TiDB 对 GBK 字符集的支持详情见 [GBK](/character-set-gbk.md)，对 GB18030 字符集的支持详情见 [GB18030](/character-set-gb18030.md)。

## TiDB 中的 `utf8` 和 `utf8mb4`

Expand Down Expand Up @@ -514,9 +517,9 @@ SELECT VARIABLE_VALUE FROM mysql.tidb WHERE VARIABLE_NAME='new_collation_enabled
1 row in set (0.00 sec)
```

在新的排序规则框架下，TiDB 能够支持 `utf8_general_ci`、`utf8mb4_general_ci`、`utf8_unicode_ci`、`utf8mb4_unicode_ci`、`utf8mb4_0900_bin`、`utf8mb4_0900_ai_ci`、`gbk_chinese_ci` 和 `gbk_bin` 这几种排序规则，与 MySQL 兼容。
在新的排序规则框架下，TiDB 能够支持 `utf8_general_ci`、`utf8mb4_general_ci`、`utf8_unicode_ci`、`utf8mb4_unicode_ci`、`utf8mb4_0900_bin`、`utf8mb4_0900_ai_ci`、`gbk_chinese_ci`、`gbk_bin`、`gb18030_chinese_ci` 和 `gb18030_bin` 这几种排序规则，与 MySQL 兼容。

使用 `utf8_general_ci`、`utf8mb4_general_ci`、`utf8_unicode_ci`、`utf8mb4_unicode_ci`、`utf8mb4_0900_ai_ci` 和 `gbk_chinese_ci` 中任一种时，字符串之间的比较是大小写不敏感 (case-insensitive) 和口音不敏感 (accent-insensitive) 的。同时，TiDB 还修正了排序规则的 `PADDING` 行为：
使用 `utf8_general_ci`、`utf8mb4_general_ci`、`utf8_unicode_ci`、`utf8mb4_unicode_ci`、`utf8mb4_0900_ai_ci`、`gbk_chinese_ci` 和 `gb18030_chinese_ci` 中任一种时，字符串之间的比较是大小写不敏感 (case-insensitive) 和口音不敏感 (accent-insensitive) 的。同时，TiDB 还修正了排序规则的 `PADDING` 行为：
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

L580 优先级排序 需要加上 gbk_chinese_ci 和 gb18030_chinese_ci 吗？


{{< copyable "sql" >}}

Expand Down
83 changes: 83 additions & 0 deletions character-set-gb18030.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
---
title: GB18030
summary: 本文介绍 TiDB 对 GB18030 字符集的支持情况。
---

# GB18030

TiDB 从 v9.0.0 开始支持 GB18030-2022 字符集。本文档介绍 TiDB 对 GB18030 字符集的支持和兼容情况。

```sql
SHOW CHARACTER SET WHERE CHARSET = 'gb18030';
```

```
+---------+---------------------------------+--------------------+--------+
| Charset | Description | Default collation | Maxlen |
+---------+---------------------------------+--------------------+--------+
| gb18030 | China National Standard GB18030 | gb18030_chinese_ci | 4 |
+---------+---------------------------------+--------------------+--------+
1 row in set (0.01 sec)
```

```
SHOW COLLATION WHERE CHARSET = 'gb18030';
```

```
+-------------+---------+-----+---------+----------+---------+---------------+
| Collation | Charset | Id | Default | Compiled | Sortlen | Pad_attribute |
+-------------+---------+-----+---------+----------+---------+---------------+
| gb18030_bin | gb18030 | 249 | Yes | Yes | 1 | PAD SPACE |
+-------------+---------+-----+---------+----------+---------+---------------+
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CbcWestwolf 这里的结果是不是漏了 ”gb18030_chinese_ci“ 这个 collation ？

1 row in set (0.00 sec)
```

## 与 MySQL 的兼容性

本节介绍 TiDB 中 GB18030 字符集与 MySQL 的兼容情况。

### 排序规则兼容性

TiDB 和 MySQL 在 `gb18030` 字符集的默认排序规则上存在差异，具体如下：

- TiDB 的 `gb18030` 字符集的默认排序规则为 `gb18030_bin`。
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: 更改默认值描述

- MySQL 的 `gb18030` 字符集的默认排序规则为 `gb18030_chinese_ci`。
Comment on lines +44 to +45
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CbcWestwolf 这里为何不与 MySQL 保持一致？如果用户 MySQL 默认字符集是 gb18030 没有显式指定 collation，那迁移到 TiDB 就会变成 gb18030_bin，会不会有问题？能否保持一致的行为，来避免类似的问题？

- TiDB 支持的 `gb18030_bin` 与 MySQL 支持的 `gb18030_bin` 排序规则实现不同，TiDB 通过将 `gb18030` 字符集转换为 `utf8mb4` 然后进行二进制排序。

如果要使 TiDB 兼容 MySQL GB18030 字符集的排序规则，你需要在首次初始化 TiDB 集群时，将 TiDB 配置项 [`new_collations_enabled_on_first_bootstrap`](/tidb-configuration-file.md#new_collations_enabled_on_first_bootstrap) 设置为 `true` 来开启[新的排序规则框架](/character-set-and-collation.md#新框架下的排序规则支持)。开启新的排序规则框架后，查看 GB18030 字符集对应的排序规则，可以看到 TiDB GB18030 默认排序规则已经切换为 `gb18030_chinese_ci`。
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CbcWestwolf @hfxsd 我看默认值就是 ture，对于新集群就不会有问题，是不是只需要强调低版本如果设置为了 false，升级到高版本需要手动设置这个配置项为 ture？


```sql
SHOW CHARACTER SET WHERE CHARSET = 'gb18030';
```

```
+---------+---------------------------------+--------------------+--------+
| Charset | Description | Default collation | Maxlen |
+---------+---------------------------------+--------------------+--------+
| gb18030 | China National Standard GB18030 | gb18030_chinese_ci | 4 |
+---------+---------------------------------+--------------------+--------+
1 row in set (0.01 sec)
```

```
SHOW COLLATION WHERE CHARSET = 'gb18030';
```

```
+--------------------+---------+-----+---------+----------+---------+---------------+
| Collation | Charset | Id | Default | Compiled | Sortlen | Pad_attribute |
+--------------------+---------+-----+---------+----------+---------+---------------+
| gb18030_bin | gb18030 | 249 | | Yes | 1 | PAD SPACE |
| gb18030_chinese_ci | gb18030 | 248 | Yes | Yes | 1 | PAD SPACE |
+--------------------+---------+-----+---------+----------+---------+---------------+
2 rows in set (0.00 sec)
```

### 非法字符兼容性

* 如果系统变量 [`character_set_client`](/system-variables.md#character_set_client) 和 [`character_set_connection`](/system-variables.md#character_set_connection) 没有同时设置为 `gb18030`，TiDB 处理非法字符的方式与 MySQL 一致。
* 如果系统变量 `character_set_client` 和 `character_set_connection` 同时设置为 `gb18030`，TiDB 处理非法字符的方式与 MySQL 有如下区别：

- MySQL 处理非法 GB18030 字符集时，对读和写操作的处理方式不同。
Copy link
Collaborator

@hfxsd hfxsd Jan 10, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

需要介绍分别如何处理的吗？或者给个 MySQL 相关内容的链接。

- TiDB 处理非法 GB18030 字符集时，对读和写操作的处理方式相同。TiDB 在严格模式下读写非法 GB18030 字符都会报错；在非严格模式下，读写非法 GB18030 字符都会用 `?` 替换。
24 changes: 12 additions & 12 deletions character-set-gbk.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,12 @@ SHOW CHARACTER SET WHERE CHARSET = 'gbk';
1 row in set (0.00 sec)

SHOW COLLATION WHERE CHARSET = 'gbk';
+----------------+---------+------+---------+----------+---------+
| Collation | Charset | Id | Default | Compiled | Sortlen |
+----------------+---------+------+---------+----------+---------+
| gbk_bin | gbk | 87 | | Yes | 1 |
+----------------+---------+------+---------+----------+---------+
1 rows in set (0.00 sec)
+-----------+---------+----+---------+----------+---------+---------------+
| Collation | Charset | Id | Default | Compiled | Sortlen | Pad_attribute |
+-----------+---------+----+---------+----------+---------+---------------+
| gbk_bin | gbk | 87 | Yes | Yes | 1 | PAD SPACE |
+-----------+---------+----+---------+----------+---------+---------------+
Comment on lines +20 to +24
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里是不是也漏了返回 gbk_chinese_ci 这个collation 结果？

1 row in set (0.00 sec)
```

## 与 MySQL 的兼容性
Expand All @@ -47,12 +47,12 @@ SHOW CHARACTER SET WHERE CHARSET = 'gbk';
1 row in set (0.00 sec)

SHOW COLLATION WHERE CHARSET = 'gbk';
+----------------+---------+------+---------+----------+---------+
| Collation | Charset | Id | Default | Compiled | Sortlen |
+----------------+---------+------+---------+----------+---------+
| gbk_bin | gbk | 87 | | Yes | 1 |
| gbk_chinese_ci | gbk | 28 | Yes | Yes | 1 |
+----------------+---------+------+---------+----------+---------+
+----------------+---------+----+---------+----------+---------+---------------+
| Collation | Charset | Id | Default | Compiled | Sortlen | Pad_attribute |
+----------------+---------+----+---------+----------+---------+---------------+
| gbk_bin | gbk | 87 | | Yes | 1 | PAD SPACE |
| gbk_chinese_ci | gbk | 28 | Yes | Yes | 1 | PAD SPACE |
+----------------+---------+----+---------+----------+---------+---------------+
2 rows in set (0.00 sec)
```

Expand Down
3 changes: 2 additions & 1 deletion dm/dm-best-practices.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,10 +117,11 @@ DM 默认会使用悲观 DDL 锁模式。在分库分表迁移与同步场景中

TiDB 默认使用的字符集为 utf8mb4。建议同步上下游及应用统一使用 utf8mb4。如果上游有显式指定的字符集或者排序规则，需要确认 TiDB 是否支持。

从 v6.0.0 起，TiDB 支持 GBK 字符集。有关字符集的限制详见：
从 v6.0.0 起，TiDB 支持 GBK 字符集；从 v9.0.0 起，TiDB 支持 GB18030 字符集。有关字符集的限制详见：

- [字符集和排序规则](/character-set-and-collation.md)
- [GBK 兼容情况](/character-set-gbk.md#与-mysql-的兼容性)
- [GB18030 兼容情况](/character-set-gb18030.md#与-mysql-的兼容性)

### 实施侧要点

Expand Down
4 changes: 2 additions & 2 deletions dm/dm-overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,9 @@ tiup install dm dmctl

- DM 不会将视图的 DDL 语句同步到下游的 TiDB 集群，也不会将针对视图的 DML 语句同步到下游。在该场景下，建议用户在下游 TiDB 集群中自行创建视图。

+ GBK 字符集兼容性限制
+ GBK/GB18030 字符集兼容性限制

- DM 在 v5.4.0 之前不支持将 `charset=GBK` 的表迁移到 TiDB。
- DM 在 v5.4.0 之前不支持将 `charset=GBK` 的表迁移到 TiDB；在 v9.0.0 之前不支持将 `charset=GB18030` 的表迁移到 TiDB。

+ Binlog 兼容性限制

Expand Down
3 changes: 2 additions & 1 deletion information-schema/information-schema-character-sets.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,13 @@ SELECT * FROM `CHARACTER_SETS`;
+--------------------+----------------------+-------------------------------------+--------+
| ascii | ascii_bin | US ASCII | 1 |
| binary | binary | binary | 1 |
| gb18030 | gb18030_chinese_ci | China National Standard GB18030 | 4 |
| gbk | gbk_chinese_ci | Chinese Internal Code Specification | 2 |
| latin1 | latin1_bin | Latin1 | 1 |
| utf8 | utf8_bin | UTF-8 Unicode | 3 |
| utf8mb4 | utf8mb4_bin | UTF-8 Unicode | 4 |
+--------------------+----------------------+-------------------------------------+--------+
6 rows in set (0.00 sec)
7 rows in set (0.00 sec)
```

`CHARACTER_SETS` 表中列的含义如下：
Expand Down
Loading
Loading