Split the original FAQ to multiple files

TOC.md

@@ -484,8 +484,12 @@
   + [Table Filter](/table-filter.md)
 + FAQs
   + [TiDB FAQs](/faq/tidb-faq.md)
-  + [TiDB Lightning FAQs](/tidb-lightning/tidb-lightning-faq.md)
+  + [SQL FAQs](/faq/sql-faq.md)
+  + [Deploy and Maintain FAQs](/faq/deploy-and-maintain-faq.md)
   + [Upgrade FAQs](/faq/upgrade-faq.md)
+  + [High Availability FAQs](/faq/high-availability-faq.md)
+  + [High Reliability FAQs](/faq/high-reliability-faq.md)
+  + [Migration FAQs](/faq/migration-tidb-faq.md)
 + [Glossary](/glossary.md)
 + Release Notes
   + [All Releases](/releases/release-notes.md)

error-codes.md

@@ -35,7 +35,7 @@ In addition, TiDB has the following unique error codes:
 
     A single transaction is too large.
 
-    See [the error message `transaction too large`](/faq/tidb-faq.md#the-error-message-transaction-too-large-is-displayed) for the cause and solution.
+    See [the error message `transaction too large`](/faq/migration-tidb-faq.md#the-error-message-transaction-too-large-is-displayed) for the cause and solution.
 
 * Error Number: 8005
 
@@ -455,7 +455,7 @@ In addition, TiDB has the following unique error codes:
 
     A single transaction is too large.
 
-    See [the error message `transaction too large`](/faq/tidb-faq.md#the-error-message-transaction-too-large-is-displayed) for the solution.
+    See [the error message `transaction too large`](/faq/migration-tidb-faq.md#the-error-message-transaction-too-large-is-displayed) for the solution.
 
 * Error Number: 9007
 

faq/high-availability-faq.md

@@ -0,0 +1,18 @@
+---
+title: High Availability FAQs
+summary: Learn about the FAQs related to high availability of TiDB.
+---
+
+# High Availability FAQs
+
+This document summarizes the FAQs related to high availability of TiDB.
+
+## How is TiDB strongly consistent?
+
+Data is redundantly copied between TiKV nodes using the [Raft consensus algorithm](https://raft.github.io/) to ensure recoverability should a node failure occur.
+
+At the bottom layer, TiKV uses a model of replication log + State Machine to replicate data. For the write requests, the data is written to a Leader and the Leader then replicates the command to its Followers in the form of log. When the majority of nodes in the cluster receive this log, this log is committed and can be applied into the State Machine.
+
+## What's the recommended solution for the deployment of three geo-distributed data centers?
+
+The architecture of TiDB guarantees that it fully supports geo-distribution and multi-activeness. Your data and applications are always-on. All the outages are transparent to your applications and your data can recover automatically. The operation depends on the network latency and stability. It is recommended to keep the latency within 5ms. Currently, we already have similar use cases. For details, contact [info@pingcap.com](mailto:info@pingcap.com).

faq/high-reliability-faq.md

@@ -0,0 +1,29 @@
+---
+title: High Reliability FAQs
+summary: Learn about the FAQs related to high reliability of TiDB.
+---
+
+# High Reliability FAQs
+
+This document summarizes the FAQs related to high reliability of TiDB.
+
+## Does TiDB support modifying the MySQL version string of the server to a specific one that is required by the security vulnerability scanning tool?
+
+Since v3.0.8, TiDB supports modifying the version string of the server by modifying [`server-version`](/tidb-configuration-file.md#server-version) in the configuration file. When you deploy TiDB using TiDB Ansible, you can also specify the proper version string by configuring `server-version` in the `conf/tidb.yml` configuration file to avoid the failure of security vulnerability scan.
+
+## What authentication protocols does TiDB support? What's the process?
+
+Like MySQL, TiDB supports the SASL protocol for user login authentication and password processing.
+
+When the client connects to TiDB, the challenge-response authentication mode starts. The process is as follows:
+
+1. The client connects to the server.
+2. The server sends a random string challenge to the client.
+3. The client sends the username and response to the server.
+4. The server verifies the response.
+
+## How to modify the user password and privilege?
+
+To modify the user password in TiDB, it is recommended to use `set password for 'root'@'%' = '0101001';` or `alter`, not `update mysql.user` which might lead to the condition that the password in other nodes is not refreshed timely.
+
+It is recommended to use the official standard statements when modifying the user password and privilege. For details, see [TiDB user account management](/user-account-management.md).

faq/migration-tidb-faq.md

@@ -0,0 +1,178 @@
+---
+title: Migration FAQs
+summary: Learn about the FAQs related to data migration.
+---
+
+# Migration FAQ
+
+This document summarizes the FAQs related to TiDB data migration.
+
+## Full data export and import
+
+### Does TiDB support Mydumper?
+
+Yes. See [Mydumper Instructions](/mydumper-overview.md).
+
+### Does TiDB support Loader?
+
+Yes. See [Loader Instructions](/loader-overview.md).
+
+### How to migrate an application running on MySQL to TiDB?
+
+Because TiDB supports most MySQL syntax, generally you can migrate your applications to TiDB without changing a single line of code in most cases.
+
+### If I accidentally import the MySQL user table into TiDB, or forget the password and cannot log in, how to deal with it?
+
+Restart the TiDB service, add the `-skip-grant-table=true` parameter in the configuration file. Log into the cluster without password and recreate the user, or recreate the `mysql.user` table. For the specific table schema, search the official documentation.
+
+### Can TiDB provide services while Loader is running?
+
+TiDB can provide services while Loader is running because Loader inserts the data logically. But do not perform the related DDL operations.
+
+### How to export the data in TiDB?
+
+Currently, TiDB does not support `select into outfile`. You can use the following methods to export the data in TiDB:
+
+- See [MySQL uses mysqldump to export part of the table data](https://blog.csdn.net/xin_yu_xin/article/details/7574662) in Chinese and export data using mysqldump and the `WHERE` clause.
+- Use the MySQL client to export the results of `select` to a file.
+
+### How to migrate from DB2 or Oracle to TiDB?
+
+To migrate all the data or migrate incrementally from DB2 or Oracle to TiDB, see the following solution:
+
+- Use the official migration tool of Oracle, such as OGG, Gateway, CDC (Change Data Capture).
+- Develop a program for importing and exporting data.
+- Export Spool as text file, and import data using Load infile.
+- Use a third-party data migration tool.
+
+Currently, it is recommended to use OGG.
+
+### Error: `java.sql.BatchUpdateExecption:statement count 5001 exceeds the transaction limitation` while using Sqoop to write data into TiDB in `batches`
+
+In Sqoop, `--batch` means committing 100 `statement`s in each batch, but by default each `statement` contains 100 SQL statements. So, 100 * 100 = 10000 SQL statements, which exceeds 5000, the maximum number of statements allowed in a single TiDB transaction.
+
+Two solutions:
+
+- Add the `-Dsqoop.export.records.per.statement=10` option as follows:
+
+    {{< copyable "shell-regular" >}}
+
+    ```bash
+    sqoop export \
+        -Dsqoop.export.records.per.statement=10 \
+        --connect jdbc:mysql://mysql.example.com/sqoop \
+        --username sqoop ${user} \
+        --password ${passwd} \
+        --table ${tab_name} \
+        --export-dir ${dir} \
+        --batch
+    ```
+
+- You can also increase the limited number of statements in a single TiDB transaction, but this will consume more memory.
+
+### Does TiDB have a function like the Flashback Query in Oracle? Does it support DDL?
+
+ Yes, it does. And it supports DDL as well. For details, see [how TiDB reads data from history versions](/read-historical-data.md).
+
+## Migrate the data online
+
+### Syncer infrastructure
+
+See [Parsing TiDB online data synchronization tool Syncer](https://pingcap.com/blog-cn/tidb-syncer/)in Chinese.
+
+#### Syncer user guide
+
+See [Syncer User Guide](/syncer-overview.md).
+
+#### How to configure to monitor Syncer status?
+
+Download and import [Syncer Json](https://github.com/pingcap/docs/blob/master/etc/Syncer.json) to Grafana. Edit the Prometheus configuration file and add the following content:
+
+```
+- job_name: 'syncer_ops' // task name
+    static_configs:
+      - targets: [’10.10.1.1:10096’] // Syncer monitoring address and port, informing Prometheus to pull the data of Syncer
+```
+
+Restart Prometheus.
+
+#### Is there a current solution to replicating data from TiDB to other databases like HBase and Elasticsearch?
+
+No. Currently, the data replication depends on the application itself.
+
+#### Does Syncer support replicating only some of the tables when Syncer is replicating data?
+
+Yes. For details, see [Syncer User Guide](/syncer-overview.md)
+
+#### Do frequent DDL operations affect the replication speed of Syncer?
+
+Frequent DDL operations may affect the replication speed. For Sycner, DDL operations are executed serially. When DDL operations are executed during data replication, data will be replicated serially and thus the replication speed will be slowed down.
+
+#### If the machine that Syncer is in is broken and the directory of the `syncer.meta` file is lost, what should I do?
+
+When you replicate data using Syncer GTID, the `syncer.meta` file is constantly updated during the replication process. The current version of Syncer does not contain the design for high availability. The `syncer.meta` configuration file of Syncer is directly stored on the hard disks, which is similar to other tools in the MySQL ecosystem, such as Mydumper.
+
+Two solutions:
+
+- Put the `syncer.meta` file in a relatively secure disk. For example, use disks with RAID 1.
+- Restore the location information of history replication according to the monitoring data that Syncer reports to Prometheus regularly. But the location information might be inaccurate due to the delay when a large amount of data is replicated.
+
+#### If the downstream TiDB data is not consistent with the MySQL data during the replication process of Syncer, will DML operations cause exits?
+
+- If the data exists in the upstream MySQL but does not exist in the downstream TiDB, when the upstream MySQL performs the `UPDATE` or `DELETE` operation on this row of data, Syncer will not report an error and the replication process will not exit, and this row of data does not exist in the downstream.
+- If a conflict exists in the primary key indexes or the unique indexes in the downstream, preforming the `UPDATE` operation will cause an exit and performing the `INSERT` operation will not cause an exit.
+
+## Migrate the traffic
+
+### How to migrate the traffic quickly?
+
+It is recommended to build a multi-source MySQL -> TiDB real-time replication environment using Syncer tool. You can migrate the read and write traffic in batches by editing the network configuration as needed. Deploy a stable network LB (HAproxy, LVS, F5, DNS, etc.) on the upper layer, in order to implement seamless migration by directly editing the network configuration.
+
+### Is there a limit for the total write and read capacity in TiDB?
+
+The total read capacity has no limit. You can increase the read capacity by adding more TiDB servers. Generally the write capacity has no limit as well. You can increase the write capacity by adding more TiKV nodes.
+
+### The error message `transaction too large` is displayed
+
+Due to the limitation of the underlying storage engine, each key-value entry (one row) in TiDB should be no more than 6MB.
+
+Distributed transactions need two-phase commit and the bottom layer performs the Raft replication. If a transaction is very large, the commit process would be quite slow and the write conflict is more likely to occur. Moreover, the rollback of a failed transaction leads to an unnecessary performance penalty. To avoid these problems, we limit the total size of key-value entries to no more than 100MB in a transaction by default. If you need larger transactions, modify the value of `txn-total-size-limit` in the TiDB configuration file. The maximum value of this configuration item is up to 10G. The actual limitation is also affected by the physical memory of the machine.
+
+There are [similar limits](https://cloud.google.com/spanner/docs/limits) on Google Cloud Spanner.
+
+### How to import data in batches?
+
+When you import data, insert in batches and keep the number of rows within 10,000 for each batch.
+
+### Does TiDB release space immediately after deleting data?
+
+None of the `DELETE`, `TRUNCATE` and `DROP` operations release data immediately. For the `TRUNCATE` and `DROP` operations, after the TiDB GC (Garbage Collection) time (10 minutes by default), the data is deleted and the space is released. For the `DELETE` operation, the data is deleted but the space is not released according to TiDB GC. When subsequent data is written into RocksDB and executes `COMPACT`, the space is reused.
+
+### Can I execute DDL operations on the target table when loading data?
+
+No. None of the DDL operations can be executed on the target table when you load data, otherwise the data fails to be loaded.
+
+### Does TiDB support the `replace into` syntax?
+
+Yes. But the `load data` does not support the `replace into` syntax.
+
+### Why does the query speed getting slow after deleting data?
+
+Deleting a large amount of data leaves a lot of useless keys, affecting the query efficiency. Currently the Region Merge feature is in development, which is expected to solve this problem. For details, see the [deleting data section in TiDB Best Practices](https://pingcap.com/blog/2017-07-24-tidbbestpractice/#write).
+
+### What is the most efficient way of deleting data?
+
+When deleting a large amount of data, it is recommended to use `Delete * from t where xx limit 5000;`. It deletes through the loop and uses `Affected Rows == 0` as a condition to end the loop, so as not to exceed the limit of transaction size. With the prerequisite of meeting business filtering logic, it is recommended to add a strong filter index column or directly use the primary key to select the range, such as `id >= 5000*n+m and id < 5000*(n+1)+m`.
+
+If the amount of data that needs to be deleted at a time is very large, this loop method will get slower and slower because each deletion traverses backward. After deleting the previous data, lots of deleted flags remain for a short period (then all will be processed by Garbage Collection) and influence the following Delete statement. If possible, it is recommended to refine the Where condition. See [details in TiDB Best Practices](https://pingcap.com/blog/2017-07-24-tidbbestpractice/#write).
+
+### How to improve the data loading speed in TiDB?
+
+- The [Lightning](/tidb-lightning/tidb-lightning-overview.md) tool is developed for distributed data import. It should be noted that the data import process does not perform a complete transaction process for performance reasons. Therefore, the ACID constraint of the data being imported during the import process cannot be guaranteed. The ACID constraint of the imported data can only be guaranteed after the entire import process ends. Therefore, the applicable scenarios mainly include importing new data (such as a new table or a new index) or the full backup and restoring (truncate the original table and then import data).
+- Data loading in TiDB is related to the status of disks and the whole cluster. When loading data, pay attention to metrics like the disk usage rate of the host, TiClient Error, Backoff, Thread CPU and so on. You can analyze the bottlenecks using these metrics.
+
+### What should I do if it is slow to reclaim storage space after deleting data?
+
+You can configure concurrent GC to increase the speed of reclaiming storage space. The default concurrency is 1, and you can modify it to at most 50% of the number of TiKV instances using the following command:
+
+{{< copyable "sql" >}}