Support New Collation

[Daemonxiao]

What problem does this PR solve?

Support new collation refers to #2444.
fix #2236.

What is changed and how it works?

Reference
TiDB collation doc
TiDB pr: new secondary index value format
TiDB pr: expression: set collation id to negative in tipb if new collations are enabled

1. When new collation is enabled, we turn the collation id to negative so that other cluster components (for example, TiKV) can be aware of it without any change to the protocol definition.

  private FieldType toPBFieldType(DataType fieldType) {
    return FieldType.newBuilder()
        .setTp(fieldType.getTypeCode())
        .setFlag(fieldType.getFlag())
        .setFlen((int) fieldType.getLength())
        .setDecimal(fieldType.getDecimal())
        .setCharset(fieldType.getCharset())
        .setCollate(Collation.rewriteNewCollationIDIfNeeded(fieldType.getCollationCode())) // we set new collation here
        .build();
  }

2. When new collation is enabled, reset the return type in some expression. Now, we only change collation in LogicalBinaryExpression, ComparisonBinaryExpression, and StringRegExpression which are related to comparing.

  private Expr.Builder scalarToPartialProto(Expression node, Object context) {
...
    // Return type
    if (Collation.isNewCollationEnabled()) {
      node.setNewCollation();        // change return collation.
    }
    builder.setFieldType(toPBFieldType(getType(node)));
...
    return builder;
  }

3. When new collation is enabled, encode restore data to the index value.

The column is a string type that uses the new collation.

• For type char, blob and unspecified, the collation is neither binary nor with the suffix “_bin”.
• For type varchar, the collation is not binary.

private static byte[] genIndexValueForCommonHandleVersion1(
      Row row, Handle handle, boolean distinct, TiIndexInfo tiIndexInfo, TiTableInfo tiTableInfo) {
...
    // encode restore data if needed.
    List<TiColumnInfo> columnInfoList = new ArrayList<>();
    List<Object> valueList = new ArrayList<>();
    for (TiIndexInfo index : tiTableInfo.getIndices()) {
      for (TiIndexColumn tiIndexColumn : index.getIndexColumns()) {
        TiColumnInfo indexColumnInfo = tiTableInfo.getColumn(tiIndexColumn.getOffset());
        DataType indexType = indexColumnInfo.getType();
        int prefixLength = (int) tiIndexColumn.getLength();
        if (needRestoreData(indexType)) {
          Object value = row.get(indexColumnInfo.getOffset(), indexColumnInfo.getType());
          if (value == null) {
            continue;
          } else if (Collation.isBinCollation(indexType.getCollationCode())) {
            continue;
          } else if (DataType.isLengthUnSpecified(prefixLength)) {
            valueList.add(value);
          } else if (indexType instanceof BytesType) {
            if (indexType.getCharset().equalsIgnoreCase("utf8")
                || indexType.getCharset().equalsIgnoreCase("utf8mb4")) {
              value = Converter.convertUtf8ToBytes(value, prefixLength);
              valueList.add(value);
            } else {
              value = Converter.convertToBytes(value, prefixLength);
              valueList.add(value);
            }
          }
          columnInfoList.add(indexColumnInfo);
        }
      }
    }
    if (valueList.size() > 0) {
      cdo.write(new RowEncoderV2().encode(columnInfoList, valueList));
    }
...
  }

Check List

Tests

• Unit test
• Integration test

Code changes

• Has exported function/method change
• Has exported variable/fields change
• Has interface methods change
• Has persistent data change

Side effects

• Possible performance regression
• Increased code complexity
• Breaking backward compatibility

Related changes

• Need to update the documentation
• Need to be included in the release note

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.