Prefix index breaks for ASCII 0xFF(ÿ)

[birdstorm]

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   If possible, provide a recipe for reproducing the error.

After #7109 merged:

CREATE TABLE `prefix` (
  `a` int(11) NOT NULL,
  `b` varchar(55) DEFAULT NULL,
  `c` int(11) DEFAULT NULL,
  PRIMARY KEY (`a`),
  KEY `prefix_index` (`b`(2)),
  KEY `prefix_complex` (`a`,`b`(2))
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;

INSERT INTO `prefix` VALUES(0, "b", 2), (1, "bbb", 3), (2, "bbc", 4), (3, "bbb", 5), (4, "abc", 6), (5, "abc", 7), (6, "abc", 7), (7, "ÿÿ", 8), (8, "ÿÿ0", 9), (9, "ÿÿÿ", 10)

2. What did you expect to see?

mysql> select c, b from prefix where b > "ÿ" and b < "ÿÿc";
+------+-------+
| c    | b     |
+------+-------+
|    8 | ÿÿ    |
|    9 | ÿÿ0   |
+------+-------+
2 rows in set (0.01 sec)

3. What did you see instead?

mysql> select c, b from prefix where b > "ÿ" and b < "ÿÿc";
Empty set (0.00 sec)

4. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?

Release Version: v2.1.0-beta-83-g48a42fe4
Git Commit Hash: 48a42fe4242b9fb1e1d274bfe3c3b9df99eba43f
Git Branch: master
UTC Build Time: 2018-07-19 07:24:43
GoVersion: go version go1.10.1 darwin/amd64
Race Enabled: false
TiKV Min Version: 2.1.0-alpha.1-ff3dd160846b7d1aed9079c389fc188f7f5ea13e

[coocood]

@birdstorm
Why is the expected result ÿ0 instead of ÿÿ0 in the insert statement?

[birdstorm]

@coocood Sorry, I copied the wrong result. The issue description has been updated.

[coocood]

I tried to reproduce the issue, but
the result is correct.

[birdstorm]

@coocood This should be a problem with your locale.

Please set your locale to zh_CN.UTF-8 and you should be able to reproduce this.

Confirmed with @winoros

[birdstorm]

Problem lies here check this line

func fixRangeDatum(v *types.Datum, length int) {
	// If this column is prefix and the prefix length is smaller than the range, cut it.
	if length != types.UnspecifiedLength && length < len(v.GetBytes()) {
		v.SetBytes(v.GetBytes()[:length])
	}
}

The ranger cut the prefix by its bytes.

@coocood

[coocood]

@birdstorm
Good job!

@winoros PTAL