fix: re-enable (xor) bloom filter index

[dantengsky]

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

as xor bloom has been merged in PR #7860, replace the vanilla bloom filter with it.

• for legacy index data of v1 bloom filter, the index will be ignored
  but will be removed if table data is dropped
• xor index is created under path "_i_b_v2"
  so that, other types of indexes could be put in separated "dir"s
• bloom filter index is enabled for string type column only

Fixes #issue

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated
databend	⬜️ Ignored (Inspect)		Sep 27, 2022 at 1:46AM (UTC)

[BohuTANG]

Cool.
The bloom filter should have a type so that we can try to fuse binary bloom(in xorfilter crate) in the future:
Binary Fuse Filters: Fast and Smaller Than Xor Filters

Fuse 8 Disadvantages:

 Fuse8 filter can only tolerate few duplicates in a given data-set. So make sure to supply a hasher that is capable of generating unique digests, (with allowed tolerance of duplicates) and while supplying the digests directly via populate_keys() and build_keys() make sure they don't have more than few duplicates.

[BohuTANG]

From the test, XOR8 is better than FUSE8 for the applications, FUSE8 should check the duplicate keys(like hyperloglog)
Test Result:

fuse8: u64 bitmap encoding:1130544 bytes, raw:8000000 bytes, ratio:0.141318
fuse8: bool bitmap encoding:1130544 bytes, raw:1000000 bytes, ratio:1.130544
fuse8: string encoding:118832 bytes, raw:3000000 bytes, ratio:0.039610665
fuse8: same string encoding:118832 bytes, raw:3000000 bytes, ratio:0.039610665
xor8: u64 bitmap encoding:1230069 bytes, raw:8000000 bytes, ratio:0.15375863
xor8: bool bitmap encoding:61 bytes, raw:1000000 bytes, ratio:0.000061
xor8: string encoding:123067 bytes, raw:3000000 bytes, ratio:0.041022334
xor8: same string encoding:61 bytes, raw:3000000 bytes, ratio:0.000020333333

BohuTANG@b45793c

[BohuTANG]

I will bench the xor bloom index loading after #7867 and #7857

[BohuTANG]

Test result:

Gen data(unique):

create table t10b as select number as c1, cast(rand() as string) as c2 from numbers(1000000000);

main branch:

mysql> select * from t10b where c2='0.4091273956277217';
Connection id:    9
Current database: bloom

+-----------+--------------------+
| c1        | c2                 |
+-----------+--------------------+
| 125000007 | 0.4091273956277217 |
+-----------+--------------------+
1 row in set (1 min 13.72 sec)
Read 1000000000 rows, 24.47 GiB in 73.697 sec., 13.57 million rows/sec., 340.05 MiB/sec.

This branch:

mysql> select * from t10b where c2='0.4091273956277217';
Connection id:    9
Current database: bloom

+-----------+--------------------+
| c1        | c2                 |
+-----------+--------------------+
| 125000007 | 0.4091273956277217 |
+-----------+--------------------+
1 row in set (41.11 sec)
Read 6000000 rows, 157.95 MiB in 41.090 sec., 146.02 thousand rows/sec., 3.84 MiB/sec.

XOR filter is ~2X faster and scans less data(only need to scan xor filter index):

mysql> explain select * from t10b where c2='0.4091273956277217';
+-----------------------------------------------------------------------------+
| explain                                                                     |
+-----------------------------------------------------------------------------+
| TableScan                                                                   |
| ├── table: default.bloom.t10b                                               |
| ├── read rows: 6000000                                                      |
| ├── read bytes: 126715443                                                   |
| ├── partitions total: 1000                                                  |
| ├── partitions scanned: 6                                                   |
| └── push downs: [filters: [(c2 = '0.4091273956277217')], limit: NONE]       |
+-----------------------------------------------------------------------------+

With #7893 would be great for the point query.

Data size:

XOR index size(This is the worst case, because all my keys are unique):

[BohuTANG]

👍