[Feature] Lightweight schema change of add/drop column

[Lchangliang]

Search before asking

• I had searched in the issues and found no similar issues.

Description

Background

Add/drop column are heavy operators. They will do linkedSchemaChange or will copy data when data in s3. When add/drop column frequently, a lot of time is wasted waiting. So we need a new way to optimize the process.

Improvement

This improvement involves three aspects, read, writer, compaction. In original impl, BE will hold the tablet schema, set unique id for each column. When read/writer/compaction, BE can get the schema from tablet meta. The core of the modification is
that get the schema from FE when read/writer. And Every rowset will hold its schema. Using the newest schema when doing compaction.

Modification

1. Generate Unique ID by FE.
2. When reading/inserting data, FE will send the newest schema to BE.
3. When inserting, BE will persistent the schema with rowset meta.
4. When doing compaction, BE will choose newest schema from compation rowsets and make it persistent with new rowset meta after compaction.
5. The improvement is only acting on add/drop value. If add/drop key, it will be done by the old way.
6. It will compatible with old table. Old table is mean that system already has tables before the upgrade. But old table will always do the change by old way although the column is value.

Result

When add/drop value column, they will be lightweight operators. They don't need rewrite the data and complete quickly.

Use case

No response

Related issues

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Lchangliang]

TODO:

1. FE synchronization unique_id from BE for old table.
2. optimize that too more tablet_schemas are in memory.
3. About flink connector. Support Light Schema Change and optimize that remove data use streaming load must need head.