Skip to content
This repository was archived by the owner on Jun 23, 2022. It is now read-only.

feat(bulk-load): check app bulk load states while sync app from remote storage #564

Merged
merged 5 commits into from
Jul 21, 2020

Conversation

hycdong
Copy link
Contributor

@hycdong hycdong commented Jul 17, 2020

The previous pull requests(#553 #559 #563) show how to continue bulk load after meta server start or restarts, which is based on the assumption that the app is executing bulk load process before. However, we should check whether the app is executing bulk load or not firstly. This pull request will do this check(in function check_app_bulk_load_states).

Before meta server really starts, meta server will sync app info and all partition info from remote storage, when an app and its all partitions' info synced succeed, call function check_app_bulk_load_states. This function will check consistency between app bulk_loading flag and app bulk load directory.

  • the app is executing bulk load should have bulk load directory on remote storage and its is_bulk_loading should be true.
  • is_bulk_loading is true, it is expected that bulk load directory existed on remote storage, if it is not existed, should set is_bulk_loading as false.
  • is_bulk_loading is false, it is expected that bulk load directory not existed on remote storage, if it exists, should remove useless bulk load directory on remote storage.

Special explanation:
I don't use meta_storage utility in this function. For one hand, check_app_bulk_load_states should execute in thread pool THREAD_POOL_META_SERVER, because meta server is not started. For another hand, this function use node_exist interface, this function callback is not an universal case, node not exists is not a wrong case.

@hycdong hycdong marked this pull request as ready for review July 18, 2020 08:27
acelyc111
acelyc111 previously approved these changes Jul 20, 2020
@hycdong hycdong closed this Jul 21, 2020
@hycdong hycdong reopened this Jul 21, 2020
@hycdong hycdong merged commit d5ba2c4 into XiaoMi:master Jul 21, 2020
@hycdong hycdong deleted the meta_failover_part3 branch July 21, 2020 02:17
hycdong added a commit to hycdong/rdsn that referenced this pull request Jul 21, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants