This repository was archived by the owner on Jun 23, 2022. It is now read-only.
feat(bulk-load): check app bulk load states while sync app from remote storage #564
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The previous pull requests(#553 #559 #563) show how to continue bulk load after meta server start or restarts, which is based on the assumption that the app is executing bulk load process before. However, we should check whether the app is executing bulk load or not firstly. This pull request will do this check(in function
check_app_bulk_load_states
).Before meta server really starts, meta server will sync app info and all partition info from remote storage, when an app and its all partitions' info synced succeed, call function
check_app_bulk_load_states
. This function will check consistency between app bulk_loading flag and app bulk load directory.is_bulk_loading
should be true.is_bulk_loading
is true, it is expected that bulk load directory existed on remote storage, if it is not existed, should setis_bulk_loading
as false.is_bulk_loading
is false, it is expected that bulk load directory not existed on remote storage, if it exists, should remove useless bulk load directory on remote storage.Special explanation:
I don't use
meta_storage
utility in this function. For one hand,check_app_bulk_load_states
should execute in thread poolTHREAD_POOL_META_SERVER
, because meta server is not started. For another hand, this function usenode_exist
interface, this function callback is not an universal case, node not exists is not a wrong case.