-
Notifications
You must be signed in to change notification settings - Fork 5.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ddl, lightning: integrate operator for global sort #46734
Merged
Merged
Changes from 30 commits
Commits
Show all changes
35 commits
Select commit
Hold shift + click to select a range
b82492e
lightning: add ExternalEngineConfig to create an external engine
lance6716 ba62aee
add two more config
lance6716 fc7e903
fix UT
lance6716 3a5da7a
fix bazel
lance6716 1ede83e
fix lint
lance6716 0c60d4c
add UT
lance6716 faae105
fix sec lint
lance6716 549d0da
fix bazel
lance6716 2c6257b
address comment
lance6716 ea7494d
have to use failpoint
lance6716 7f7bef5
fix CI
lance6716 4a89257
address comment
lance6716 f6de222
switch to failpoint
lance6716 820326d
Merge branch 'master' of github.com:pingcap/tidb into external-config
lance6716 83ec65d
ddl, lightning: integrate operator for global sort
tangenta 6f098c4
update bazel
tangenta be3e412
use jobID as external storage prefix
tangenta feb25b1
Merge remote-tracking branch 'upstream/master' into global-sort-operator
tangenta 8791caa
fix file prefix and linter
tangenta e093425
fix linter
tangenta 7ecb83f
use the same writer for each index
tangenta c7fee41
fix GetRegionSplitSizeKeys
tangenta 37d9fd4
Merge remote-tracking branch 'upstream/master' into HEAD
tangenta 45d24c9
use one write worker for global sort
tangenta c06da77
remove unnecessary changes
tangenta 395bd5f
unregister backend ctx after merge sort
tangenta 7849e2e
ddl: get all file names from subtask meta
tangenta ba083ae
address comments
tangenta 5aa5e09
Merge remote-tracking branch 'upstream/master' into HEAD
tangenta 5c209b5
apply tidb_cloud_storage_uri
tangenta 56a7378
Merge remote-tracking branch 'upstream/master' into HEAD
tangenta 271d1fd
fix (#127)
ywqzzy d9e6bce
Merge remote-tracking branch 'upstream/master' into HEAD
tangenta a19bd15
collect min max key correctly
tangenta 5ab3433
add comment
tangenta File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why we uses a pointer to interface as return type? @lance6716
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
R can be a struct who implements sortedReader[T], not an interface. For example, R is
kvReaderProxy
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ugly to have pointers to interface
we should unify using one way. and in this case, the implementer should know the result doing it, i.e. the struct might be copied around.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I remember one problem is the zero value of R maybe ready to use. so I need nil to distinguish them. will check it tomorrow.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can change
*R
toR
later, and also letmergeIter
usereaders []R, readerClosed []bool
instead ofreaders []*R
. Because R may not be "comparable"