Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement](MaxCompute)Refactoring maxCompute catalog using Storage API.(#40225 , #40888 ,#41386 ) #41610

Merged
merged 6 commits into from
Oct 11, 2024
Merged

[Enhancement](MaxCompute)Refactoring maxCompute catalog using Storage API.(#40225 , #40888 ,#41386 ) #41610

merged 6 commits into from
Oct 11, 2024

Conversation

hubgeter
Copy link
Contributor

@hubgeter hubgeter commented Oct 9, 2024 •
edited
Loading

bp #40225 , #40888 ,#41386

Proposed changes

Among them, #40225 is the new api of mc,
#40888 is used to fix the bug when reading null between the new and old apis,
#41386 is used for compatibility between the new and old versions

@hubgeter
[Enhancement](MaxCompute)Refactoring maxCompute catalog using Storage…
1c00e65 
… API. (apache#40225)

Refactoring maxCompute catalog using Storage API.

Storage API :
https://help.aliyun.com/zh/maxcompute/user-guide/open-storage-sample-java-sdk?spm=a2c4g.11186623.0.i0

```
The following are required:

CREATE CATALOG mc PROPERTIES (
"type" = "max_compute",
"mc.default.project" = "xxx",
"mc.access_key" = "xxx",
"mc.secret_key" = "xxxx",
"mc.endpoint" = "xxxx"
);

Optional parameters:
Configuration Item      Default Value
"mc.quota"              =    "pay-as-you-go"

"mc.split_strategy" =   "byte_size" Split according to file size
"mc.split_byte_size" = "268435456" You can set the file size of each split

"mc.split_strategy" = "row_count" Split according to the number of rows of data
"mc.split_row_count" = "1048576" You can set how many lines to read for each split
```
@hubgeter
[fix](maxCompute)Fixed the bug that extra nulls are read when reading…
bff8d73 
… maxcompute. (apache#40888)

before pr apache#40225 
## Proposed changes

Fixed a bug where when reading maxcompute, if there are null values ​​in
a batch, null values ​​will always be read out.
@hubgeter
[Enhancement](maxCompute)Implement compatibility with existing maxcom…
d212a98 
…pute catalogs from previous versions. (apache#41386)

before pr apache#40225
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@hubgeter
Copy link
Contributor Author

hubgeter commented Oct 9, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.09% (9340/25879)
Line Coverage: 27.66% (76725/277358)
Region Coverage: 26.44% (39376/148902)
Branch Coverage: 23.22% (20028/86240)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d212a980bf517b398c0ad5dbbf2e7d3aa0250ae0_d212a980bf517b398c0ad5dbbf2e7d3aa0250ae0/report/index.html

@hubgeter hubgeter force-pushed the pick_21_mc_storage_api branch from 68a2fee to d1961ff Compare October 10, 2024 06:28
@hubgeter
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.08% (9338/25880)
Line Coverage: 27.66% (76710/277367)
Region Coverage: 26.44% (39371/148905)
Branch Coverage: 23.23% (20030/86242)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d1961ff1b8f71a1956a893230583df6bff8df741_d1961ff1b8f71a1956a893230583df6bff8df741/report/index.html

@hubgeter hubgeter force-pushed the pick_21_mc_storage_api branch from d1961ff to a5dcbfd Compare October 10, 2024 09:09
@hubgeter
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.09% (9340/25880)
Line Coverage: 27.65% (76690/277367)
Region Coverage: 26.44% (39377/148905)
Branch Coverage: 23.23% (20031/86242)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a5dcbfd692ed333f5a87901f4f64159abfe66baa_a5dcbfd692ed333f5a87901f4f64159abfe66baa/report/index.html

@morningman morningman merged commit 8c0f73c into apache:branch-2.1 Oct 11, 2024
19 of 21 checks passed
@yiguolei yiguolei mentioned this pull request Nov 6, 2024
Open
hubgeter added a commit to hubgeter/doris that referenced this pull request Nov 20, 2024
@hubgeter
Revert "[Enhancement](MaxCompute)Refactoring maxCompute catalog using…
edab727 
… Storage API.(apache#40225 , apache#40888 ,apache#41386 ) (apache#41610)"

This reverts commit 8c0f73c.
hubgeter added a commit to hubgeter/doris that referenced this pull request Nov 20, 2024
@hubgeter
Revert "[Enhancement](MaxCompute)Refactoring maxCompute catalog using…
b551c95 
… Storage API.(apache#40225 , apache#40888 ,apache#41386 ) (apache#41610)"

This reverts commit 8c0f73c.
@yiguolei yiguolei mentioned this pull request Jan 19, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
meta-change
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@hubgeter @doris-robot @morningman