[Opt](orc)Optimize the merge io when orc reader read multiple tiny stripes. #42004
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
clang-tidy review says "All clean, LGTM! 👍" |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| @@ -109,11 +110,13 @@ static constexpr char EMPTY_STRING_FOR_OVERFLOW[ColumnString::MAX_STRINGS_OVERFL | |||
| M(TypeIndex::Float64, Float64, orc::DoubleVectorBatch) | |||
|
|
|||
| void ORCFileInputStream::read(void* buf, uint64_t length, uint64_t offset) { | |||
| read_impl(reinterpret_cast<char*>(buf), length, offset); | |||
| } | |||
| void ORCFileInputStream::read_impl(char* out, uint64_t length, uint64_t offset) { | |||
There was a problem hiding this comment.
warning: method 'read_impl' can be made const [readability-make-member-function-const]
Suggested change
| void ORCFileInputStream::read_impl(char* out, uint64_t length, uint64_t offset) { | |
| void ORCFileInputStream::read_impl(char* out, uint64_t length, uint64_t offset) const { |
be/src/vec/exec/format/orc/vorc_reader.h:662:
- ;
+ const;
|
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
|
clang-tidy review says "All clean, LGTM! 👍" |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
hubgeter
force-pushed
the
orc_merge_io
branch
from
bcbe521 to
0b64a8f
Compare
|
clang-tidy review says "All clean, LGTM! 👍" |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
hubgeter
changed the title
morningman
reviewed
Oct 21, 2024
| auto* orcInputStreamPtr = static_cast<ORCFileInputStream*>(_reader->getStream()); | ||
| orcInputStreamPtr->set_all_tiny_stripes(); | ||
| auto& orc_file_reader = orcInputStreamPtr->get_file_reader(); | ||
| orc_file_reader->collect_profile_before_close(); |
There was a problem hiding this comment.
Because before this, the reader needs to read the footer of the orc file
There was a problem hiding this comment.
I suddenly realized that I didn't need to call this thing
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
Outdated
Show resolved
Hide resolved
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
Outdated
Show resolved
Hide resolved
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
Outdated
Show resolved
Hide resolved
kaka11chen
reviewed
Oct 21, 2024
hubgeter
force-pushed
the
orc_merge_io
branch
2 times, most recently
from
b4a1e02 to
e7e9235
Compare
|
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
|
clang-tidy review says "All clean, LGTM! 👍" |
|
run buildall |
|
TeamCity be ut coverage result: |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
morningman
pushed a commit
to morningman/doris
that referenced
this pull request
Nov 7, 2024
…ripes. (apache#42004) When reading orc files, we may encounter a scenario where the stripe byte size is very small but the number of stripes is very large. This pr introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading for the above scenarios. If a stripe byte size is less than `orc_tiny_stripe_threshold_bytes`, we will consider it as a tiny stripe. For multiple tiny stripes, we will perform IO merge reading according to the `orc_once_max_read_bytes` and `orc_max_merge_distance_bytes` parameters. Among them, `orc_once_max_read_bytes` indicates the maximum size of the merged IO. You should not set `orc_once_max_read_bytes` less than `orc_tiny_stripe_threshold_bytes`, although we will not force an error. When using tiny stripe reading optimization, since tiny stripes are not necessarily continuous, when the distance between two tiny stripes is greater than `orc_max_merge_distance_bytes`, we will not merge them into one IO. If you don't want to use this optimization, you can `set orc_tiny_stripe_threshold_bytes = 0`. Default parameters: ```mysql orc_tiny_stripe_threshold_bytes = 8388608 (8M) orc_once_max_read_bytes = 8388608 (8M) orc_max_merge_distance_bytes = 1048576 (1M) ``` We also add relevant profiles for this purpose so that parameters can be adjusted to optimize reading. `RangeCacheFileReader`: 1. `CacheRefreshCount`: how many IOs are merged 2. `ReadToCacheBytes`: how much data is actually read after merging 3. `ReadToCacheTime`: how long it takes to read data after merging 4. `RequestBytes`: how many bytes does the apache-orc library actually need to read the orc file 5. `RequestIO`: how many times the apache-orc library calls this read interface 6. `RequestTime`: how long it takes the apache-orc library to call this read interface It should be noted that `RangeCacheFileReader` is a wrapper of the reader that actually reads data, such as the hdfs reader, so strictly speaking, `CacheRefreshCount` is not equal to how many IOs are initiated to hdfs, because each time the hdfs reader is requested, the hdfs reader may not be able to read all the data at once. This pr also involves changes to the apache-orc third-party library: apache/doris-thirdparty#244. Reference implementation: https://github.com/trinodb/trino/blob/master/lib/trino-orc/src/main/java/io/trino/orc/OrcDataSourceUtils.java#L36 ```mysql set orc_tiny_stripe_threshold_bytes = xxx; set orc_once_max_read_bytes = xxx; set orc_max_merge_distance_bytes = xxx; ``` Introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading of scenarios where the orc stripe byte size is very small but the number of stripes is very large. Co-authored-by: kaka11chen <kaka11.chen@gmail.com> Co-authored-by: daidai <changyuwei@selectdb.com>
morningman
pushed a commit
to morningman/doris
that referenced
this pull request
Nov 9, 2024
…ripes. (apache#42004) When reading orc files, we may encounter a scenario where the stripe byte size is very small but the number of stripes is very large. This pr introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading for the above scenarios. If a stripe byte size is less than `orc_tiny_stripe_threshold_bytes`, we will consider it as a tiny stripe. For multiple tiny stripes, we will perform IO merge reading according to the `orc_once_max_read_bytes` and `orc_max_merge_distance_bytes` parameters. Among them, `orc_once_max_read_bytes` indicates the maximum size of the merged IO. You should not set `orc_once_max_read_bytes` less than `orc_tiny_stripe_threshold_bytes`, although we will not force an error. When using tiny stripe reading optimization, since tiny stripes are not necessarily continuous, when the distance between two tiny stripes is greater than `orc_max_merge_distance_bytes`, we will not merge them into one IO. If you don't want to use this optimization, you can `set orc_tiny_stripe_threshold_bytes = 0`. Default parameters: ```mysql orc_tiny_stripe_threshold_bytes = 8388608 (8M) orc_once_max_read_bytes = 8388608 (8M) orc_max_merge_distance_bytes = 1048576 (1M) ``` We also add relevant profiles for this purpose so that parameters can be adjusted to optimize reading. `RangeCacheFileReader`: 1. `CacheRefreshCount`: how many IOs are merged 2. `ReadToCacheBytes`: how much data is actually read after merging 3. `ReadToCacheTime`: how long it takes to read data after merging 4. `RequestBytes`: how many bytes does the apache-orc library actually need to read the orc file 5. `RequestIO`: how many times the apache-orc library calls this read interface 6. `RequestTime`: how long it takes the apache-orc library to call this read interface It should be noted that `RangeCacheFileReader` is a wrapper of the reader that actually reads data, such as the hdfs reader, so strictly speaking, `CacheRefreshCount` is not equal to how many IOs are initiated to hdfs, because each time the hdfs reader is requested, the hdfs reader may not be able to read all the data at once. This pr also involves changes to the apache-orc third-party library: apache/doris-thirdparty#244. Reference implementation: https://github.com/trinodb/trino/blob/master/lib/trino-orc/src/main/java/io/trino/orc/OrcDataSourceUtils.java#L36 ```mysql set orc_tiny_stripe_threshold_bytes = xxx; set orc_once_max_read_bytes = xxx; set orc_max_merge_distance_bytes = xxx; ``` Introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading of scenarios where the orc stripe byte size is very small but the number of stripes is very large. Co-authored-by: kaka11chen <kaka11.chen@gmail.com> Co-authored-by: daidai <changyuwei@selectdb.com>
morningman
added
dev/3.0.3-merged
and removed
dev/3.0.x
cherry-pick-conflict-in-3.0
labels
morningman
pushed a commit
to morningman/doris
that referenced
this pull request
Nov 18, 2024
…ripes. (apache#42004) ### What problem does this PR solve? When reading orc files, we may encounter a scenario where the stripe byte size is very small but the number of stripes is very large. This pr introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading for the above scenarios. If a stripe byte size is less than `orc_tiny_stripe_threshold_bytes`, we will consider it as a tiny stripe. For multiple tiny stripes, we will perform IO merge reading according to the `orc_once_max_read_bytes` and `orc_max_merge_distance_bytes` parameters. Among them, `orc_once_max_read_bytes` indicates the maximum size of the merged IO. You should not set `orc_once_max_read_bytes` less than `orc_tiny_stripe_threshold_bytes`, although we will not force an error. When using tiny stripe reading optimization, since tiny stripes are not necessarily continuous, when the distance between two tiny stripes is greater than `orc_max_merge_distance_bytes`, we will not merge them into one IO. If you don't want to use this optimization, you can `set orc_tiny_stripe_threshold_bytes = 0`. Default parameters: ```mysql orc_tiny_stripe_threshold_bytes = 8388608 (8M) orc_once_max_read_bytes = 8388608 (8M) orc_max_merge_distance_bytes = 1048576 (1M) ``` We also add relevant profiles for this purpose so that parameters can be adjusted to optimize reading. `RangeCacheFileReader`: 1. `CacheRefreshCount`: how many IOs are merged 2. `ReadToCacheBytes`: how much data is actually read after merging 3. `ReadToCacheTime`: how long it takes to read data after merging 4. `RequestBytes`: how many bytes does the apache-orc library actually need to read the orc file 5. `RequestIO`: how many times the apache-orc library calls this read interface 6. `RequestTime`: how long it takes the apache-orc library to call this read interface It should be noted that `RangeCacheFileReader` is a wrapper of the reader that actually reads data, such as the hdfs reader, so strictly speaking, `CacheRefreshCount` is not equal to how many IOs are initiated to hdfs, because each time the hdfs reader is requested, the hdfs reader may not be able to read all the data at once. This pr also involves changes to the apache-orc third-party library: apache/doris-thirdparty#244. Reference implementation: https://github.com/trinodb/trino/blob/master/lib/trino-orc/src/main/java/io/trino/orc/OrcDataSourceUtils.java#L36 #### Summary: ```mysql set orc_tiny_stripe_threshold_bytes = xxx; set orc_once_max_read_bytes = xxx; set orc_max_merge_distance_bytes = xxx; # xxx is the size in bytes ``` ### Release note Introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading of scenarios where the orc stripe byte size is very small but the number of stripes is very large. Co-authored-by: kaka11chen <kaka11.chen@gmail.com> Co-authored-by: daidai <changyuwei@selectdb.com>
hubgeter
added a commit
to hubgeter/doris
that referenced
this pull request
Nov 19, 2024
…ripes. (apache#42004) ### What problem does this PR solve? When reading orc files, we may encounter a scenario where the stripe byte size is very small but the number of stripes is very large. This pr introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading for the above scenarios. If a stripe byte size is less than `orc_tiny_stripe_threshold_bytes`, we will consider it as a tiny stripe. For multiple tiny stripes, we will perform IO merge reading according to the `orc_once_max_read_bytes` and `orc_max_merge_distance_bytes` parameters. Among them, `orc_once_max_read_bytes` indicates the maximum size of the merged IO. You should not set `orc_once_max_read_bytes` less than `orc_tiny_stripe_threshold_bytes`, although we will not force an error. When using tiny stripe reading optimization, since tiny stripes are not necessarily continuous, when the distance between two tiny stripes is greater than `orc_max_merge_distance_bytes`, we will not merge them into one IO. If you don't want to use this optimization, you can `set orc_tiny_stripe_threshold_bytes = 0`. Default parameters: ```mysql orc_tiny_stripe_threshold_bytes = 8388608 (8M) orc_once_max_read_bytes = 8388608 (8M) orc_max_merge_distance_bytes = 1048576 (1M) ``` We also add relevant profiles for this purpose so that parameters can be adjusted to optimize reading. `RangeCacheFileReader`: 1. `CacheRefreshCount`: how many IOs are merged 2. `ReadToCacheBytes`: how much data is actually read after merging 3. `ReadToCacheTime`: how long it takes to read data after merging 4. `RequestBytes`: how many bytes does the apache-orc library actually need to read the orc file 5. `RequestIO`: how many times the apache-orc library calls this read interface 6. `RequestTime`: how long it takes the apache-orc library to call this read interface It should be noted that `RangeCacheFileReader` is a wrapper of the reader that actually reads data, such as the hdfs reader, so strictly speaking, `CacheRefreshCount` is not equal to how many IOs are initiated to hdfs, because each time the hdfs reader is requested, the hdfs reader may not be able to read all the data at once. This pr also involves changes to the apache-orc third-party library: apache/doris-thirdparty#244. Reference implementation: https://github.com/trinodb/trino/blob/master/lib/trino-orc/src/main/java/io/trino/orc/OrcDataSourceUtils.java#L36 #### Summary: ```mysql set orc_tiny_stripe_threshold_bytes = xxx; set orc_once_max_read_bytes = xxx; set orc_max_merge_distance_bytes = xxx; # xxx is the size in bytes ``` ### Release note Introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading of scenarios where the orc stripe byte size is very small but the number of stripes is very large. Co-authored-by: kaka11chen <kaka11.chen@gmail.com> Co-authored-by: daidai <changyuwei@selectdb.com>
hubgeter
added a commit
to hubgeter/doris
that referenced
this pull request
Nov 19, 2024
…ripes. (apache#42004) ### What problem does this PR solve? When reading orc files, we may encounter a scenario where the stripe byte size is very small but the number of stripes is very large. This pr introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading for the above scenarios. If a stripe byte size is less than `orc_tiny_stripe_threshold_bytes`, we will consider it as a tiny stripe. For multiple tiny stripes, we will perform IO merge reading according to the `orc_once_max_read_bytes` and `orc_max_merge_distance_bytes` parameters. Among them, `orc_once_max_read_bytes` indicates the maximum size of the merged IO. You should not set `orc_once_max_read_bytes` less than `orc_tiny_stripe_threshold_bytes`, although we will not force an error. When using tiny stripe reading optimization, since tiny stripes are not necessarily continuous, when the distance between two tiny stripes is greater than `orc_max_merge_distance_bytes`, we will not merge them into one IO. If you don't want to use this optimization, you can `set orc_tiny_stripe_threshold_bytes = 0`. Default parameters: ```mysql orc_tiny_stripe_threshold_bytes = 8388608 (8M) orc_once_max_read_bytes = 8388608 (8M) orc_max_merge_distance_bytes = 1048576 (1M) ``` We also add relevant profiles for this purpose so that parameters can be adjusted to optimize reading. `RangeCacheFileReader`: 1. `CacheRefreshCount`: how many IOs are merged 2. `ReadToCacheBytes`: how much data is actually read after merging 3. `ReadToCacheTime`: how long it takes to read data after merging 4. `RequestBytes`: how many bytes does the apache-orc library actually need to read the orc file 5. `RequestIO`: how many times the apache-orc library calls this read interface 6. `RequestTime`: how long it takes the apache-orc library to call this read interface It should be noted that `RangeCacheFileReader` is a wrapper of the reader that actually reads data, such as the hdfs reader, so strictly speaking, `CacheRefreshCount` is not equal to how many IOs are initiated to hdfs, because each time the hdfs reader is requested, the hdfs reader may not be able to read all the data at once. This pr also involves changes to the apache-orc third-party library: apache/doris-thirdparty#244. Reference implementation: https://github.com/trinodb/trino/blob/master/lib/trino-orc/src/main/java/io/trino/orc/OrcDataSourceUtils.java#L36 #### Summary: ```mysql set orc_tiny_stripe_threshold_bytes = xxx; set orc_once_max_read_bytes = xxx; set orc_max_merge_distance_bytes = xxx; # xxx is the size in bytes ``` ### Release note Introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading of scenarios where the orc stripe byte size is very small but the number of stripes is very large. Co-authored-by: kaka11chen <kaka11.chen@gmail.com> Co-authored-by: daidai <changyuwei@selectdb.com>
16 tasks
hubgeter
added a commit
to hubgeter/doris
that referenced
this pull request
Nov 20, 2024
…ripes. (apache#42004) ### What problem does this PR solve? When reading orc files, we may encounter a scenario where the stripe byte size is very small but the number of stripes is very large. This pr introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading for the above scenarios. If a stripe byte size is less than `orc_tiny_stripe_threshold_bytes`, we will consider it as a tiny stripe. For multiple tiny stripes, we will perform IO merge reading according to the `orc_once_max_read_bytes` and `orc_max_merge_distance_bytes` parameters. Among them, `orc_once_max_read_bytes` indicates the maximum size of the merged IO. You should not set `orc_once_max_read_bytes` less than `orc_tiny_stripe_threshold_bytes`, although we will not force an error. When using tiny stripe reading optimization, since tiny stripes are not necessarily continuous, when the distance between two tiny stripes is greater than `orc_max_merge_distance_bytes`, we will not merge them into one IO. If you don't want to use this optimization, you can `set orc_tiny_stripe_threshold_bytes = 0`. Default parameters: ```mysql orc_tiny_stripe_threshold_bytes = 8388608 (8M) orc_once_max_read_bytes = 8388608 (8M) orc_max_merge_distance_bytes = 1048576 (1M) ``` We also add relevant profiles for this purpose so that parameters can be adjusted to optimize reading. `RangeCacheFileReader`: 1. `CacheRefreshCount`: how many IOs are merged 2. `ReadToCacheBytes`: how much data is actually read after merging 3. `ReadToCacheTime`: how long it takes to read data after merging 4. `RequestBytes`: how many bytes does the apache-orc library actually need to read the orc file 5. `RequestIO`: how many times the apache-orc library calls this read interface 6. `RequestTime`: how long it takes the apache-orc library to call this read interface It should be noted that `RangeCacheFileReader` is a wrapper of the reader that actually reads data, such as the hdfs reader, so strictly speaking, `CacheRefreshCount` is not equal to how many IOs are initiated to hdfs, because each time the hdfs reader is requested, the hdfs reader may not be able to read all the data at once. This pr also involves changes to the apache-orc third-party library: apache/doris-thirdparty#244. Reference implementation: https://github.com/trinodb/trino/blob/master/lib/trino-orc/src/main/java/io/trino/orc/OrcDataSourceUtils.java#L36 #### Summary: ```mysql set orc_tiny_stripe_threshold_bytes = xxx; set orc_once_max_read_bytes = xxx; set orc_max_merge_distance_bytes = xxx; # xxx is the size in bytes ``` ### Release note Introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading of scenarios where the orc stripe byte size is very small but the number of stripes is very large. Co-authored-by: kaka11chen <kaka11.chen@gmail.com> Co-authored-by: daidai <changyuwei@selectdb.com>
hubgeter
added a commit
to hubgeter/doris
that referenced
this pull request
Nov 21, 2024
…ripes. (apache#42004) ### What problem does this PR solve? When reading orc files, we may encounter a scenario where the stripe byte size is very small but the number of stripes is very large. This pr introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading for the above scenarios. If a stripe byte size is less than `orc_tiny_stripe_threshold_bytes`, we will consider it as a tiny stripe. For multiple tiny stripes, we will perform IO merge reading according to the `orc_once_max_read_bytes` and `orc_max_merge_distance_bytes` parameters. Among them, `orc_once_max_read_bytes` indicates the maximum size of the merged IO. You should not set `orc_once_max_read_bytes` less than `orc_tiny_stripe_threshold_bytes`, although we will not force an error. When using tiny stripe reading optimization, since tiny stripes are not necessarily continuous, when the distance between two tiny stripes is greater than `orc_max_merge_distance_bytes`, we will not merge them into one IO. If you don't want to use this optimization, you can `set orc_tiny_stripe_threshold_bytes = 0`. Default parameters: ```mysql orc_tiny_stripe_threshold_bytes = 8388608 (8M) orc_once_max_read_bytes = 8388608 (8M) orc_max_merge_distance_bytes = 1048576 (1M) ``` We also add relevant profiles for this purpose so that parameters can be adjusted to optimize reading. `RangeCacheFileReader`: 1. `CacheRefreshCount`: how many IOs are merged 2. `ReadToCacheBytes`: how much data is actually read after merging 3. `ReadToCacheTime`: how long it takes to read data after merging 4. `RequestBytes`: how many bytes does the apache-orc library actually need to read the orc file 5. `RequestIO`: how many times the apache-orc library calls this read interface 6. `RequestTime`: how long it takes the apache-orc library to call this read interface It should be noted that `RangeCacheFileReader` is a wrapper of the reader that actually reads data, such as the hdfs reader, so strictly speaking, `CacheRefreshCount` is not equal to how many IOs are initiated to hdfs, because each time the hdfs reader is requested, the hdfs reader may not be able to read all the data at once. This pr also involves changes to the apache-orc third-party library: apache/doris-thirdparty#244. Reference implementation: https://github.com/trinodb/trino/blob/master/lib/trino-orc/src/main/java/io/trino/orc/OrcDataSourceUtils.java#L36 #### Summary: ```mysql set orc_tiny_stripe_threshold_bytes = xxx; set orc_once_max_read_bytes = xxx; set orc_max_merge_distance_bytes = xxx; # xxx is the size in bytes ``` ### Release note Introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading of scenarios where the orc stripe byte size is very small but the number of stripes is very large. Co-authored-by: kaka11chen <kaka11.chen@gmail.com> Co-authored-by: daidai <changyuwei@selectdb.com>
hubgeter
added a commit
to hubgeter/doris
that referenced
this pull request
Nov 21, 2024
…ripes. (apache#42004) ### What problem does this PR solve? When reading orc files, we may encounter a scenario where the stripe byte size is very small but the number of stripes is very large. This pr introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading for the above scenarios. If a stripe byte size is less than `orc_tiny_stripe_threshold_bytes`, we will consider it as a tiny stripe. For multiple tiny stripes, we will perform IO merge reading according to the `orc_once_max_read_bytes` and `orc_max_merge_distance_bytes` parameters. Among them, `orc_once_max_read_bytes` indicates the maximum size of the merged IO. You should not set `orc_once_max_read_bytes` less than `orc_tiny_stripe_threshold_bytes`, although we will not force an error. When using tiny stripe reading optimization, since tiny stripes are not necessarily continuous, when the distance between two tiny stripes is greater than `orc_max_merge_distance_bytes`, we will not merge them into one IO. If you don't want to use this optimization, you can `set orc_tiny_stripe_threshold_bytes = 0`. Default parameters: ```mysql orc_tiny_stripe_threshold_bytes = 8388608 (8M) orc_once_max_read_bytes = 8388608 (8M) orc_max_merge_distance_bytes = 1048576 (1M) ``` We also add relevant profiles for this purpose so that parameters can be adjusted to optimize reading. `RangeCacheFileReader`: 1. `CacheRefreshCount`: how many IOs are merged 2. `ReadToCacheBytes`: how much data is actually read after merging 3. `ReadToCacheTime`: how long it takes to read data after merging 4. `RequestBytes`: how many bytes does the apache-orc library actually need to read the orc file 5. `RequestIO`: how many times the apache-orc library calls this read interface 6. `RequestTime`: how long it takes the apache-orc library to call this read interface It should be noted that `RangeCacheFileReader` is a wrapper of the reader that actually reads data, such as the hdfs reader, so strictly speaking, `CacheRefreshCount` is not equal to how many IOs are initiated to hdfs, because each time the hdfs reader is requested, the hdfs reader may not be able to read all the data at once. This pr also involves changes to the apache-orc third-party library: apache/doris-thirdparty#244. Reference implementation: https://github.com/trinodb/trino/blob/master/lib/trino-orc/src/main/java/io/trino/orc/OrcDataSourceUtils.java#L36 #### Summary: ```mysql set orc_tiny_stripe_threshold_bytes = xxx; set orc_once_max_read_bytes = xxx; set orc_max_merge_distance_bytes = xxx; # xxx is the size in bytes ``` ### Release note Introduces three session variables `orc_tiny_stripe_threshold_bytes`, `orc_once_max_read_bytes`, and `orc_max_merge_distance_bytes` to optimize io reading of scenarios where the orc stripe byte size is very small but the number of stripes is very large. Co-authored-by: kaka11chen <kaka11.chen@gmail.com> Co-authored-by: daidai <changyuwei@selectdb.com>
morningman
added
dev/2.1.8-merged
and removed
cherry-pick-conflict-in-2.1.x
dev/2.1.x
labels
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
When reading orc files, we may encounter a scenario where the stripe byte size is very small but the number of stripes is very large.
This pr introduces three session variables
orc_tiny_stripe_threshold_bytes,orc_once_max_read_bytes, andorc_max_merge_distance_bytesto optimize io reading for the above scenarios.If a stripe byte size is less than
orc_tiny_stripe_threshold_bytes, we will consider it as a tiny stripe. For multiple tiny stripes, we will perform IO merge reading according to theorc_once_max_read_bytesandorc_max_merge_distance_bytesparameters. Among them,orc_once_max_read_bytesindicates the maximum size of the merged IO. You should not setorc_once_max_read_bytesless thanorc_tiny_stripe_threshold_bytes, although we will not force an error. When using tiny stripe reading optimization, since tiny stripes are not necessarily continuous, when the distance between two tiny stripes is greater thanorc_max_merge_distance_bytes, we will not merge them into one IO.If you don't want to use this optimization, you can
set orc_tiny_stripe_threshold_bytes = 0.Default parameters:
We also add relevant profiles for this purpose so that parameters can be adjusted to optimize reading.
RangeCacheFileReader:CacheRefreshCount: how many IOs are mergedReadToCacheBytes: how much data is actually read after mergingReadToCacheTime: how long it takes to read data after mergingRequestBytes: how many bytes does the apache-orc library actually need to read the orc fileRequestIO: how many times the apache-orc library calls this read interfaceRequestTime: how long it takes the apache-orc library to call this read interfaceIt should be noted that
RangeCacheFileReaderis a wrapper of the reader that actually reads data, such as the hdfs reader, so strictly speaking,CacheRefreshCountis not equal to how many IOs are initiated to hdfs, because each time the hdfs reader is requested, the hdfs reader may not be able to read all the data at once.This pr also involves changes to the apache-orc third-party library: apache/doris-thirdparty#244.
Reference implementation: https://github.com/trinodb/trino/blob/master/lib/trino-orc/src/main/java/io/trino/orc/OrcDataSourceUtils.java#L36
Summary:
Check List (For Committer)
Test
Behavior changed:
Does this need documentation?
doing...
Release note
Introduces three session variables
orc_tiny_stripe_threshold_bytes,orc_once_max_read_bytes, andorc_max_merge_distance_bytesto optimize io reading of scenarios where the orc stripe byte size is very small but the number of stripes is very large.Check List (For Reviewer who merge this PR)