Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: timeplus-io/proton
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.5.4
Choose a base ref
...
head repository: timeplus-io/proton
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v1.5.5
Choose a head ref
  • 15 commits
  • 176 files changed
  • 6 contributors

Commits on Mar 25, 2024

  1. porting more fixes and improvement for aggregation (#615)

    * Merge pull request #37093 from Clickhouse
    
    Fix GROUP BY AggregateFunction
    
    * Merge pull request #34305 from amosbird/projection-fix27
    
    Fix various issues when projection is enabled by default
    
    * Merge pull request #37469 from azat/projections-optimize_aggregation_in_order
    
    Implement in order aggregation (optimize_aggregation_in_order) for projections for tables with fully materialized projections
    
    * Merge pull request #37163 from ClickHouse/grouping-function
    
    Add GROUPING function
    
    * Merge pull request #39429 from Clickhouse
    
    Improve memory usage during memory efficient merging of aggregation results
    
    Merge pull request #39908 from Clickhouse
    
    Improve 02354_distributed_with_external_aggregation_memory_usage
    
    * Merge pull request #38295 from Clickhouse
    
    Quick fix for aggregation pipeline
    
    * Merge pull request #39138 from nickitat/control_block_size_in_aggregator
    
    Control block size in aggregator
    
    * Merge pull request #39304 from Clickhouse
    
    Prefetching in aggregation
    
    * Merge pull request #40762 from ClickHouse/grouping-comp
    
    Fix GROUPING function SQL compatibility
    
    * Merge pull request #45375 from ClibMouse/window_view_fix
    
    Window view functional test failed on platform s390x
    
    * Merge pull request #45772 from Clickhouse
    
    Optimize one nullable key aggregate performance
    
    * Merge pull request #49282 from loneylee/49280
    
    Fix all key value is null and group use rollup return wrong answer
    
    * Merge pull request #50291 from kitaisreal/jit-aggregation-nullable-key-fix
    
    JIT aggregation nullable key fix
    
    * Merge pull request #52439 from Clickhouse
    
    Fix possible double-free in Aggregator
    
    * Merge pull request #50748 from Clickhouse
    
    Convert hashSets in parallel before merge
    
    * Merge pull request #51566 from Clickhouse
    
    Fix memory consumption when `max_block_size` is huge in Aggregator + more memory checks
    
    * Merge pull request #53589 from kitaisreal/aggregator-merge-states-in-batch
    
    Experiment Aggregator merge and destroy states in batch
    
    * Merge pull request #56120 from kitaisreal/window-functions-decrease-amount-of-virtual-function-calls
    
    WindowTransform decrease amount of virtual function calls
    
    * Merge pull request #57258 from Algunenano/cleanup_interval
    
    Remove addBatchSinglePlaceFromInterval
    
    * Merge pull request #58440 from ClickHouse/kitaisreal-enable-jit-for-aggregation-without-key
    
    Enable JIT compilation for aggregation without key
    
    * Merge pull request #43945 from Clickhouse
    
    Improve hash table preallocation optimisation
    
    * do not preallocate if max_size_to_preallocate_for_aggregation is too small
    
    * skip optimisation for aggr without key
    
    * increase default for max_size_to_preallocate_for_aggregation
    
    * Merge pull request #58745 from Algunenano/i58727
    
    Fix double destroy call on exception throw in addBatchLookupTable8
    
    * Merge pull request #59009 from nickitat/uniq_optimisation_for_distributed
    
    `uniqExact` state parallel merging for distributed queries
    
    * Merge pull request #58542 from ClickHouse/revert-58440-kitaisreal-enable-jit-for-aggregation-without-key
    
    Revert "Merging #53757"
    
    * fix build after merge
    
    * porting functional test and fix more
    yl-lisen authored Mar 25, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    4b004ac View commit details

Commits on Mar 26, 2024

  1. improve checkpoint compatibility (#633)

    yl-lisen authored Mar 26, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    e87fc44 View commit details

Commits on Mar 27, 2024

  1. Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    d559d28 View commit details
  2. add metrics log for join (#3988) (#630)

    yl-lisen authored Mar 27, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    e2d9f19 View commit details

Commits on Mar 29, 2024

  1. Enhancement/issue 632 modify default retention policy for streaming s…

    …tore (#636)
    yl-lisen authored Mar 29, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    4c07a12 View commit details

Commits on Mar 31, 2024

  1. Enhancement/issue 613 control max block size for streaming processing (

    yl-lisen authored Mar 31, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    c19b770 View commit details

Commits on Apr 1, 2024

  1. Use wait instead of sleep for materialized view recovery (#638)

    * Use wait instead of sleep for materialized view recovery
    
    * fix flaky smoke test
    yl-lisen authored Apr 1, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    aa9d940 View commit details

Commits on Apr 5, 2024

  1. feat: new SSL related kafka external stream settings (#635)

    * feat: support top-level setting for skipping cert check for external stream
    * added setting ssl_ca_pem
    zliang-min authored Apr 5, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    b38f2ce View commit details

Commits on Apr 6, 2024

  1. logging enhancement for kafka external stream

    zliang-min authored Apr 6, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    3f14c1d View commit details

Commits on Apr 7, 2024

  1. Kafka consumer pool waits for all consumers being freed before being …

    …destroyed (#643)
    zliang-min authored Apr 7, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    fe2c55f View commit details

Commits on Apr 9, 2024

  1. Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    82ada8f View commit details
  2. Shutdown Kafka consumers and producer when the external stream is sto…

    …pped (#645)
    zliang-min authored Apr 9, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    996c5f3 View commit details

Commits on Apr 10, 2024

  1. skip checkpointing if there is no new data processed (#649)

    Co-authored-by: Ken Chen <zlchen.ken@gmail.com>
    Co-authored-by: yoko <haohang.shi@timeplus.io>
    3 people authored Apr 10, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    a5e87b4 View commit details
  2. sudo proton install to install to /usr/local/bin on Mac instead of …

    …the locked /usr/bin (#651)
    
    * Update Install.cpp
    
    /usr/bin is locked on Mac. Change it to /usr/local/bin. This should also work on Linux
    
    * 2 more usr/bin to usr/local/bin
    
    * minor clean:
    1. increase open files limit. ClickHouse/ClickHouse#41345
    2. use stringview instead char*
    3. maybe better for throw exception
    4. try take a hardlink first before copy. ClickHouse/ClickHouse#48578
    
    * conditional compile
    
    * fix the incorrect usage, the default is usr/bin otherwise only tag for macOS
    
    ---------
    
    Co-authored-by: yoko <haohang.shi@timeplus.io>
    jovezhong and yokofly authored Apr 10, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    9ef372c View commit details
  3. Auto version update to [1.5.5] [9]

    proton-robot committed Apr 10, 2024
    Copy the full SHA
    412bf88 View commit details
Showing with 3,814 additions and 1,511 deletions.
  1. +0 −1 .clang-format
  2. +5 −5 cmake/autogenerated_versions.txt
  3. +2 −2 debian/changelog
  4. +1 −1 docker/client/Dockerfile
  5. +1 −1 docker/server/Dockerfile
  6. +1 −1 docker/test/Dockerfile
  7. +36 −15 programs/install/Install.cpp
  8. +2 −0 programs/server/config.yaml
  9. +1 −1 src/AggregateFunctions/AggregateFunctionCount.h
  10. +1 −0 src/AggregateFunctions/AggregateFunctionSum.h
  11. +38 −1 src/AggregateFunctions/AggregateFunctionUniq.h
  12. +21 −67 src/AggregateFunctions/IAggregateFunction.h
  13. +84 −1 src/AggregateFunctions/UniqExactSet.h
  14. +81 −70 src/Checkpoint/CheckpointCoordinator.cpp
  15. +3 −5 src/Checkpoint/CheckpointCoordinator.h
  16. +56 −26 src/Common/ColumnsHashing.h
  17. +47 −6 src/Common/ColumnsHashingImpl.h
  18. +3 −0 src/Common/ErrorCodes.cpp
  19. +1 −1 src/Common/HashTable/FixedHashMap.h
  20. +38 −17 src/Common/HashTable/HashMap.h
  21. +55 −0 src/Common/HashTable/Prefetching.h
  22. +5 −5 src/Common/HashTable/StringHashMap.h
  23. +5 −3 src/Common/HashTable/TwoLevelHashMap.h
  24. +3 −3 src/Common/HashTable/TwoLevelStringHashMap.h
  25. +2 −2 src/Common/MemoryTracker.cpp
  26. +29 −3 src/Common/PoolBase.h
  27. +2 −0 src/Common/ProtonCommon.h
  28. +15 −11 src/Core/Block.cpp
  29. +5 −0 src/Core/Block.h
  30. +1 −0 src/Core/Names.h
  31. +14 −0 src/Core/NamesAndTypes.cpp
  32. +5 −2 src/Core/Settings.h
  33. +13 −4 src/Functions/array/arrayReduce.cpp
  34. +11 −2 src/Functions/initializeAggregation.cpp
  35. +62 −3 src/Interpreters/ActionsVisitor.cpp
  36. +45 −3 src/Interpreters/ActionsVisitor.h
  37. +602 −346 src/Interpreters/Aggregator.cpp
  38. +207 −95 src/Interpreters/Aggregator.h
  39. +36 −7 src/Interpreters/ExpressionAnalyzer.cpp
  40. +10 −0 src/Interpreters/ExpressionAnalyzer.h
  41. +41 −11 src/Interpreters/InterpreterSelectQuery.cpp
  42. +43 −10 src/Interpreters/JIT/compileFunction.cpp
  43. +2 −2 src/Interpreters/JIT/compileFunction.h
  44. +158 −67 src/Interpreters/Streaming/Aggregator.cpp
  45. +56 −33 src/Interpreters/Streaming/Aggregator.h
  46. +13 −0 src/Interpreters/Streaming/ConcurrentHashJoin.cpp
  47. +1 −0 src/Interpreters/Streaming/ConcurrentHashJoin.h
  48. +12 −8 src/Interpreters/Streaming/HashJoin.cpp
  49. +2 −0 src/Interpreters/Streaming/HashJoin.h
  50. +2 −0 src/Interpreters/Streaming/IHashJoin.h
  51. +1 −1 src/Parsers/ASTProjectionDeclaration.cpp
  52. +11 −1 src/Parsers/ASTProjectionSelectQuery.cpp
  53. +18 −18 src/Parsers/ASTSelectQuery.h
  54. +16 −0 src/Parsers/ExpressionElementParsers.cpp
  55. +1 −0 src/Parsers/ParserProjectionSelectQuery.cpp
  56. +1 −0 src/Processors/Chunk.h
  57. +17 −3 src/Processors/Executors/ExecutingGraph.cpp
  58. +2 −0 src/Processors/Executors/ExecutingGraph.h
  59. +5 −0 src/Processors/Executors/PipelineExecutor.cpp
  60. +2 −0 src/Processors/Executors/PipelineExecutor.h
  61. +36 −18 src/Processors/QueryPlan/AggregatingStep.cpp
  62. +10 −1 src/Processors/QueryPlan/AggregatingStep.h
  63. +23 −3 src/Processors/QueryPlan/CubeStep.cpp
  64. +1 −0 src/Processors/QueryPlan/CubeStep.h
  65. +8 −4 src/Processors/QueryPlan/MergingAggregatedStep.cpp
  66. +6 −1 src/Processors/QueryPlan/MergingAggregatedStep.h
  67. +7 −3 src/Processors/QueryPlan/RollupStep.cpp
  68. +1 −0 src/Processors/QueryPlan/RollupStep.h
  69. +6 −2 src/Processors/QueryPlan/TotalsHavingStep.cpp
  70. +3 −0 src/Processors/QueryPlan/TotalsHavingStep.h
  71. +6 −6 src/Processors/Streaming/ISource.cpp
  72. +12 −5 src/Processors/Streaming/ISource.h
  73. +2 −1 src/Processors/TTL/TTLAggregationAlgorithm.cpp
  74. +41 −9 src/Processors/Transforms/AggregatingInOrderTransform.cpp
  75. +4 −5 src/Processors/Transforms/AggregatingInOrderTransform.h
  76. +22 −27 src/Processors/Transforms/AggregatingTransform.cpp
  77. +6 −3 src/Processors/Transforms/AggregatingTransform.h
  78. +6 −2 src/Processors/Transforms/CubeTransform.cpp
  79. +4 −1 src/Processors/Transforms/CubeTransform.h
  80. +4 −0 src/Processors/Transforms/JoiningTransform.cpp
  81. +10 −2 src/Processors/Transforms/MergingAggregatedMemoryEfficientTransform.cpp
  82. +6 −2 src/Processors/Transforms/RollupTransform.cpp
  83. +5 −1 src/Processors/Transforms/RollupTransform.h
  84. +40 −34 src/Processors/Transforms/Streaming/AggregatingHelper.cpp
  85. +9 −7 src/Processors/Transforms/Streaming/AggregatingHelper.h
  86. +58 −40 src/Processors/Transforms/Streaming/AggregatingTransform.cpp
  87. +6 −5 src/Processors/Transforms/Streaming/AggregatingTransform.h
  88. +61 −34 src/Processors/Transforms/Streaming/AggregatingTransformWithSubstream.cpp
  89. +6 −5 src/Processors/Transforms/Streaming/AggregatingTransformWithSubstream.h
  90. +5 −3 src/Processors/Transforms/Streaming/ChangelogConvertTransform.cpp
  91. +1 −1 src/Processors/Transforms/Streaming/ChangelogConvertTransform.h
  92. +1 −1 src/Processors/Transforms/Streaming/ChangelogTransform.h
  93. +13 −12 src/Processors/Transforms/Streaming/GlobalAggregatingTransform.cpp
  94. +14 −12 src/Processors/Transforms/Streaming/GlobalAggregatingTransformWithSubstream.cpp
  95. +7 −0 src/Processors/Transforms/Streaming/JoinTransform.cpp
  96. +3 −1 src/Processors/Transforms/Streaming/JoinTransform.h
  97. +3 −2 src/Processors/Transforms/Streaming/JoinTransformWithAlignment.cpp
  98. +1 −1 src/Processors/Transforms/Streaming/JoinTransformWithAlignment.h
  99. +8 −4 src/Processors/Transforms/Streaming/UserDefinedEmitStrategyAggregatingTransform.cpp
  100. +8 −4 src/Processors/Transforms/Streaming/UserDefinedEmitStrategyAggregatingTransformWithSubstream.cpp
  101. +10 −6 src/Processors/Transforms/Streaming/WindowAggregatingTransform.cpp
  102. +11 −6 src/Processors/Transforms/Streaming/WindowAggregatingTransformWithSubstream.cpp
  103. +18 −27 src/Processors/Transforms/TotalsHavingTransform.cpp
  104. +4 −3 src/Processors/Transforms/TotalsHavingTransform.h
  105. +13 −5 src/Processors/Transforms/WindowTransform.cpp
  106. +3 −0 src/Processors/Transforms/WindowTransform.h
  107. +32 −0 src/Processors/Transforms/finalizeChunk.cpp
  108. +20 −0 src/Processors/Transforms/finalizeChunk.h
  109. +2 −0 src/Storages/ExternalStream/ExternalStreamSettings.h
  110. +7 −2 src/Storages/ExternalStream/Kafka/Consumer.cpp
  111. +8 −7 src/Storages/ExternalStream/Kafka/Consumer.h
  112. +25 −9 src/Storages/ExternalStream/Kafka/ConsumerPool.h
  113. +49 −18 src/Storages/ExternalStream/Kafka/Kafka.cpp
  114. +31 −8 src/Storages/ExternalStream/Kafka/Kafka.h
  115. +25 −8 src/Storages/ExternalStream/Kafka/KafkaSink.cpp
  116. +2 −3 src/Storages/ExternalStream/Kafka/KafkaSink.h
  117. +53 −47 src/Storages/ExternalStream/Kafka/KafkaSource.cpp
  118. +5 −22 src/Storages/ExternalStream/Kafka/KafkaSource.h
  119. +4 −1 src/Storages/ExternalStream/Kafka/Producer.cpp
  120. +10 −7 src/Storages/ExternalStream/Kafka/Producer.h
  121. +5 −7 src/Storages/ExternalStream/Log/FileLog.cpp
  122. +1 −7 src/Storages/ExternalStream/Log/FileLog.h
  123. +4 −3 src/Storages/ExternalStream/StorageExternalStream.cpp
  124. +2 −1 src/Storages/ExternalStream/StorageExternalStream.h
  125. +28 −0 src/Storages/ExternalStream/StorageExternalStreamImpl.cpp
  126. +13 −1 src/Storages/ExternalStream/StorageExternalStreamImpl.h
  127. +104 −34 src/Storages/MergeTree/MergeTreeData.cpp
  128. +2 −0 src/Storages/MergeTree/MergeTreeData.h
  129. +172 −92 src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp
  130. +2 −2 src/Storages/MergeTree/MergeTreeSettings.h
  131. +9 −2 src/Storages/ProjectionsDescription.cpp
  132. +1 −1 src/Storages/ProjectionsDescription.h
  133. +2 −1 src/Storages/StorageBuffer.cpp
  134. +3 −0 src/Storages/StorageProxy.h
  135. +32 −15 src/Storages/Streaming/StorageMaterializedView.cpp
  136. +6 −0 src/Storages/Streaming/StorageMaterializedView.h
  137. +9 −3 src/Storages/Streaming/StreamingStoreSource.cpp
  138. +10 −6 src/Storages/Streaming/StreamingStoreSourceBase.cpp
  139. +1 −5 src/Storages/Streaming/StreamingStoreSourceBase.h
  140. +1 −1 src/Storages/System/StorageSystemContributors.generated.cpp
  141. +6 −3 src/Storages/VirtualColumnUtils.cpp
  142. +1 −1 tests/integration/test_distributed_directory_monitor_split_batch_on_failure/test.py
  143. +4 −4 tests/integration/test_quota/test.py
  144. +31 −0 tests/performance/group_by_fixed_keys.xml
  145. +49 −0 tests/performance/groupby_onekey_nullable.xml
  146. +32 −0 tests/performance/prefetch_in_aggregation.xml
  147. +4 −0 tests/performance/prepare_hash_before_merge.xml
  148. +11 −0 tests/performance/queries_over_aggregation.xml
  149. +22 −0 tests/performance/uniq_without_key_dist.xml
  150. +9 −10 tests/queries/0_stateless/01572_kill_window_function.sh
  151. +2 −1 tests/queries/0_stateless/02355_control_block_size_in_aggregator.sql
  152. +10 −0 tests/queries/0_stateless/02725_null_group_key_with_rollup.reference
  153. +13 −0 tests/queries/0_stateless/02725_null_group_key_with_rollup.sql
  154. +4 −0 tests/queries/0_stateless/02770_jit_aggregation_nullable_key_fix.reference
  155. +39 −0 tests/queries/0_stateless/02770_jit_aggregation_nullable_key_fix.sql
  156. 0 tests/queries/0_stateless/02797_aggregator_huge_mem_usage_bug.reference
  157. +12 −0 tests/queries/0_stateless/02797_aggregator_huge_mem_usage_bug.sql
  158. 0 tests/queries/0_stateless/02963_msan_agg_addBatchLookupTable8.reference
  159. +2 −0 tests/queries/0_stateless/02963_msan_agg_addBatchLookupTable8.sql
  160. +20 −0 ...supported/0_stateless/AggregatingInOrderTransform/01710_projection_aggregation_in_order.reference
  161. +66 −0 ...s_not_supported/0_stateless/AggregatingInOrderTransform/01710_projection_aggregation_in_order.sql
  162. +4 −0 tests/queries_not_supported/0_stateless/Null/02770_jit_aggregation_nullable_key_fix.reference
  163. +39 −0 tests/queries_not_supported/0_stateless/Null/02770_jit_aggregation_nullable_key_fix.sql
  164. +1 −0 tests/queries_ported/0_stateless/02355_control_block_size_in_aggregator.reference
  165. +10 −0 tests/queries_ported/0_stateless/02355_control_block_size_in_aggregator.sql
  166. +10 −0 tests/queries_ported/0_stateless/02725_null_group_key_with_rollup.reference
  167. +13 −0 tests/queries_ported/0_stateless/02725_null_group_key_with_rollup.sql
  168. 0 tests/queries_ported/0_stateless/02797_aggregator_huge_mem_usage_bug.reference
  169. +12 −0 tests/queries_ported/0_stateless/02797_aggregator_huge_mem_usage_bug.sql
  170. 0 tests/queries_ported/0_stateless/02963_msan_agg_addBatchLookupTable8.reference
  171. +2 −0 tests/queries_ported/0_stateless/02963_msan_agg_addBatchLookupTable8.sql
  172. +2 −0 tests/stream/test_stream_smoke/0013_changelog_stream13.yaml
  173. +4 −4 tests/stream/test_stream_smoke/0018_query_state5_substream.json
  174. +2 −2 tests/stream/test_stream_smoke/0018_query_state6_view.json
  175. +50 −0 tests/stream/test_stream_smoke/0030_two_level_global_aggr.yaml
  176. +163 −0 tests/stream/test_stream_smoke/0098_fixed_issues2.yaml
1 change: 0 additions & 1 deletion .clang-format
Original file line number Diff line number Diff line change
@@ -74,7 +74,6 @@ ConstructorInitializerIndentWidth: 4
ContinuationIndentWidth: 4
DerivePointerAlignment: false
DisableFormat: false
IndentWidth: 4
IndentWrappedFunctionNames: false
MacroBlockBegin: ''
MacroBlockEnd: ''
10 changes: 5 additions & 5 deletions cmake/autogenerated_versions.txt
Original file line number Diff line number Diff line change
@@ -2,11 +2,11 @@

# NOTE: has nothing common with DBMS_TCP_PROTOCOL_VERSION,
# only DBMS_TCP_PROTOCOL_VERSION should be incremented on protocol changes.
SET(VERSION_REVISION 8)
SET(VERSION_REVISION 9)
SET(VERSION_MAJOR 1)
SET(VERSION_MINOR 5)
SET(VERSION_PATCH 4)
SET(VERSION_GITHASH 15be075fdf1031d8fdebf73e567f474679980683)
SET(VERSION_DESCRIBE v1.5.4)
SET(VERSION_STRING 1.5.4)
SET(VERSION_PATCH 5)
SET(VERSION_GITHASH 9ef372cf796359039c9b894f766a5bf33da47591)
SET(VERSION_DESCRIBE v1.5.5)
SET(VERSION_STRING 1.5.5)
# end of autochange
4 changes: 2 additions & 2 deletions debian/changelog
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
proton (1.5.4) unstable; urgency=low
proton (1.5.5) unstable; urgency=low

* Modified source code

-- proton-robot <root@timeplus.io> Mon, 25 Mar 2024 00:48:29 +0000
-- proton-robot <root@timeplus.io> Wed, 10 Apr 2024 15:53:52 +0000
2 changes: 1 addition & 1 deletion docker/client/Dockerfile
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@ ARG apt_archive="http://archive.ubuntu.com"
RUN sed -i "s|http://archive.ubuntu.com|$apt_archive|g" /etc/apt/sources.list

ARG repository="deb https://repo.clickhouse.com/deb/stable/ main/"
ARG version=1.5.4
ARG version=1.5.5


RUN apt-get update \
2 changes: 1 addition & 1 deletion docker/server/Dockerfile
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@ FROM ubuntu:20.04
# FIXME, XXX, timeplus deb repo
# ARG repository="deb https://repo.timeplus/deb/stable/ main/"
ARG repository=""
ARG version=1.5.4
ARG version=1.5.5
ARG gosu_ver=1.10

# set non-empty deb_location_url url to create a docker image
2 changes: 1 addition & 1 deletion docker/test/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
FROM ubuntu:18.04

ARG repository="deb https://repo.clickhouse.com/deb/stable/ main/"
ARG version=1.5.4
ARG version=1.5.5

RUN apt-get update && \
apt-get install -y apt-transport-https dirmngr && \
51 changes: 36 additions & 15 deletions programs/install/Install.cpp
Original file line number Diff line number Diff line change
@@ -40,7 +40,7 @@
*
* The following steps are performed:
*
* - copying the binary to binary directory (/usr/bin).
* - copying the binary to binary directory (/usr/local/bin (Apple macOS) or /usr/bin (Others)).
* - creation of symlinks for tools.
* - creation of proton user and group.
* - creation of config directory (/etc/proton-server).
@@ -214,7 +214,11 @@ int mainInstall(int argc, char ** argv)
desc.add_options()
("help,h", "produce help message")
("prefix", po::value<std::string>()->default_value("/"), "prefix for all paths")
#if defined (OS_DARWIN)
("binary-path", po::value<std::string>()->default_value("usr/local/bin"), "where to install binaries")
#else
("binary-path", po::value<std::string>()->default_value("usr/bin"), "where to install binaries")
#endif
("config-path", po::value<std::string>()->default_value("etc/proton-server"), "where to install configs")
("log-path", po::value<std::string>()->default_value("var/log/proton-server"), "where to create log directory")
("data-path", po::value<std::string>()->default_value("var/lib/proton"), "directory for data")
@@ -240,12 +244,12 @@ int mainInstall(int argc, char ** argv)
uint32_t path_length = 0;
_NSGetExecutablePath(nullptr, &path_length);
if (path_length <= 1)
Exception(ErrorCodes::FILE_DOESNT_EXIST, "Cannot obtain path to the binary");
throw Exception(ErrorCodes::FILE_DOESNT_EXIST, "Cannot obtain path to the binary");

std::string path(path_length, std::string::value_type());
auto res = _NSGetExecutablePath(&path[0], &path_length);
if (res != 0)
Exception(ErrorCodes::FILE_DOESNT_EXIST, "Cannot obtain path to the binary");
throw Exception(ErrorCodes::FILE_DOESNT_EXIST, "Cannot obtain path to the binary");

if (path.back() == '\0')
path.pop_back();
@@ -322,15 +326,22 @@ int mainInstall(int argc, char ** argv)

try
{
ReadBufferFromFile in(binary_self_path.string());
WriteBufferFromFile out(main_bin_tmp_path.string());
copyData(in, out);
out.sync();
String source = binary_self_path.string();
String destination = main_bin_tmp_path.string();

if (0 != fchmod(out.getFD(), S_IRUSR | S_IRGRP | S_IROTH | S_IXUSR | S_IXGRP | S_IXOTH))
throwFromErrno(fmt::format("Cannot chmod {}", main_bin_tmp_path.string()), ErrorCodes::SYSTEM_ERROR);
/// Try to make a hard link first, as an optimization.
/// It is possible if the source and the destination are on the same filesystems.
if (0 != link(source.c_str(), destination.c_str()))
{
ReadBufferFromFile in(binary_self_path.string());
WriteBufferFromFile out(main_bin_tmp_path.string());
copyData(in, out);
out.sync();
out.finalize();
}

out.finalize();
if (0 != chmod(destination.c_str(), S_IRUSR | S_IRGRP | S_IROTH | S_IXUSR | S_IXGRP | S_IXOTH))
throwFromErrno(fmt::format("Cannot chmod {}", main_bin_tmp_path.string()), ErrorCodes::SYSTEM_ERROR);
}
catch (const Exception & e)
{
@@ -354,7 +365,7 @@ int mainInstall(int argc, char ** argv)

/// Create symlinks.

std::initializer_list<const char *> tools
std::initializer_list<std::string_view> tools
{
"proton-server",
"proton-client",
@@ -435,8 +446,8 @@ int mainInstall(int argc, char ** argv)
fs::path ulimits_file = ulimits_dir / fmt::format("{}.conf", user);
fmt::print("Will set ulimits for {} user in {}.\n", user, ulimits_file.string());
std::string ulimits_content = fmt::format(
"{0}\tsoft\tnofile\t262144\n"
"{0}\thard\tnofile\t262144\n", user);
"{0}\tsoft\tnofile\t1048576\n"
"{0}\thard\tnofile\t1048576\n", user);

fs::create_directories(ulimits_dir);

@@ -1150,7 +1161,11 @@ int mainStart(int argc, char ** argv)
desc.add_options()
("help,h", "produce help message")
("prefix", po::value<std::string>()->default_value("/"), "prefix for all paths")
#if defined (OS_DARWIN)
("binary-path", po::value<std::string>()->default_value("usr/local/bin"), "directory with binary")
#else
("binary-path", po::value<std::string>()->default_value("usr/bin"), "directory with binary")
#endif
("config-path", po::value<std::string>()->default_value("etc/proton-server"), "directory with configs")
("pid-path", po::value<std::string>()->default_value("var/run/proton-server"), "directory for pid file")
("user", po::value<std::string>()->default_value(DEFAULT_CLICKHOUSE_SERVER_USER), "proton user")
@@ -1162,8 +1177,10 @@ int mainStart(int argc, char ** argv)

if (options.count("help"))
{
std::cout << "Usage: " << formatWithSudo(std::string(argv[0]) + " start", getuid() != 0) << '\n';
return 1;
std::cout << "Install Proton without .deb/.rpm/.tgz packages (having the binary only)\n\n";
std::cout << "Usage: " << formatWithSudo(std::string(argv[0]) + " install [options]", getuid() != 0) << '\n';
std::cout << desc << '\n';
return 0;
}

std::string user = options["user"].as<std::string>();
@@ -1266,7 +1283,11 @@ int mainRestart(int argc, char ** argv)
desc.add_options()
("help,h", "produce help message")
("prefix", po::value<std::string>()->default_value("/"), "prefix for all paths")
#if defined (OS_DARWIN)
("binary-path", po::value<std::string>()->default_value("usr/local/bin"), "directory with binary")
#else
("binary-path", po::value<std::string>()->default_value("usr/bin"), "directory with binary")
#endif
("config-path", po::value<std::string>()->default_value("etc/proton-server"), "directory with configs")
("pid-path", po::value<std::string>()->default_value("var/run/proton-server"), "directory for pid file")
("user", po::value<std::string>()->default_value(DEFAULT_CLICKHOUSE_SERVER_USER), "proton user")
2 changes: 2 additions & 0 deletions programs/server/config.yaml
Original file line number Diff line number Diff line change
@@ -990,6 +990,8 @@ settings:
logstore_replication_factor: 1
storage_type: "hybrid"
logstore_codec: none
logstore_retention_bytes: -1 # when this threshold reaches, streaming storage delete old data. -1 means no limit
logstore_retention_ms: 86400000 # when this threshold reaches, streaming storage delete old data. -1 means no limit

cluster_settings:
logstore:
2 changes: 1 addition & 1 deletion src/AggregateFunctions/AggregateFunctionCount.h
Original file line number Diff line number Diff line change
@@ -77,7 +77,7 @@ class AggregateFunctionCount final : public IAggregateFunctionDataHelper<Aggrega
if (if_argument_pos >= 0)
{
const auto & flags = assert_cast<const ColumnUInt8 &>(*columns[if_argument_pos]).getData();
data(place).count += countBytesInFilter(flags);
data(place).count += countBytesInFilter(flags.data(), row_begin, row_end);
}
else
{
1 change: 1 addition & 0 deletions src/AggregateFunctions/AggregateFunctionSum.h
Original file line number Diff line number Diff line change
@@ -129,6 +129,7 @@ struct AggregateFunctionSumData
), addManyConditionalInternalImpl, MULTITARGET_FUNCTION_BODY((const Value * __restrict ptr, const UInt8 * __restrict condition_map, size_t start, size_t end) /// NOLINT
{
ptr += start;
condition_map += start;
size_t count = end - start;
const auto * end_ptr = ptr + count;

39 changes: 38 additions & 1 deletion src/AggregateFunctions/AggregateFunctionUniq.h
Original file line number Diff line number Diff line change
@@ -29,6 +29,10 @@
#include <AggregateFunctions/UniqVariadicHash.h>
#include <AggregateFunctions/UniquesHashSet.h>

namespace ErrorCodes
{
extern const int NOT_IMPLEMENTED;
}

namespace DB
{
@@ -42,6 +46,7 @@ struct AggregateFunctionUniqUniquesHashSetData
Set set;

constexpr static bool is_able_to_parallelize_merge = false;
constexpr static bool is_parallelize_merge_prepare_needed = false;
constexpr static bool is_variadic = false;

static String getName() { return "unique"; }
@@ -55,6 +60,7 @@ struct AggregateFunctionUniqUniquesHashSetDataForVariadic
Set set;

constexpr static bool is_able_to_parallelize_merge = false;
constexpr static bool is_parallelize_merge_prepare_needed = false;
constexpr static bool is_variadic = true;
constexpr static bool is_exact = is_exact_;
constexpr static bool argument_is_tuple = argument_is_tuple_;
@@ -72,6 +78,7 @@ struct AggregateFunctionUniqHLL12Data
Set set;

constexpr static bool is_able_to_parallelize_merge = is_able_to_parallelize_merge_;
constexpr static bool is_parallelize_merge_prepare_needed = false;
constexpr static bool is_variadic = false;

static String getName() { return "unique_hll12"; }
@@ -84,6 +91,7 @@ struct AggregateFunctionUniqHLL12Data<String, false>
Set set;

constexpr static bool is_able_to_parallelize_merge = false;
constexpr static bool is_parallelize_merge_prepare_needed = false;
constexpr static bool is_variadic = false;

static String getName() { return "unique_hll12"; }
@@ -96,6 +104,7 @@ struct AggregateFunctionUniqHLL12Data<UUID, false>
Set set;

constexpr static bool is_able_to_parallelize_merge = false;
constexpr static bool is_parallelize_merge_prepare_needed = false;
constexpr static bool is_variadic = false;

static String getName() { return "unique_hll12"; }
@@ -108,6 +117,7 @@ struct AggregateFunctionUniqHLL12DataForVariadic
Set set;

constexpr static bool is_able_to_parallelize_merge = is_able_to_parallelize_merge_;
constexpr static bool is_parallelize_merge_prepare_needed = false;
constexpr static bool is_variadic = true;
constexpr static bool is_exact = is_exact_;
constexpr static bool argument_is_tuple = argument_is_tuple_;
@@ -131,6 +141,7 @@ struct AggregateFunctionUniqExactData
Set set;

constexpr static bool is_able_to_parallelize_merge = is_able_to_parallelize_merge_;
constexpr static bool is_parallelize_merge_prepare_needed = true;
constexpr static bool is_variadic = false;

static String getName() { return "unique_exact"; }
@@ -150,6 +161,7 @@ struct AggregateFunctionUniqExactData<String, is_able_to_parallelize_merge_>
Set set;

constexpr static bool is_able_to_parallelize_merge = is_able_to_parallelize_merge_;
constexpr static bool is_parallelize_merge_prepare_needed = true;
constexpr static bool is_variadic = false;

static String getName() { return "unique_exact"; }
@@ -159,6 +171,7 @@ template <bool is_exact_, bool argument_is_tuple_, bool is_able_to_parallelize_m
struct AggregateFunctionUniqExactDataForVariadic : AggregateFunctionUniqExactData<String, is_able_to_parallelize_merge_>
{
constexpr static bool is_able_to_parallelize_merge = is_able_to_parallelize_merge_;
constexpr static bool is_parallelize_merge_prepare_needed = true;
constexpr static bool is_variadic = true;
constexpr static bool is_exact = is_exact_;
constexpr static bool argument_is_tuple = argument_is_tuple_;
@@ -173,6 +186,7 @@ struct AggregateFunctionUniqThetaData
Set set;

constexpr static bool is_able_to_parallelize_merge = false;
constexpr static bool is_parallelize_merge_prepare_needed = false;
constexpr static bool is_variadic = false;

static String getName() { return "unique_theta"; }
@@ -182,6 +196,7 @@ template <bool is_exact_, bool argument_is_tuple_>
struct AggregateFunctionUniqThetaDataForVariadic : AggregateFunctionUniqThetaData
{
constexpr static bool is_able_to_parallelize_merge = false;
constexpr static bool is_parallelize_merge_prepare_needed = false;
constexpr static bool is_variadic = true;
constexpr static bool is_exact = is_exact_;
constexpr static bool argument_is_tuple = argument_is_tuple_;
@@ -303,7 +318,7 @@ struct Adder

if constexpr (Data::is_able_to_parallelize_merge)
{
if (data.set.isSingleLevel() && data.set.size() > 100'000)
if (data.set.isSingleLevel() && data.set.worthConvertingToTwoLevel(data.set.size()))
data.set.convertToTwoLevel();
}
}
@@ -353,8 +368,10 @@ template <typename T, typename Data>
class AggregateFunctionUniq final : public IAggregateFunctionDataHelper<Data, AggregateFunctionUniq<T, Data>>
{
private:
using DataSet = typename Data::Set;
static constexpr size_t num_args = 1;
static constexpr bool is_able_to_parallelize_merge = Data::is_able_to_parallelize_merge;
static constexpr bool is_parallelize_merge_prepare_needed = Data::is_parallelize_merge_prepare_needed;

public:
explicit AggregateFunctionUniq(const DataTypes & argument_types_)
@@ -421,6 +438,26 @@ class AggregateFunctionUniq final : public IAggregateFunctionDataHelper<Data, Ag
detail::Adder<T, Data>::add(this->data(place), columns, num_args, row_begin, row_end, flags, null_map);
}

bool isParallelizeMergePrepareNeeded() const override { return is_parallelize_merge_prepare_needed;}

void parallelizeMergePrepare(AggregateDataPtrs & places, ThreadPool & thread_pool) const override
{
if constexpr (is_parallelize_merge_prepare_needed)
{
std::vector<DataSet *> data_vec;
data_vec.resize(places.size());

for (unsigned long i = 0; i < data_vec.size(); i++)
data_vec[i] = &this->data(places[i]).set;

DataSet::parallelizeMergePrepare(data_vec, thread_pool);
}
else
{
throw Exception(ErrorCodes::NOT_IMPLEMENTED, "parallelizeMergePrepare() is only implemented when is_parallelize_merge_prepare_needed is true for {} ", getName());
}
}

void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena *) const override
{
this->data(place).set.merge(this->data(rhs).set);
Loading