Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release 2.1.0 #577

Closed
hycdong opened this issue Aug 17, 2020 · 0 comments
Closed

Release 2.1.0 #577

hycdong opened this issue Aug 17, 2020 · 0 comments
Labels
release-note Notes on the version release

Comments

@hycdong
Copy link
Contributor

hycdong commented Aug 17, 2020 •
edited
Loading

2.1.0-RC1

rdsn

PR (107 TOTAL) TITLE
XiaoMi/rdsn#587 fix(bulk-load): fix bug during ingestion
XiaoMi/rdsn#584 refactor: make read batch size of fds configurable
XiaoMi/rdsn#583 fix(bulk_load): fix bug of clear_bulk_load_states_if_needed
XiaoMi/rdsn#581 fix(bulk-load): fix bug about concurrent downloading count
XiaoMi/rdsn#580 fix: remote-command max_copy_rate_megabytes failed when set "DEFAULT"
XiaoMi/rdsn#579 fix(bulk-load): fix bug while check_all_partitions
XiaoMi/rdsn#578 fix: fix building errors by clang-9
XiaoMi/rdsn#577 refactor: reduce compilation size for building each meta test
XiaoMi/rdsn#576 fix(bulk-load): fix bug while check app bulk load states
XiaoMi/rdsn#574 build(ci): refine build scripts
XiaoMi/rdsn#572 fix(bulk-load): fix bug while rolling back to downloading
XiaoMi/rdsn#571 feat(bulk-load): ignore bulk loading app while load balance
XiaoMi/rdsn#570 fix(bulk-load): update cleanup download task
XiaoMi/rdsn#568 perf(slog): Use a special thread pool for slog write callback
XiaoMi/rdsn#567 feat(bulk-load): add perf-counter
XiaoMi/rdsn#566 refactor: move runtime module from src/core/core to src/runtime
XiaoMi/rdsn#565 feat(bulk-load): add bulk load ddl client interface
XiaoMi/rdsn#564 feat(bulk-load): check app bulk load states while sync app from remote storage
XiaoMi/rdsn#563 feat(bulk-load): continue bulk load while meta server start part2
XiaoMi/rdsn#562 refactor: use DSN_REGISTER_COMPONENT_PROVIDER to register logger
XiaoMi/rdsn#560 fix: cleaup lock in ut abnormal_api_call
XiaoMi/rdsn#559 feat(bulk-load): continue bulk load while meta server start part1
XiaoMi/rdsn#558 refactor: move tools module from src/dist/replication/tool_lib to src/tools
XiaoMi/rdsn#557 refactor: change tests module name to test
XiaoMi/rdsn#556 feat(bulk-load): add query bulk load
XiaoMi/rdsn#555 refactor: move replica module from src/dist/replication/lib to src/replica
XiaoMi/rdsn#554 refactor: move meta module from src/dist/replication/meta_server to src/meta
XiaoMi/rdsn#553 feat(bulk-load): sync bulk load from remote storage
XiaoMi/rdsn#552 fix: fix compile thrift error in compile_thrift.py
XiaoMi/rdsn#551 fix: compile thrift error
XiaoMi/rdsn#550 refactor: move common module from src/dist/replication/common to src/common
XiaoMi/rdsn#549 refactor: move zookeeper module from src/dist/replication to src/zookeeper
XiaoMi/rdsn#548 feat(bulk-load): small update for meta bulk load service
XiaoMi/rdsn#547 refactor: combine src/dist/replication/client and src/dist/replication/ddl_lib into src/client
XiaoMi/rdsn#546 refactor: delete replication_checker which is unused
XiaoMi/rdsn#545 refactor: make throttle controller decoupled from dsn_runtime
XiaoMi/rdsn#543 feat(bulk-load): replica validate status
XiaoMi/rdsn#541 refactor: move perf_counter module from src/core/perf_counter to src/perf_counter
XiaoMi/rdsn#540 refactor: combine src/dist/failure_detector and src/dist/failure_detector_multimaster into src/failure_detector
XiaoMi/rdsn#539 refactor: move remote_cmd module from src/dist/cmd to src/remote_cmd
XiaoMi/rdsn#538 refactor: move nfs module from src/dist/nfs to src/nfs
XiaoMi/rdsn#537 refactor: move http module from src/dist/http to src/http
XiaoMi/rdsn#536 refactor: move block_service module from src/dist/block_service to src/block_service
XiaoMi/rdsn#535 feat(bulk-load): handle recoverable errors during bulk load
XiaoMi/rdsn#534 feat(bulk-load): replica handle errors during bulk load
XiaoMi/rdsn#533 refactor: move aio module from src/core/aio to src/aio
XiaoMi/rdsn#532 handle bulk load failed and app unavailable
XiaoMi/rdsn#531 feat(bulk-load): cancel bulk load
XiaoMi/rdsn#530 refactor: make simple logger decoupled from dsn_runtime
XiaoMi/rdsn#529 refactor: decouple sys_exit_type with dsn_runtime
XiaoMi/rdsn#528 refactor(aio): simplify disk_engine interfaces
XiaoMi/rdsn#526 fix(scripts): change the download source of zookeeper
XiaoMi/rdsn#525 build: use regex to find rocksdb depend module path
XiaoMi/rdsn#524 feat(bulk-load): restart bulk load
XiaoMi/rdsn#523 feat(thirdparty): Update rocksdb to rocksdb-6.6.4-compatible
XiaoMi/rdsn#522 refactor: make command manager decoupled from dsn_runtime
XiaoMi/rdsn#521 refactor(log): separate log_file from mutation_log
XiaoMi/rdsn#519 fix: revert "dassert if shared log is damaged when replica server init" commit
XiaoMi/rdsn#518 add pause bulk load
XiaoMi/rdsn#517 refactor: make utils decoupled from dsn_runtime
XiaoMi/rdsn#516 refactor(aio): refine code style of aio module
XiaoMi/rdsn#515 refactor: move task to a seperated directory
XiaoMi/rdsn#514 feat(bulk-load): pause bulk load part1 - add control_bulk_load rpc and start to pause bulk load
XiaoMi/rdsn#513 fix: dassert if shared log is damaged when replica server init
XiaoMi/rdsn#512 refactor: remove the unused struct io_engine
XiaoMi/rdsn#510 feat(utils): add latency trace tool
XiaoMi/rdsn#508 feat(bulk-load): bulk load succeed part2 - meta handle bulk load succeed
XiaoMi/rdsn#507 refactor: use DSN_REGISTER_COMPONENT_PROVIDER to register aio_provider
XiaoMi/rdsn#505 refactor: add a interface of rpc_holder in storage_serverlet
XiaoMi/rdsn#504 refactor: move rpc module to a separated directory
XiaoMi/rdsn#502 feat(bulk-load): bulk load succeed part1 - replica handle bulk load succeed
XiaoMi/rdsn#501 refactor: use check_status to reimplement some rpc interfaces of meta_service
XiaoMi/rdsn#500 feat(bulk-load): bulk load ingestion part6 - meta handle bulk_load_response during ingestion
XiaoMi/rdsn#498 feat: support adding ignored apps for load balancer
XiaoMi/rdsn#497 fix: fix RPC_CHECK_STATUS incorrect usage in meta_service
XiaoMi/rdsn#496 feat(bulk-load): bulk load ingestion part5 - replica handle bulk load request during ingestion
XiaoMi/rdsn#495 feat: add perf counter for group check failure
XiaoMi/rdsn#494 refactor: use flag api to load the config in block_service module
XiaoMi/rdsn#493 feat(bulk-load): bulk load ingestion part4 - meta handle ingestion response
XiaoMi/rdsn#492 refactor: refactor restore progress
XiaoMi/rdsn#491 refactor: move function verify_file to file_system
XiaoMi/rdsn#490 feat(bulk-load): bulk load ingestion part2 - replica handle ingestion request
XiaoMi/rdsn#489 refactor: use rpc holder to reimplement rpc interface of replica server
XiaoMi/rdsn#488 refactor: use flag api to load the config in nfs module
XiaoMi/rdsn#487 refactor: use rpc holder to reimplement rpc interface of meta server
XiaoMi/rdsn#486 feat(bulk-load): bulk load ingestion part1 - meta start ingestion
XiaoMi/rdsn#484 build: fix build failure when compiling pegasus
XiaoMi/rdsn#483 build: add rocksdb as thirdparty
XiaoMi/rdsn#482 feat(bulk-load): bulk load download part5 - meta handle download status and progress
XiaoMi/rdsn#481 fix(dup): convert type of failure perf-counter to VOLATILE_NUMBER
XiaoMi/rdsn#480 refactor: refactor block service manager
XiaoMi/rdsn#479 feat(bulk-load): bulk load download part4 - replica report download status and progress
XiaoMi/rdsn#478 fix: fix memory leak in unit test netprovider.cpp
XiaoMi/rdsn#477 fix: fix memory leak in asio_udp_provider
XiaoMi/rdsn#476 fix: use reference instead of value
XiaoMi/rdsn#475 feat(bulk-load): bulk load download part3 - replica parse metadata, verify files and update progress
XiaoMi/rdsn#473 refactor: simplify remote-command rpc
XiaoMi/rdsn#471 feat(bulk-load): bulk load download part2 - replica download files
XiaoMi/rdsn#469 fix: fix bug in download-thirdparty
XiaoMi/rdsn#467 refactor(bulk-load): add replica_bulk_loader class
XiaoMi/rdsn#466 refactor: make aio decoupled from dsn_runtime
XiaoMi/rdsn#465 feat(bulk-load): bulk load download part1 - replica start download
XiaoMi/rdsn#464 feat(task): add an interface to query thread info
XiaoMi/rdsn#463 feat(bulk-load): meta handle bulk_load_response
XiaoMi/rdsn#460 feat(bulk-load): group bulk load request in replica group
XiaoMi/rdsn#453 feat(thirdparty): Introduce readerwriterqueue util
XiaoMi/rdsn#438 build(deps): use cmake ExternalProject to manage dependencies

pegasus

PR (26 TOTAL) TITLE
#600 refactor: update license statement
#594 fix: fix db is not nullptr when release_db
#593 fix: Use default options to open db when latest option file has incompatible db options
#589 chore: add disclaimer for not compliant with asf policy
#587 fix: Load options from file when open an exist DB
#573 fix: change the construction parameters of hotspot calculator
#572 fix: delete outdated algorithm of table_hotspot_policy
#569 fix: update pack scripts to remove boost paths
#568 feat(bulk-load): add bulk load function test
#566 perf(slog): add new pool for slog callback
#565 feat(bulk-load): add query bulk load shell command
#563 fix: modify pack_server.sh to copy libdsn_utils.so
#561 feat(bulk-load): add bulk load shell command
#559 refactor: add the dependend lib which name is dsn_utils
#557 refactor: modify the dependent lib names which are generated by rdsn
#556 feat(rocksdb): Remove all calls of Pegasus introduced APIs on RocksDB
#554 refactor: use rpc_holder to receive RPC in pegasus_server_impl
#553 fix: fix incorrect log message in info_collector::on_app_stat
#551 refactor: refactor rrdb_server into pegasus _read_service
#550 refactor: use flag api to define configs in pegasus_reporter
#546 feat(bulk-load): bulk load ingestion part3 - execute ingestion
#545 build: remove rocksdb as a submodule
#544 refactor: adapt to rdsn dist cmd refactoring
#543 feat(rocksdb): Adapt rate limiter to prevent bust writes and huge compaction
#542 fix: fix bug in function_test/test_restore
#535 feat(rocksdb): Use original CreateCheckpoint and read meta data from meta CF

Incompatitable Modifications

PR TITLE
XiaoMi/rdsn#486 feat(bulk-load): bulk load ingestion part1 - meta start ingestion
XiaoMi/rdsn#523 feat(thirdparty): Update rocksdb to rocksdb-6.6.4-compatible
#535 feat(rocksdb): Use original CreateCheckpoint and read meta data from meta CF
#556 feat(rocksdb): Remove all calls of Pegasus introduced APIs on RocksDB

New perf-counter

Configuration update

[apps.replica]
+  pools = ...THREAD_POOL_INGESTION, THREAD_POOL_SLOG

+[threadpool.THREAD_POOL_INGESTION]
+  name = ingestion
+  partitioned = false
+  worker_priority = THREAD_xPRIORITY_NORMAL
+  worker_count = 24

+[threadpool.THREAD_POOL_SLOG]
+ name = slog
+ worker_count = 1

[replication]
+ bulk_load_provider_root = bulk_load_root
+ max_concurrent_bulk_load_downloading_count = 5
+ enable_latency_tracer = false
+ fds_read_batch_size = 100

[pegasus.server]
+ rocksdb_limiter_max_write_megabytes_per_sec = 500
+ rocksdb_limiter_enable_auto_tune = false
- get_meta_store_type = metacf
@hycdong hycdong added the release-note Notes on the version release label Aug 17, 2020
@apache apache deleted a comment from hycdong Dec 4, 2020
acelyc111 pushed a commit that referenced this issue Jun 23, 2022
refactor: reduce compilation size for building each meta test (#577)
adeb3e9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
release-note Notes on the version release
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@neverchanje @hycdong