Skip to content

Commit

Permalink
Merge branch 'master' into master
Browse files Browse the repository at this point in the history
  • Loading branch information
Tuvie committed Aug 15, 2022
2 parents 7d38c8f + 0e10bdb commit 568b1ac
Show file tree
Hide file tree
Showing 67 changed files with 637 additions and 175 deletions.
35 changes: 35 additions & 0 deletions .github/workflows/license-eyes.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
---
name: License Check
on:
pull_request:
push:
branches:
- master
jobs:
license-check:
name: "License Check"
runs-on: ubuntu-latest
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v2
- name: Check License
uses: apache/skywalking-eyes@v0.4.0
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
217 changes: 217 additions & 0 deletions .licenserc.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,217 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
---
header:
license:
spdx-id: Apache-2.0
copyright-owner: Apache Software Foundation

paths-ignore:
- '**/*.md'
- '**/*.svg'
- '**/*.yml'
- '*/TBD'
- '.idea'
- 'LICENSE'
- 'NOTICE'
- 'docs'
- 'example/*/*.flags'
- 'example/*/*.json'
- 'example/*/*.pem'
- 'example/*/*.port'
- 'src/bthread/offset_inl.list'
- 'test/*.crt'
- 'test/*.key'
- 'test/jsonout'
- 'tools/trackme_server/bugs'

# Apache
- 'src/butil/recordio.h'

# Boost Software License
- 'src/bthread/context.*'
- 'src/butil/intrusive_ptr.hpp'
- 'src/butil/unique_ptr.h'

# BSD
- 'src/butil/crc32c.*'
- 'src/butil/third_party/modp_b64/*'
- 'src/butil/third_party/superfasthash/*'
- 'src/butil/third_party/valgrind/*'

# Chromium
- 'src/butil/at_exit.*'
- 'src/butil/atomic*'
- 'src/butil/auto_reset.h'
- 'src/butil/base64.*'
- 'src/butil/base_export.h'
- 'src/butil/base_paths.cc'
- 'src/butil/basictypes.h'
- 'src/butil/big_endian.*'
- 'src/butil/bits.h'
- 'src/butil/build_config.h'
- 'src/butil/cancelable_callback.h'
- 'src/butil/compiler_specific.h'
- 'src/butil/containers/*'
- 'src/butil/cpu.*'
- 'src/butil/debug/*'
- 'src/butil/environment.*'
- 'src/butil/file_*'
- 'src/butil/files/dir_reader_fallback.h'
- 'src/butil/files/dir_reader_linux.h'
- 'src/butil/files/dir_reader_posix.h'
- 'src/butil/files/file.*'
- 'src/butil/files/file_enumerator*'
- 'src/butil/files/file_p*'
- 'src/butil/files/memory_mapped_file*'
- 'src/butil/files/scoped*'
- 'src/butil/float_util.h'
- 'src/butil/format_macros.h'
- 'src/butil/gtest_prod_util.h'
- 'src/butil/guid.cc'
- 'src/butil/guid.h'
- 'src/butil/guid_posix.cc'
- 'src/butil/hash.cc'
- 'src/butil/hash.h'
- 'src/butil/lazy_instance.cc'
- 'src/butil/lazy_instance.h'
- 'src/butil/location.cc'
- 'src/butil/location.h'
- 'src/butil/mac/bundle_locations.h'
- 'src/butil/mac/bundle_locations.mm'
- 'src/butil/mac/foundation_util.h'
- 'src/butil/mac/foundation_util.mm'
- 'src/butil/mac/scoped_cftyperef.h'
- 'src/butil/mac/scoped_mach_port.cc'
- 'src/butil/mac/scoped_mach_port.h'
- 'src/butil/mac/scoped_typeref.h'
- 'src/butil/macros.h'
- 'src/butil/memory/aligned_memory.cc'
- 'src/butil/memory/aligned_memory.h'
- 'src/butil/memory/linked_ptr.h'
- 'src/butil/memory/manual_constructor.h'
- 'src/butil/memory/raw_scoped_refptr_mismatch_checker.h'
- 'src/butil/memory/ref_counted.cc'
- 'src/butil/memory/ref_counted.h'
- 'src/butil/memory/ref_counted_memory.cc'
- 'src/butil/memory/ref_counted_memory.h'
- 'src/butil/memory/scoped_open_process.h'
- 'src/butil/memory/scoped_policy.h'
- 'src/butil/memory/scoped_ptr.h'
- 'src/butil/memory/scoped_vector.h'
- 'src/butil/memory/singleton.cc'
- 'src/butil/memory/singleton.h'
- 'src/butil/memory/singleton_objc.h'
- 'src/butil/memory/weak_ptr.cc'
- 'src/butil/memory/weak_ptr.h'
- 'src/butil/move.h'
- 'src/butil/numerics/safe_conversions.h'
- 'src/butil/numerics/safe_conversions_impl.h'
- 'src/butil/numerics/safe_math.h'
- 'src/butil/numerics/safe_math_impl.h'
- 'src/butil/observer_list.h'
- 'src/butil/port.h'
- 'src/butil/posix/eintr_wrapper.h'
- 'src/butil/posix/file_descriptor_shuffle.cc'
- 'src/butil/posix/file_descriptor_shuffle.h'
- 'src/butil/posix/global_descriptors.cc'
- 'src/butil/posix/global_descriptors.h'
- 'src/butil/rand_util.cc'
- 'src/butil/rand_util.h'
- 'src/butil/rand_util_posix.cc'
- 'src/butil/safe_strerror_posix.cc'
- 'src/butil/safe_strerror_posix.h'
- 'src/butil/scoped_clear_errno.h'
- 'src/butil/scoped_generic.h'
- 'src/butil/scoped_observer.h'
- 'src/butil/sha1.h'
- 'src/butil/sha1_portable.cc'
- 'src/butil/stl_util.h'
- 'src/butil/strings/latin1_string_conversions.cc'
- 'src/butil/strings/latin1_string_conversions.h'
- 'src/butil/strings/nullable_string16.cc'
- 'src/butil/strings/nullable_string16.h'
- 'src/butil/strings/safe_sprintf.cc'
- 'src/butil/strings/safe_sprintf.h'
- 'src/butil/strings/string16.cc'
- 'src/butil/strings/string16.h'
- 'src/butil/strings/string_number_conversions.cc'
- 'src/butil/strings/string_number_conversions.h'
- 'src/butil/strings/string_piece.cc'
- 'src/butil/strings/string_piece.h'
- 'src/butil/strings/string_split.cc'
- 'src/butil/strings/string_split.h'
- 'src/butil/strings/string_tokenizer.h'
- 'src/butil/strings/string_util.cc'
- 'src/butil/strings/string_util.h'
- 'src/butil/strings/string_util_constants.cc'
- 'src/butil/strings/string_util_posix.h'
- 'src/butil/strings/stringize_macros.h'
- 'src/butil/strings/stringprintf.cc'
- 'src/butil/strings/stringprintf.h'
- 'src/butil/strings/sys_string_conversions.h'
- 'src/butil/strings/sys_string_conversions_mac.mm'
- 'src/butil/strings/sys_string_conversions_posix.cc'
- 'src/butil/strings/utf_offset_string_conversions.cc'
- 'src/butil/strings/utf_offset_string_conversions.h'
- 'src/butil/strings/utf_string_conversion_utils.cc'
- 'src/butil/strings/utf_string_conversion_utils.h'
- 'src/butil/strings/utf_string_conversions.cc'
- 'src/butil/strings/utf_string_conversions.h'
- 'src/butil/synchronization/cancellation_flag.*'
- 'src/butil/synchronization/condition_variable.h'
- 'src/butil/synchronization/condition_variable_posix.cc'
- 'src/butil/synchronization/spin_wait.h'
- 'src/butil/synchronization/waitable_event.h'
- 'src/butil/synchronization/waitable_event_posix.cc'
- 'src/butil/sys_byteorder.h'
- 'src/butil/third_party/symbolize/glog/*'
- 'src/butil/threading/*'
- 'src/butil/time/*'
- 'src/butil/type_traits.h'
- 'src/butil/version.*'
- 'test/*.cc'
- 'test/memory_unittest_mac.h'
- 'test/multiprocess_func_list.h'
- 'test/scoped_locale.h'
- 'test/test_switches.h'

# David M. Gay
- 'src/butil/third_party/dmg_fp/*'

# Google
- 'src/brpc/builtin/pprof_perl.cpp'
- 'src/brpc/callback.h'
- 'src/brpc/details/tcmalloc_extension.h'
- 'src/butil/gperftools_profiler.h'
- 'src/butil/third_party/dynamic_annotations/*'
- 'src/butil/third_party/snappy/*'
- 'src/butil/third_party/symbolize/*'
- 'tools/pprof'

# ICU
- 'src/butil/third_party/icu/*'

# MIT
- 'src/brpc/policy/dh.*'
- 'src/butil/third_party/rapidjson/**'

# NGINX
- 'src/brpc/details/http_parser.*'

comment: on-failure
2 changes: 1 addition & 1 deletion BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ config_setting(

COPTS = [
"-DBTHREAD_USE_FAST_PTHREAD_MUTEX",
"-D__const__=",
"-D__const__=__unused__",
"-D_GNU_SOURCE",
"-DUSE_SYMBOLIZE",
"-DNO_TCMALLOC",
Expand Down
3 changes: 1 addition & 2 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,6 @@ endif()
set(WITH_RDMA_VAL "0")
if(WITH_RDMA)
set(WITH_RDMA_VAL "1")
set(BRPC_WITH_RDMA 1)
endif()

include(GNUInstallDirs)
Expand Down Expand Up @@ -117,7 +116,7 @@ set(CMAKE_CPP_FLAGS "${CMAKE_CPP_FLAGS} ${DEFINE_CLOCK_GETTIME} -DBRPC_WITH_GLOG
if(WITH_MESALINK)
set(CMAKE_CPP_FLAGS "${CMAKE_CPP_FLAGS} -DUSE_MESALINK")
endif()
set(CMAKE_CPP_FLAGS "${CMAKE_CPP_FLAGS} -DBTHREAD_USE_FAST_PTHREAD_MUTEX -D__const__= -D_GNU_SOURCE -DUSE_SYMBOLIZE -DNO_TCMALLOC -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -DBRPC_REVISION=\\\"${BRPC_REVISION}\\\" -D__STRICT_ANSI__")
set(CMAKE_CPP_FLAGS "${CMAKE_CPP_FLAGS} -DBTHREAD_USE_FAST_PTHREAD_MUTEX -D__const__=__unused__ -D_GNU_SOURCE -DUSE_SYMBOLIZE -DNO_TCMALLOC -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -DBRPC_REVISION=\\\"${BRPC_REVISION}\\\" -D__STRICT_ANSI__")
set(CMAKE_CPP_FLAGS "${CMAKE_CPP_FLAGS} ${DEBUG_SYMBOL} ${THRIFT_CPP_FLAG}")
set(CMAKE_CXX_FLAGS "${CMAKE_CPP_FLAGS} -O2 -pipe -Wall -W -fPIC -fstrict-aliasing -Wno-invalid-offsetof -Wno-unused-parameter -fno-omit-frame-pointer")
set(CMAKE_C_FLAGS "${CMAKE_CPP_FLAGS} -O2 -pipe -Wall -W -fPIC -fstrict-aliasing -Wno-unused-parameter -fno-omit-frame-pointer")
Expand Down
5 changes: 2 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,9 @@ include config.mk

# Notes on the flags:
# 1. Added -fno-omit-frame-pointer: perf/tcmalloc-profiler use frame pointers by default
# 2. Added -D__const__= : Avoid over-optimizations of TLS variables by GCC>=4.8
# 3. Removed -Werror: Not block compilation for non-vital warnings, especially when the
# 2. Removed -Werror: Not block compilation for non-vital warnings, especially when the
# code is tested on newer systems. If the code is used in production, add -Werror back
CPPFLAGS+=-DBTHREAD_USE_FAST_PTHREAD_MUTEX -D__const__= -D_GNU_SOURCE -DUSE_SYMBOLIZE -DNO_TCMALLOC -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -DNDEBUG -DBRPC_REVISION=\"$(shell ./tools/get_brpc_revision.sh .)\"
CPPFLAGS+=-DBTHREAD_USE_FAST_PTHREAD_MUTEX -D_GNU_SOURCE -DUSE_SYMBOLIZE -DNO_TCMALLOC -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -DNDEBUG -DBRPC_REVISION=\"$(shell ./tools/get_brpc_revision.sh .)\"
CXXFLAGS=$(CPPFLAGS) -O2 -pipe -Wall -W -fPIC -fstrict-aliasing -Wno-invalid-offsetof -Wno-unused-parameter -fno-omit-frame-pointer -std=c++0x
CFLAGS=$(CPPFLAGS) -O2 -pipe -Wall -W -fPIC -fstrict-aliasing -Wno-unused-parameter -fno-omit-frame-pointer
DEBUG_CXXFLAGS = $(filter-out -DNDEBUG,$(CXXFLAGS)) -DUNIT_TEST -DBVAR_NOT_LINK_DEFAULT_VARIABLES
Expand Down
1 change: 1 addition & 0 deletions community/newcommitter.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,4 +58,5 @@ Doing these things will make everyone's job easier.
3. 在private@brpc中发邮件,结束投票,并通知private@incubator.apache.org
4. 在private@brpc中和dev中announce new PPMC
5. 设定他的权限,通过访问https://whimsy.apache.org/roster/ppmc/brpc
6. 帮他订阅private邮件组,参见https://whimsy.apache.org/committers/moderationhelper.cgi

5 changes: 5 additions & 0 deletions config_brpc.sh
Original file line number Diff line number Diff line change
Expand Up @@ -322,6 +322,10 @@ append_to_output "STATIC_LINKINGS=$STATIC_LINKINGS"
append_to_output "DYNAMIC_LINKINGS=$DYNAMIC_LINKINGS"
CPPFLAGS="-DBRPC_WITH_GLOG=$WITH_GLOG -DGFLAGS_NS=$GFLAGS_NS"

# Avoid over-optimizations of TLS variables by GCC>=4.8
# See: https://github.com/apache/incubator-brpc/issues/1693
CPPFLAGS="${CPPFLAGS} -D__const__=__unused__"

if [ ! -z "$DEBUGSYMBOLS" ]; then
CPPFLAGS="${CPPFLAGS} $DEBUGSYMBOLS"
fi
Expand Down Expand Up @@ -357,6 +361,7 @@ if [ $WITH_RDMA != 0 ]; then
CPPFLAGS="${CPPFLAGS} -DBRPC_WITH_RDMA"

append_to_output "DYNAMIC_LINKINGS+=-libverbs"
append_to_output "WITH_RDMA=1"
fi

if [ $WITH_MESALINK != 0 ]; then
Expand Down
2 changes: 1 addition & 1 deletion docs/cn/auto_concurrency_limiter.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ min_latency是最近一段时间测量到的latency较小值的ema,是noload_l
为了减少个别窗口的抖动对限流算法的影响,同时尽量降低计算开销,计算min_latency时会通过使用[EMA](https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average)来进行平滑处理:

```
if latency > min_latency:
if latency < min_latency:
min_latency = latency * ema_alpha + (1 - ema_alpha) * min_latency
else:
do_nothing
Expand Down
9 changes: 7 additions & 2 deletions docs/cn/getting_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,11 @@ $ sh run_tests.sh
```

### 使用cmake编译brpc

```shell
mkdir build && cd build && cmake .. && cmake --build . -j6
```
对于 cmake 3.13+ 也可以使用如下命令进行编译:
```shell
cmake -B build && cmake --build build -j6
```
Expand Down Expand Up @@ -282,15 +287,15 @@ $ sh run_tests.sh

# 支持的依赖

## GCC: 4.8-7.1
## GCC: 4.8-11.2

c++11被默认启用,以去除去boost的依赖(比如atomic)。

GCC7中over-aligned的问题暂时被禁止。

使用其他版本的gcc可能会产生编译警告,请联系我们予以修复。

请在makefile中给cxxflags增加`-D__const__=`选项以避免[gcc4+中的errno问题](thread_local.md).
请在makefile中给cxxflags增加`-D__const__=__unused__`选项以避免[gcc4+中的errno问题](thread_local.md).

## Clang: 3.5-4.0

Expand Down
16 changes: 13 additions & 3 deletions docs/cn/rdma.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

使用config_brpc:
```bash
sh config_brpc.sh --with-rdma
sh config_brpc.sh --with-rdma --headers="/usr/include" --libs="/usr/lib64 /usr/bin"
make

cd example/rdma_performance # 示例程序
Expand All @@ -23,9 +23,19 @@ make

# 基本实现

brpc内部使用RDMA RC模式,每个Socket对应一个QP。RDMA连接建立依赖于前置TCP建连,TCP建连后双方交换必要参数(含GID、QPN等),再发起RDMA连接并实现数据传输。建连用的TCP连接在RDMA连接活跃期间并不传输数据,但仍保持EST状态。一旦TCP连接中断,其上对应的RDMA连接同样会置错
RDMA与TCP不同,不使用socket接口进行通信。但是在实现上仍然复用了brpc中原本的Socket类。当用户选择ChannelOptions或ServerOptions中的use_rdma为true时,创建出的Socket类中则有对应的RdmaEndpoint(参见src/brpc/rdma/rdma_endpoint.cpp)。当RDMA被使能时,写入Socket的数据会通过RdmaEndpoint提交给RDMA QP(通过verbs API),而非拷贝到fd。对于数据读取,RdmaEndpoint中则调用verbs API从RDMA CQ中获取对应完成信息(事件获取有独立的fd,复用EventDispatcher,处理函数采用RdmaEndpoint::PollCq),最后复用InputMessenger完成RPC消息解析

RDMA要求数据收发所使用的内存空间必须被注册(memory register),这一操作非常耗时,所以通常都会使用内存池方案来加速。brpc内部的数据收发都使用IOBuf,为了在兼容IOBuf的情况下实现完全零拷贝,整个IOBuf所使用的内存空间整体由统一内存池接管。注意,由于IOBuf内存池不由用户直接控制,因此实际使用中需要注意IOBuf所消耗的总内存,建议根据实际业务需求,一次性注册足够的内存池以实现性能最大化。
brpc内部使用RDMA RC模式,每个RdmaEndpoint对应一个QP。RDMA连接建立依赖于前置TCP建连,TCP建连后双方交换必要参数,如GID、QPN等,再发起RDMA连接并实现数据传输。这个过程我们称为握手(参见RdmaEndpoint)。因为握手需要TCP连接,因此RdmaEndpoint所在的Socket类中,原本的TCP fd仍然有效。握手过程采用了brpc中已有的AppConnect逻辑。注意,握手用的TCP连接在后续数据传输阶段并不会收发数据,但仍保持为EST状态。一旦TCP连接中断,其上对应的RDMA连接同样会置错。

RdmaEndpoint数据传输逻辑的第一个重要特性是零拷贝。要发送的所有数据默认都存放在IOBuf的Block中,因此所发送的Block需要等到对端确认接收完成后才可以释放,这些Block的引用被存放于RdmaEndpoint::_sbuf中。而要实现接收零拷贝,则需要确保接受端所预提交的接收缓冲区必须直接在IOBuf的Block里面,被存放于RdmaEndpoint::_rbuf。注意,接收端预提交的每一段Block,有一个固定的大小(recv_block_size)。发送端发送时,一个请求最多只能有这么大,否则接收端则无法成功接收。

RdmaEndpoint数据传输逻辑的第二个重要特性是滑动窗口流控。这一流控机制是为了避免发送端持续在发送,其速度超过了接收端处理的速度。TCP传输中也有类似的逻辑,但是是由内核协议栈来实现的。RdmaEndpoint内实现了这一流控机制,通过接收端显式回复ACK来确认接收端处理完毕。为了减少ACK本身的开销,让ACK以立即数形式返回,可以被附在数据消息里。

RdmaEndpoint数据传输逻辑的第三个重要特性是事件聚合。每个消息的大小被限定在一个recv_block_size,默认为8KB。如果每个消息都触发事件进行处理,会导致性能退化严重,甚至不如TCP传输(TCP拥有GSO、GRO等诸多优化)。因此,RdmaEndpoint综合考虑数据大小、窗口与ACK的情况,对每个发送消息选择性设置solicited标志,来控制是否在发送端触发事件通知。

RDMA要求数据收发所使用的内存空间必须被注册(memory register),把对应的页表映射注册给网卡,这一操作非常耗时,所以通常都会使用内存池方案来加速。brpc内部的数据收发都使用IOBuf,为了在兼容IOBuf的情况下实现完全零拷贝,整个IOBuf所使用的内存空间整体由统一内存池接管(参见src/brpc/rdma/block_pool.cpp)。注意,由于IOBuf内存池不由用户直接控制,因此实际使用中需要注意IOBuf所消耗的总内存,建议根据实际业务需求,一次性注册足够的内存池以实现性能最大化。

RDMA是硬件相关的通信技术,有很多独特的概念,比如device、port、GID、LID、MaxSge等。这些参数在初始化时会从对应的网卡中读取出来,并且做出默认的选择(参见src/brpc/rdma/rdma_helper.cpp)。有时默认的选择并非用户的期望,则可以通过flag参数方式指定。

# 参数

Expand Down
Loading

0 comments on commit 568b1ac

Please sign in to comment.