Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added max async export support using separate AsyncBatchSpan/LogProcessor #1306

Merged
merged 11 commits into from
May 4, 2022
22 changes: 22 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,28 @@ jobs:
- name: run tests
run: ./ci/do_ci.sh bazel.test

bazel_test_async:
name: Bazel with async export
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
submodules: 'recursive'
- name: Mount Bazel Cache
uses: actions/cache@v3
env:
cache-name: bazel_cache
with:
path: /home/runner/.cache/bazel
key: bazel_test
- name: setup
run: |
sudo ./ci/setup_thrift.sh dependencies_only
sudo ./ci/setup_ci_environment.sh
sudo ./ci/install_bazelisk.sh
- name: run tests
run: ./ci/do_ci.sh bazel.with_async_export
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you enable -DENABLE_ASYNC_EXPORT_PREVIEW for all jobs but left one without it?I think we also need benchmark,address sanitizer, thread sanitizer and so on for this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please point me to some examples to refer to for benchmark, address sanitizer and thread sanitizer for this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I missed something, bazel.asan, bazel.tsan and bazel.valgrind already enable async APIs now, but benchmark still use $BAZEL_OPTIONS, I think we should also use $BAZEL_OPTIONS_ASYNC.
And we should also add --copt=-DENABLE_ASYNC_EXPORT_PREVIEW or -DWITH_ASYNC_EXPORT_PREVIEW=ON into all actions in ci/do_ci.ps1, it's used on Windows.


bazel_with_abseil:
name: Bazel with external abseil
runs-on: ubuntu-latest
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ Increment the:
* [SDK] Bugfix: span SetAttribute crash ([#1283](https://github.com/open-telemetry/opentelemetry-cpp/pull/1283))
* [EXPORTER] Jaeger Exporter - Populate Span Links ([#1251](https://github.com/open-telemetry/opentelemetry-cpp/pull/1251))
* [EXPORTER] OTLP http exporter allow concurrency session ([#1209](https://github.com/open-telemetry/opentelemetry-cpp/pull/1209))
* [SDK] Async Batch Span/Log processor with max async support ([#1306](https://github.com/open-telemetry/opentelemetry-cpp/pull/1306))

## [1.2.0] 2022-01-31

Expand Down
6 changes: 5 additions & 1 deletion ci/do_ci.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ $action = $args[0]

$SRC_DIR=(Get-Item -Path ".\").FullName

$BAZEL_OPTIONS="--copt=-DENABLE_METRICS_PREVIEW --copt=-DENABLE_LOGS_PREVIEW"
$BAZEL_OPTIONS="--copt=-DENABLE_METRICS_PREVIEW --copt=-DENABLE_LOGS_PREVIEW --copt=-DENABLE_ASYNC_EXPORT"
$BAZEL_TEST_OPTIONS="$BAZEL_OPTIONS --test_output=errors"

if (!(test-path build)) {
Expand All @@ -32,6 +32,7 @@ switch ($action) {
cd "$BUILD_DIR"
cmake $SRC_DIR `
-DVCPKG_TARGET_TRIPLET=x64-windows `
-DWITH_ASYNC_EXPORT_PREVIEW=ON `
"-DCMAKE_TOOLCHAIN_FILE=$VCPKG_DIR\scripts\buildsystems\vcpkg.cmake"
$exit = $LASTEXITCODE
if ($exit -ne 0) {
Expand All @@ -52,6 +53,7 @@ switch ($action) {
cd "$BUILD_DIR"
cmake $SRC_DIR `
-DVCPKG_TARGET_TRIPLET=x64-windows `
-DWITH_ASYNC_EXPORT_PREVIEW=ON `
-DWITH_OTPROTCOL=ON `
"-DCMAKE_TOOLCHAIN_FILE=$VCPKG_DIR\scripts\buildsystems\vcpkg.cmake"
$exit = $LASTEXITCODE
Expand All @@ -73,6 +75,7 @@ switch ($action) {
cd "$BUILD_DIR"
cmake $SRC_DIR `
-DVCPKG_TARGET_TRIPLET=x64-windows `
-DWITH_ASYNC_EXPORT_PREVIEW=ON `
"-DCMAKE_TOOLCHAIN_FILE=$VCPKG_DIR\scripts\buildsystems\vcpkg.cmake"
$exit = $LASTEXITCODE
if ($exit -ne 0) {
Expand All @@ -89,6 +92,7 @@ switch ($action) {
cd "$BUILD_DIR"
cmake $SRC_DIR `
-DVCPKG_TARGET_TRIPLET=x64-windows `
-DWITH_ASYNC_EXPORT_PREVIEW=ON `
"-DCMAKE_TOOLCHAIN_FILE=$VCPKG_DIR\scripts\buildsystems\vcpkg.cmake"
$exit = $LASTEXITCODE
if ($exit -ne 0) {
Expand Down
39 changes: 26 additions & 13 deletions ci/do_ci.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ function run_benchmarks

[ -z "${BENCHMARK_DIR}" ] && export BENCHMARK_DIR=$HOME/benchmark
mkdir -p $BENCHMARK_DIR
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS -c opt -- \
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS_ASYNC -c opt -- \
$(bazel query 'attr("tags", "benchmark_result", ...)')
echo ""
echo "Benchmark results in $BENCHMARK_DIR:"
Expand Down Expand Up @@ -62,8 +62,11 @@ mkdir -p "${PLUGIN_DIR}"
BAZEL_OPTIONS="--copt=-DENABLE_METRICS_PREVIEW --copt=-DENABLE_LOGS_PREVIEW --copt=-DENABLE_TEST"
BAZEL_TEST_OPTIONS="$BAZEL_OPTIONS --test_output=errors"

BAZEL_OPTIONS_ASYNC="--copt=-DENABLE_METRICS_PREVIEW --copt=-DENABLE_LOGS_PREVIEW --copt=-DENABLE_TEST --copt=-DENABLE_ASYNC_EXPORT"
BAZEL_TEST_OPTIONS_ASYNC="$BAZEL_OPTIONS_ASYNC --test_output=errors"

# https://github.com/bazelbuild/bazel/issues/4341
BAZEL_MACOS_OPTIONS="$BAZEL_OPTIONS --features=-supports_dynamic_linker --build_tag_filters=-jaeger"
BAZEL_MACOS_OPTIONS="$BAZEL_OPTIONS_ASYNC --features=-supports_dynamic_linker --build_tag_filters=-jaeger"
BAZEL_MACOS_TEST_OPTIONS="$BAZEL_MACOS_OPTIONS --test_output=errors"

BAZEL_STARTUP_OPTIONS="--output_user_root=$HOME/.cache/bazel"
Expand All @@ -81,6 +84,7 @@ if [[ "$1" == "cmake.test" ]]; then
-DWITH_METRICS_PREVIEW=ON \
-DWITH_LOGS_PREVIEW=ON \
-DCMAKE_CXX_FLAGS="-Werror" \
-DWITH_ASYNC_EXPORT_PREVIEW=ON \
"${SRC_DIR}"
make
make test
Expand All @@ -92,6 +96,7 @@ elif [[ "$1" == "cmake.abseil.test" ]]; then
-DWITH_METRICS_PREVIEW=ON \
-DWITH_LOGS_PREVIEW=ON \
-DCMAKE_CXX_FLAGS="-Werror" \
-DWITH_ASYNC_EXPORT_PREVIEW=ON \
-DWITH_ABSEIL=ON \
"${SRC_DIR}"
make
Expand All @@ -102,6 +107,7 @@ elif [[ "$1" == "cmake.c++20.test" ]]; then
rm -rf *
cmake -DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_CXX_FLAGS="-Werror" \
-DWITH_ASYNC_EXPORT_PREVIEW=ON \
-DCMAKE_CXX_STANDARD=20 \
"${SRC_DIR}"
make
Expand All @@ -114,6 +120,7 @@ elif [[ "$1" == "cmake.c++20.stl.test" ]]; then
-DWITH_METRICS_PREVIEW=ON \
-DWITH_LOGS_PREVIEW=ON \
-DCMAKE_CXX_FLAGS="-Werror" \
-DWITH_ASYNC_EXPORT_PREVIEW=ON \
-DWITH_STL=ON \
"${SRC_DIR}"
make
Expand Down Expand Up @@ -141,6 +148,7 @@ elif [[ "$1" == "cmake.legacy.exporter.otprotocol.test" ]]; then
cmake -DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_CXX_STANDARD=11 \
-DWITH_OTLP=ON \
-DWITH_ASYNC_EXPORT_PREVIEW=ON \
"${SRC_DIR}"
grpc_cpp_plugin=`which grpc_cpp_plugin`
proto_make_file="CMakeFiles/opentelemetry_proto.dir/build.make"
Expand All @@ -153,6 +161,7 @@ elif [[ "$1" == "cmake.exporter.otprotocol.test" ]]; then
rm -rf *
cmake -DCMAKE_BUILD_TYPE=Debug \
-DWITH_OTLP=ON \
-DWITH_ASYNC_EXPORT_PREVIEW=ON \
"${SRC_DIR}"
grpc_cpp_plugin=`which grpc_cpp_plugin`
proto_make_file="CMakeFiles/opentelemetry_proto.dir/build.make"
Expand All @@ -161,8 +170,8 @@ elif [[ "$1" == "cmake.exporter.otprotocol.test" ]]; then
cd exporters/otlp && make test
exit 0
elif [[ "$1" == "bazel.with_abseil" ]]; then
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS --//api:with_abseil=true //...
bazel $BAZEL_STARTUP_OPTIONS test $BAZEL_TEST_OPTIONS --//api:with_abseil=true //...
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS_ASYNC --//api:with_abseil=true //...
bazel $BAZEL_STARTUP_OPTIONS test $BAZEL_TEST_OPTIONS_ASYNC --//api:with_abseil=true //...
exit 0
elif [[ "$1" == "cmake.test_example_plugin" ]]; then
# Build the plugin
Expand Down Expand Up @@ -202,6 +211,10 @@ elif [[ "$1" == "bazel.test" ]]; then
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS //...
bazel $BAZEL_STARTUP_OPTIONS test $BAZEL_TEST_OPTIONS //...
exit 0
elif [[ "$1" == "bazel.with_async_export" ]]; then
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS_ASYNC //...
bazel $BAZEL_STARTUP_OPTIONS test $BAZEL_TEST_OPTIONS_ASYNC //...
exit 0
elif [[ "$1" == "bazel.benchmark" ]]; then
run_benchmarks
exit 0
Expand All @@ -212,28 +225,28 @@ elif [[ "$1" == "bazel.macos.test" ]]; then
elif [[ "$1" == "bazel.legacy.test" ]]; then
# we uses C++ future and async() function to test the Prometheus Exporter functionality,
# that make this test always fail. ignore Prometheus exporter here.
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS -- //... -//exporters/otlp/... -//exporters/prometheus/...
bazel $BAZEL_STARTUP_OPTIONS test $BAZEL_TEST_OPTIONS -- //... -//exporters/otlp/... -//exporters/prometheus/...
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS_ASYNC -- //... -//exporters/otlp/... -//exporters/prometheus/...
bazel $BAZEL_STARTUP_OPTIONS test $BAZEL_TEST_OPTIONS_ASYNC -- //... -//exporters/otlp/... -//exporters/prometheus/...
exit 0
elif [[ "$1" == "bazel.noexcept" ]]; then
# there are some exceptions and error handling code from the Prometheus and Jaeger Clients
# that make this test always fail. ignore Prometheus and Jaeger exporters in the noexcept here.
bazel $BAZEL_STARTUP_OPTIONS build --copt=-fno-exceptions --build_tag_filters=-jaeger $BAZEL_OPTIONS -- //... -//exporters/prometheus/... -//exporters/jaeger/...
bazel $BAZEL_STARTUP_OPTIONS test --copt=-fno-exceptions --build_tag_filters=-jaeger $BAZEL_TEST_OPTIONS -- //... -//exporters/prometheus/... -//exporters/jaeger/...
bazel $BAZEL_STARTUP_OPTIONS build --copt=-fno-exceptions --build_tag_filters=-jaeger $BAZEL_OPTIONS_ASYNC -- //... -//exporters/prometheus/... -//exporters/jaeger/...
bazel $BAZEL_STARTUP_OPTIONS test --copt=-fno-exceptions --build_tag_filters=-jaeger $BAZEL_TEST_OPTIONS_ASYNC -- //... -//exporters/prometheus/... -//exporters/jaeger/...
exit 0
elif [[ "$1" == "bazel.asan" ]]; then
bazel $BAZEL_STARTUP_OPTIONS test --config=asan $BAZEL_TEST_OPTIONS //...
bazel $BAZEL_STARTUP_OPTIONS test --config=asan $BAZEL_TEST_OPTIONS_ASYNC //...
exit 0
elif [[ "$1" == "bazel.tsan" ]]; then
bazel $BAZEL_STARTUP_OPTIONS test --config=tsan $BAZEL_TEST_OPTIONS //...
bazel $BAZEL_STARTUP_OPTIONS test --config=tsan $BAZEL_TEST_OPTIONS_ASYNC //...
exit 0
elif [[ "$1" == "bazel.valgrind" ]]; then
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS //...
bazel $BAZEL_STARTUP_OPTIONS test --run_under="/usr/bin/valgrind --leak-check=full --error-exitcode=1 --suppressions=\"${SRC_DIR}/ci/valgrind-suppressions\"" $BAZEL_TEST_OPTIONS //...
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS_ASYNC //...
bazel $BAZEL_STARTUP_OPTIONS test --run_under="/usr/bin/valgrind --leak-check=full --error-exitcode=1 --suppressions=\"${SRC_DIR}/ci/valgrind-suppressions\"" $BAZEL_TEST_OPTIONS_ASYNC //...
exit 0
elif [[ "$1" == "benchmark" ]]; then
[ -z "${BENCHMARK_DIR}" ] && export BENCHMARK_DIR=$HOME/benchmark
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS -c opt -- \
bazel $BAZEL_STARTUP_OPTIONS build $BAZEL_OPTIONS_ASYNC -c opt -- \
$(bazel query 'attr("tags", "benchmark_result", ...)')
echo ""
echo "Benchmark results in $BENCHMARK_DIR:"
Expand Down
Loading