Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: run fake upstream initialization from the dispatcher thread #22856

Merged
merged 1 commit into from
Aug 29, 2022

Conversation

benjaminp
Copy link
Contributor

//test/integration:tcp_proxy_odcds_integration_test was observed to fail as follows:

==================== Test output for //test/integration:tcp_proxy_odcds_integration_test:
[==========] Running 24 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 24 tests from IpVersionsClientType/TcpProxyOdcdsIntegrationTest
[ RUN      ] IpVersionsClientType/TcpProxyOdcdsIntegrationTest.SingleTcpClient/0
[2022-08-25 20:22:46.750][3969][critical][assert] [test/integration/fake_upstream.cc:832] assert failure: !dispatcher_->isThreadSafe().
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x6b00000f81
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 0/1.24.0-dev/test/DEBUG/BoringSSL
[2022-08-25 20:22:46.773][3969][critical][backtrace] [./source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x4c46a78]->[0x2cf2a78] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1267
[2022-08-25 20:22:46.773][3969][critical][backtrace] [./source/server/backtrace.h:96] #1: __restore_rt [0x7ffbdaa79420]->[0x7ffbd8b25420] ??:0
[2022-08-25 20:22:46.802][3969][critical][backtrace] [./source/server/backtrace.h:96] #2: Envoy::FakeUpstream::assertPendingConnectionsEmpty() [0x245bf0b]->[0x507f0b] ??:0
[2022-08-25 20:22:46.846][3969][critical][backtrace] [./source/server/backtrace.h:96] #3: Envoy::(anonymous namespace)::TcpProxyOdcdsIntegrationTest_SingleTcpClient_Test::TestBody() [0x1f596cb]->[0x56cb] ??:0
[2022-08-25 20:22:46.877][3969][critical][backtrace] [./source/server/backtrace.h:96] #4: testing::internal::HandleSehExceptionsInMethodIfSupported<>() [0x588b61b]->[0x393761b] ??:0
[2022-08-25 20:22:46.924][3969][critical][backtrace] [./source/server/backtrace.h:96] #5: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x587c2bd]->[0x39282bd] ??:0
[2022-08-25 20:22:46.966][3969][critical][backtrace] [./source/server/backtrace.h:96] #6: testing::Test::Run() [0x5864ba3]->[0x3910ba3] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:0
[2022-08-25 20:22:47.003][3969][critical][backtrace] [./source/server/backtrace.h:96] #7: testing::TestInfo::Run() [0x586576a]->[0x391176a] external/com_google_absl/absl/container/internal/raw_hash_set.h:1259
[2022-08-25 20:22:47.037][3969][critical][backtrace] [./source/server/backtrace.h:96] #8: testing::TestSuite::Run() [0x5865fbb]->[0x3911fbb] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_algo.h:1925
[2022-08-25 20:22:47.089][3969][critical][backtrace] [./source/server/backtrace.h:96] #9: testing::internal::UnitTestImpl::RunAllTests() [0x5874a28]->[0x3920a28] envoy/registry/registry.h:509
[2022-08-25 20:22:47.114][3969][critical][backtrace] [./source/server/backtrace.h:96] #10: testing::internal::HandleSehExceptionsInMethodIfSupported<>() [0x588ddbb]->[0x3939dbb] envoy/registry/registry.h:0
[2022-08-25 20:22:47.160][3969][critical][backtrace] [./source/server/backtrace.h:96] #11: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x587e683]->[0x392a683] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unique_ptr.h:154
[2022-08-25 20:22:47.191][3969][critical][backtrace] [./source/server/backtrace.h:96] #12: testing::UnitTest::Run() [0x5874568]->[0x3920568] envoy/registry/registry.h:508
[2022-08-25 20:22:47.237][3969][critical][backtrace] [./source/server/backtrace.h:96] #13: RUN_ALL_TESTS() [0x4878d51]->[0x2924d51] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1203
[2022-08-25 20:22:47.284][3969][critical][backtrace] [./source/server/backtrace.h:96] #14: Envoy::TestRunner::RunTests() [0x48783b1]->[0x29243b1] external/com_google_googletest/googlemock/include/gmock/gmock-actions.h:485
[2022-08-25 20:22:47.316][3969][critical][backtrace] [./source/server/backtrace.h:96] #15: main [0x4874c3a]->[0x2920c3a] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1181
[2022-08-25 20:22:47.316][3969][critical][backtrace] [./source/server/backtrace.h:96] #16: __libc_start_main [0x7ffbda897083]->[0x7ffbd8943083] ??:0
================================================================================

This is due to the race described by #22855. Making sure the dispatcher thread is running before starting the test avoids this problem.

Signed-off-by: Benjamin Peterson benjamin@engflow.com

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

`//test/integration:tcp_proxy_odcds_integration_test` was observed to fail as follows:
```
==================== Test output for //test/integration:tcp_proxy_odcds_integration_test:
[==========] Running 24 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 24 tests from IpVersionsClientType/TcpProxyOdcdsIntegrationTest
[ RUN      ] IpVersionsClientType/TcpProxyOdcdsIntegrationTest.SingleTcpClient/0
[2022-08-25 20:22:46.750][3969][critical][assert] [test/integration/fake_upstream.cc:832] assert failure: !dispatcher_->isThreadSafe().
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x6b00000f81
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2022-08-25 20:22:46.752][3969][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 0/1.24.0-dev/test/DEBUG/BoringSSL
[2022-08-25 20:22:46.773][3969][critical][backtrace] [./source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x4c46a78]->[0x2cf2a78] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1267
[2022-08-25 20:22:46.773][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#1: __restore_rt [0x7ffbdaa79420]->[0x7ffbd8b25420] ??:0
[2022-08-25 20:22:46.802][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#2: Envoy::FakeUpstream::assertPendingConnectionsEmpty() [0x245bf0b]->[0x507f0b] ??:0
[2022-08-25 20:22:46.846][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#3: Envoy::(anonymous namespace)::TcpProxyOdcdsIntegrationTest_SingleTcpClient_Test::TestBody() [0x1f596cb]->[0x56cb] ??:0
[2022-08-25 20:22:46.877][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#4: testing::internal::HandleSehExceptionsInMethodIfSupported<>() [0x588b61b]->[0x393761b] ??:0
[2022-08-25 20:22:46.924][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#5: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x587c2bd]->[0x39282bd] ??:0
[2022-08-25 20:22:46.966][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#6: testing::Test::Run() [0x5864ba3]->[0x3910ba3] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:0
[2022-08-25 20:22:47.003][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#7: testing::TestInfo::Run() [0x586576a]->[0x391176a] external/com_google_absl/absl/container/internal/raw_hash_set.h:1259
[2022-08-25 20:22:47.037][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#8: testing::TestSuite::Run() [0x5865fbb]->[0x3911fbb] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_algo.h:1925
[2022-08-25 20:22:47.089][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#9: testing::internal::UnitTestImpl::RunAllTests() [0x5874a28]->[0x3920a28] envoy/registry/registry.h:509
[2022-08-25 20:22:47.114][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#10: testing::internal::HandleSehExceptionsInMethodIfSupported<>() [0x588ddbb]->[0x3939dbb] envoy/registry/registry.h:0
[2022-08-25 20:22:47.160][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#11: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x587e683]->[0x392a683] /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unique_ptr.h:154
[2022-08-25 20:22:47.191][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#12: testing::UnitTest::Run() [0x5874568]->[0x3920568] envoy/registry/registry.h:508
[2022-08-25 20:22:47.237][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#13: RUN_ALL_TESTS() [0x4878d51]->[0x2924d51] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1203
[2022-08-25 20:22:47.284][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#14: Envoy::TestRunner::RunTests() [0x48783b1]->[0x29243b1] external/com_google_googletest/googlemock/include/gmock/gmock-actions.h:485
[2022-08-25 20:22:47.316][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#15: main [0x4874c3a]->[0x2920c3a] external/com_google_googletest/googlemock/include/gmock/gmock-spec-builders.h:1181
[2022-08-25 20:22:47.316][3969][critical][backtrace] [./source/server/backtrace.h:96] envoyproxy#16: __libc_start_main [0x7ffbda897083]->[0x7ffbd8943083] ??:0
================================================================================
```

This is due to the race described by envoyproxy#22855. Making sure the dispatcher thread is running before starting the test avoids this problem.

Signed-off-by: Benjamin Peterson <benjamin@engflow.com>
@mattklein123 mattklein123 self-assigned this Aug 29, 2022
Copy link
Member

@mattklein123 mattklein123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, thank you.

@mattklein123
Copy link
Member

/retest

@repokitteh-read-only
Copy link

Retrying Azure Pipelines:
Check envoy-presubmit didn't fail.

🐱

Caused by: a #22856 (comment) was created by @mattklein123.

see: more, trace.

@mattklein123 mattklein123 merged commit 97af8e5 into envoyproxy:main Aug 29, 2022
@benjaminp benjaminp deleted the fake-upstream-race branch August 29, 2022 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants