Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DocDB] Optimize stack trace collection and symbolization #19085

Open
1 task done
mbautin opened this issue Sep 11, 2023 · 0 comments
Open
1 task done

[DocDB] Optimize stack trace collection and symbolization #19085

mbautin opened this issue Sep 11, 2023 · 0 comments
Assignees
Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue

Comments

@mbautin
Copy link
Contributor

mbautin commented Sep 11, 2023

Jira Link: DB-7896

Description

To collect a stack trace (an array of raw program counters) we currently have the following ways:

  • StackTrace::Collect
    • Uses google::GetStackTrace in ASAN/TSAN
    • Uses the standard backtrace function (not to be confused with libbacktrace) otherwise

To symbolize a stack trace (convert program counters to function names as well as file names / line numbers, if possible) in StackTrace::Symbolize we do one of the following:

There is also absl::Symbolize which is much faster than google::Symbolize according to @SrivastavaAnubhav. We should prefer to use that function.

libbacktrace is a library that has some issues (internal deadlocks under some circumstances) so we don't use it in release by default. It is controlled by the FLAGS_libbacktrace, which is true by default in debug mode and false in release mode on Linux, and always false on macOS. libbacktrace is only supported on Linux.

Also, we have a separate implementation of a combined collect + symbolize function called GetStackTrace:

We need to clean up this inconsistency and make GetStackTrace() the same as StackTrace::Collect() followed by Symbolize().

A proposed format for stack traces is as follows (using function names provided by absl::Symbolize):

#1  | 0x55f482394ccc | yb/util/debug-util-test.cc:80                  | DebugUtilTest_TestStackTrace2_Test::TestBody()
#2  | 0x7fb8c29fdb8c | googletest-1.12.1/googletest/src/gtest.cc:2599 | testing::internal::HandleExceptionsInMethodIfSupported<>()
#3  | 0x7fb8c29fdb8c | googletest-1.12.1/googletest/src/gtest.cc:2635 | testing::internal::HandleExceptionsInMethodIfSupported<>()
#4  | 0x7fb8c29e3d87 | googletest-1.12.1/googletest/src/gtest.cc:2674 | testing::Test::Run()
#5  | 0x7fb8c29e4da6 | googletest-1.12.1/googletest/src/gtest.cc:2853 | testing::TestInfo::Run()
#6  | 0x7fb8c29e5a1e | googletest-1.12.1/googletest/src/gtest.cc:3012 | testing::TestSuite::Run()
#7  | 0x7fb8c29f51cd | googletest-1.12.1/googletest/src/gtest.cc:5870 | testing::internal::UnitTestImpl::RunAllTests()
#8  | 0x7fb8c29fe86c | googletest-1.12.1/googletest/src/gtest.cc:2599 | testing::internal::HandleExceptionsInMethodIfSupported<>()
#9  | 0x7fb8c29fe86c | googletest-1.12.1/googletest/src/gtest.cc:2635 | testing::internal::HandleExceptionsInMethodIfSupported<>()
#10 | 0x7fb8c29f4cdf | googletest-1.12.1/googletest/src/gtest.cc:5444 | testing::UnitTest::Run()
#11 | 0x7fb8c2f6a760 | uninstrumented/include/gtest/gtest.h:2293      | RUN_ALL_TESTS()
#12 | 0x7fb8c2f6a132 | yb/util/test_main.cc:110                       | main
#13 | 0x7fb8c0942cf2 | :0                                             | __libc_start_main
#14 | 0x55f482394bad | :0                                             | _start

Also, libbacktrace produces really long function names/signatures with all the template arguments, and the resulting stack traces become unreadable. absl::Symbolize produces much more reasonable function names/signatures.

Warning: Please confirm that this issue does not contain any sensitive information

  • I confirm this issue does not contain any sensitive information.
@mbautin mbautin added area/docdb YugabyteDB core features status/awaiting-triage Issue awaiting triage labels Sep 11, 2023
@mbautin mbautin self-assigned this Sep 11, 2023
@yugabyte-ci yugabyte-ci added kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue labels Sep 11, 2023
@yugabyte-ci yugabyte-ci removed the status/awaiting-triage Issue awaiting triage label Sep 19, 2023
mbautin added a commit to yugabyte/yugabyte-db-thirdparty that referenced this issue Nov 2, 2023
Also some improvements in handling the `--dev-repo` flag.

Issue: yugabyte/yugabyte-db#19085
es1024 added a commit that referenced this issue Nov 7, 2023
Summary:
Upgrade OpenSSL to 3.0.8 as 1.1.1 has reached EOL.

Also pulls in the following thirdparty changes:
- Disable Linuxbrew builds (yugabyte/yugabyte-db-thirdparty@55eee2b)
- Update glog to use stack unwinding based on backtrace function (yugabyte/yugabyte-db-thirdparty@ec8ab75, #19085)
Jira: DB-8566

Test Plan: Jenkins

Reviewers: mbautin, rthallam

Reviewed By: rthallam

Subscribers: rthallam, yql, ybase

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D29701
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue
Projects
None yet
Development

No branches or pull requests

2 participants