Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Single-line std::string::_Rep::_S_create stack traces in heap profiles #3116

Closed
ttyusupov opened this issue Dec 9, 2019 · 1 comment
Closed

Single-line std::string::_Rep::_S_create stack traces in heap profiles #3116

ttyusupov opened this issue Dec 9, 2019 · 1 comment
Assignees
@ttyusupov

Comments

@ttyusupov
Copy link
Contributor

Several times investigating OOM issues we noticed that there are single-line std::string::_Rep::_S_create stack traces in heap profiles which are responsible for a high amount of memory consumption.
Some related issues: #3041, #2566
@ttyusupov ttyusupov self-assigned this Dec 9, 2019
@ttyusupov
Copy link
Contributor Author

Some investigation results:

  • This is only happening in GCC builds.
  • Memory consumption by std::string is higher in GCC builds than in clang builds.
  • I was able to get a real stack trace using GDB and found that these calls are actually from std::string construction.
  • We have -D_GLIBCXX_USE_CXX11_ABI=0 build option which was causing usage of old std::string implementation. Removing that build option makes memory consumption lower and also fixes std::string::_Rep::_S_create stack traces issue in heap profiles.

We don't need to use -D_GLIBCXX_USE_CXX11_ABI=0, because we are not using any LLVM codegen in YugaByte. Also, confirmed that removing that option is not breaking backward compatibility for rolling upgrades.

@ttyusupov ttyusupov mentioned this issue Dec 9, 2019
Merged
mbautin pushed a commit to yugabyte/yugabyte-db-thirdparty that referenced this issue Dec 9, 2019
@ttyusupov @mbautin
Enable CXX11 ABI (#8)
7fc63d1 
yugabyte/yugabyte-db#3116
ttyusupov added a commit that referenced this issue Dec 12, 2019
@ttyusupov
#3116: Enable CXX11 ABI
aa35010 
Summary:
We use a `-D_GLIBCXX_USE_CXX11_ABI=0` build option which we actually don't need and this causes single-line `std::string::_Rep::_S_create` stack traces in heap profiles which are responsible for a high amount of memory consumption in some of the scenarios we've seen investigating OOM issues.

This change removes this option from both YB and third-party builds, so we will be using correct `std::string` implementation instead of the old one for GCC builds.

Test Plan:
- Jenkins
- Populated cluster with data using `java -jar ~/code/yb-sample-apps.jar --workload CassandraBatchTimeseries --nodes $NODES --num_writes 100000 --num_threads_read 1 --num_threads_write 4 --min_metrics_count 10000 --max_metrics_count 20000 --num_unique_keys 1000000000 --nouuid --value_size 100`. Performed rolling upgrade. Checked that data is kept and there are no backward compatibility errors.
- Checked local RF=3 cluster on AWS m3.xlarge with Xeon E5-2665. Tried load and query ysql sample data.

Reviewers: sergei, mikhail

Reviewed By: mikhail

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D7673
mbautin added a commit that referenced this issue Mar 19, 2020
@mbautin
Backport Jenkins-related fixes to the 2.0.5 branch
9b3c7ad 
Summary:
Backport Jenkins-related fixes to remove NFS usage to the 2.0.5 branch

Updating third-party dependencies to point to the same versions as in
the C++11 ABI change by @ttyusupov
(http://github.com/yugabyte/yugabyte-db/commit/aa35010bdd44dc7d3c5f2cb7cf58873b57402a6e):

- build-support/shared_thirdparty_version_for_jenkins_centos.txt
  (2019-12-07T23_26_21)

- build-support/shared_thirdparty_version_for_jenkins_mac.txt
  (2019-12-07T15_28_30)

- build-support/thirdparty_url_centos.txt
  (https://github.com/yugabyte/yugabyte-db-thirdparty/releases/download/v20191209181439-7fc63d1583/yugabyte-db-thirdparty-v20191209181439-7fc63d1583-centos.tar.gz)

Here is the list of commits from master that were used in this diff:

- [#3663] Remove unused macro-based DSL in pggate
Author: @mbautin
fb14bd1
https://phabricator.dev.yugabyte.com/D7940

- Fix how we choose the compiler to use in the third-party build
Author: @mbautin
44f2aee
https://phabricator.dev.yugabyte.com/D7851

- Fix yb_release after recent NFS build infra changes
Author: @bmatican
20e2073
https://phabricator.dev.yugabyte.com/D7834

- Fix build in CLion and YB_SRC_ROOT detection in Java tests
Author: @mbautin
bb6d7eb
https://phabricator.dev.yugabyte.com/D7826

- Fix run_tests_on_spark.py when run without --send_archive_to_workers
Author: @mbautin
4b0eabb
https://phabricator.dev.yugabyte.com/D7822

- #3411 Run tests on Spark without NFS
Author: @mbautin
http://github.com/yugabyte/yugabyte-db/commit/7ffccc8cf87239778bf87d36d5888297256f568b
https://phabricator.dev.yugabyte.com/D7801

- [YCQL] Disabling test org.yb.cql.TestBigNumShards for ASAN configuration
Author: @OlegLoginov
95abd92
https://phabricator.dev.yugabyte.com/D7747

- Fix ASAN build on Jenkins
Author: @mbautin
7372863
https://phabricator.dev.yugabyte.com/D7841

- #3116: Enable CXX11 ABI
Author: @ttyusupov
http://github.com/yugabyte/yugabyte-db/commit/aa35010bdd44dc7d3c5f2cb7cf58873b57402a6e
https://phabricator.dev.yugabyte.com/D7673

- [#2979] Allow controlling ccache directory location with an environment variable
Author: @mbautin
d030aa0
https://phabricator.dev.yugabyte.com/D7595

- Update CppCassandraDriverTest to use only cassandra.h
Author: @spolitov
a1572d1
https://phabricator.dev.yugabyte.com/D7553

- Automatic download of prebuilt third-party dependencies and Homebrew/Linuxbrew
Author: @mbautin
de42d48
https://phabricator.dev.yugabyte.com/D7511

Test Plan: Jenkins: auto rebase: no

Reviewers: bogdan

Reviewed By: bogdan

Subscribers: bogdan

Differential Revision: https://phabricator.dev.yugabyte.com/D7871
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ttyusupov ttyusupov

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ttyusupov