Code

Significantly better OOM detection#110

Code

Merged

itamarst merged 60 commits intomasterpythonspeed/filprofiler:masterfrom 96-better-oompythonspeed/filprofiler:96-better-oomCopy head branch name to clipboardJan 12, 2021

+803-101

Commits

Commits on Nov 25, 2020

• A slow leak causing OOM, a situation that isn't handled by naive solution.

  itamarst

  committed
  319e090

  Copy full SHA for 319e090

Commits on Nov 28, 2020

• Merge remote-tracking branch 'origin/master' into 96-better-oom

  itamarst

  committed
  419783b

  Copy full SHA for 419783b

Commits on Dec 10, 2020

• Merge remote-tracking branch 'origin/master' into 96-better-oom

  itamarst

  committed
  1e92b6d

  Copy full SHA for 1e92b6d

Commits on Dec 11, 2020

• Sketch of improved out-of-memory interface.

  itamarst

  committed
  a96e19a

  Copy full SHA for a96e19a

• Move tracking on/off switch into Rust.

  itamarst

  committed
  592da20

  Copy full SHA for 592da20

Commits on Dec 18, 2020

• Rewrite of OOM checks to also check for free memory.

  itamarst

  committed
  ef8c1dc

  Copy full SHA for ef8c1dc

• Test for slow out-of-memory case.

  itamarst

  committed
  3bd5c0c

  Copy full SHA for 3bd5c0c

• Another TODO.

  itamarst

  committed
  f7bdd21

  Copy full SHA for f7bdd21

• Not going to do that here.

  itamarst

  committed
  55b6679

  Copy full SHA for 55b6679

Commits on Dec 29, 2020

• Merge remote-tracking branch 'origin/master' into 96-better-oom

  itamarst

  committed
  8b9b483

  Copy full SHA for 8b9b483

• Some microoptimizations.

  itamarst

  committed
  9e3d2d4

  Copy full SHA for 9e3d2d4

Commits on Dec 31, 2020

• setrlimit memory limits (attempted).

  itamarst

  committed
  2c1e082

  Copy full SHA for 2c1e082

• Rip our rlimit support, it's too painful for users.

  itamarst

  committed
  874eb72

  Copy full SHA for 874eb72

• Refactor OOM support into its own module.

  itamarst

  committed
  6913ca3

  Copy full SHA for 6913ca3

• A failing test (and placeholder sketch) for cgroup memory limits.

  itamarst

  committed
  0fde1ab

  Copy full SHA for 0fde1ab

• Sketch of working cgroup OOM detection.

  itamarst

  committed
  d1f2f9e

  Copy full SHA for d1f2f9e

• Correct OS.

  itamarst

  committed
  23d37c0

  Copy full SHA for 23d37c0

Commits on Jan 1, 2021

• Disable code for now.

  itamarst

  committed
  66d5a50

  Copy full SHA for 66d5a50

• Try using library for cgroup parsing.

  itamarst

  committed
  db82e93

  Copy full SHA for db82e93

• Don't break macOS.

  itamarst

  committed
  29ed0cc

  Copy full SHA for 29ed0cc

• Fix syntax.

  itamarst

  committed
  deff2d7

  Copy full SHA for deff2d7

• Try to fix the tests for cgroups v1.

  itamarst

  committed
  0fa7c90

  Copy full SHA for 0fa7c90

• Fix bug introduced by merge.

  itamarst

  committed
  87bee56

  Copy full SHA for 87bee56

• Try to fix sudo.

  itamarst

  committed
  94b8e23

  Copy full SHA for 94b8e23

• Try to fix cgroups v1.

  itamarst

  committed
  d7de72a

  Copy full SHA for d7de72a

• Preserve directory.

  itamarst

  committed
  bc96721

  Copy full SHA for bc96721

• Pass to correct program.

  itamarst

  committed
  ab8d494

  Copy full SHA for ab8d494

Commits on Jan 2, 2021

• Dependencies.

  itamarst

  committed
  5e35cc5

  Copy full SHA for 5e35cc5

• getenv in malloc() code path is a bad idea, since it leads to reentrancy with

  Show description for 149a05c
  itamarst

  committed
  149a05c

  Copy full SHA for 149a05c

Commits on Jan 3, 2021

• Initialize Rust before Rust starts getting called for every malloc().

  itamarst

  committed
  8425ce3

  Copy full SHA for 8425ce3

• Revert tracking-switch in Rust.

  itamarst

  committed
  a499d34

  Copy full SHA for a499d34

Commits on Jan 4, 2021

• Garbage.

  itamarst

  committed
  6ddcceb

  Copy full SHA for 6ddcceb

Commits on Jan 5, 2021

• A basic test for allocation size being too big.

  itamarst

  committed
  28ec820

  Copy full SHA for 28ec820

• Basic test for OutOfMemoryEstimator.are_we_oom().

  itamarst

  committed
  a60e5af

  Copy full SHA for a60e5af

Commits on Jan 7, 2021

• Implemented cgroups already.

  itamarst

  committed
  80c2bce

  Copy full SHA for 80c2bce

Commits on Jan 8, 2021

• Check for OOM less frequently when under memory pressure.

  itamarst

  committed
  73c87d2

  Copy full SHA for 73c87d2

• TODO: A better idea for handling large allocations that use up the safety buffer.

  itamarst

  committed
  64ddf7f

  Copy full SHA for 64ddf7f

• Free up some more memory safely during OOM.

  itamarst

  committed
  b6a80eb

  Copy full SHA for b6a80eb

• Not going to do a checking thread for now.

  itamarst

  committed
  e9e7f77

  Copy full SHA for e9e7f77

• Probably fine, real focus should be on reducing memory usage of dumping SVG.

  itamarst

  committed
  cb06447

  Copy full SHA for cb06447

• Runnable benchmark for getting cgroup availability.

  itamarst

  committed
  47b000f

  Copy full SHA for 47b000f

• Speed up get cgroup memory availability, at the cost of not supporting changes

  Show description for 23b67e3
  itamarst

  committed
  23b67e3

  Copy full SHA for 23b67e3

• Fix compilation for benchmarks.

  itamarst

  committed
  08b22c2

  Copy full SHA for 08b22c2

Commits on Jan 9, 2021

• Use a struct+trait instead of a function pointer, for easier changes.

  itamarst

  committed
  8617f7e

  Copy full SHA for 8617f7e

• Add swapping heuristic for macOS.

  itamarst

  committed
  775f596

  Copy full SHA for 775f596

• Temporary disable unit tests.

  itamarst

  committed
  54df809

  Copy full SHA for 54df809

• Disable.

  itamarst

  committed
  2d57750

  Copy full SHA for 2d57750

Commits on Jan 10, 2021

• Resurrect unit tests and add test for swap heuristic.

  itamarst

  committed
  cc0d896

  Copy full SHA for cc0d896

• Get rid of global state.

  itamarst

  committed
  2e95297

  Copy full SHA for 2e95297

• Cache the current process, for a small speedup.

  itamarst

  committed
  d2d1a57

  Copy full SHA for d2d1a57

• Re-run benchmarks.

  itamarst

  committed
  d54bf80

  Copy full SHA for d54bf80

• Update documentation.

  itamarst

  committed
  48e6680

  Copy full SHA for 48e6680

• Try cgroups v2 testing.

  itamarst

  committed
  77cb38b

  Copy full SHA for 77cb38b

• Activate venv.

  itamarst

  committed
  89486d2

  Copy full SHA for 89486d2

• Give up on cgroups v2 testing for now.

  itamarst

  committed
  55026b6

  Copy full SHA for 55026b6

Commits on Jan 12, 2021

• Get rid of unnecessary and misleading exit.

  itamarst

  committed
  f4e3c08

  Copy full SHA for f4e3c08

• Better documentation of out-of-memory detection.

  itamarst

  committed
  c0adf28

  Copy full SHA for c0adf28

• Documentation corrections and improvements.

  itamarst

  committed
  4a5366f

  Copy full SHA for 4a5366f

• Try upgrading pip.

  itamarst

  committed
  376d17e

  Copy full SHA for 376d17e

• Try upgrading setuptools too.

  itamarst

  committed
  8029559

  Copy full SHA for 8029559