Profiling

[ericsnowcurrently]

Ideally our optimization work will be guided by profiling data. There are a number of factors to sort out in this area:

• results already posted by other parties
• tooling
• targets (e.g. C-level eval loop code vs. opcode hits vs. ???)
• workloads

[gvanrossum]

One avenue we could try is dtrace. IIRC it's supported on Linix and macOS and there's already some code to deal with it.

On Windows I've been recommended ETW (Event Tracing for Windows).

[ericsnowcurrently]

Per https://stackoverflow.com/q/4141307:

• gprof - CPY "hotspot" profiler
• gperftools (AKA Google Performance Tools) - includes a "sampling profiler"
• "random pausing technique"
• callgrind - part of valgrind

[ericsnowcurrently]

https://instagram-engineering.com/profiling-cpython-at-instagram-89d4cbeeb898

[markshannon]

There are two types of information that are important; timings and event counts.

Timings are best got from a statistical/sampling profiler.
Hardware and O/S events can be gathered by something like perf (or the Windows equivalent).
Instrumenting profilers like gprof or cProfile distort the results too much.

VM events counts (like allocation counts, bytecode instructions executed, etc.) are not modified by instrumentation,
so are easier to gather reliably.
Since the events are specific to CPython, we can't use general purpose tools, but we can use GP formats and tools to display them.
IMO, such instrumentation is best inserted by the interpreter generator, see #5

[ericsnowcurrently]

VM events counts (like allocation counts, bytecode instructions executed, etc.) are not modified by instrumentation,
so are easier to gather reliably.
Since the events are specific to CPython, we can't use general purpose tools, but we can use GP formats and tools to display them.

By "GP formats" do you mean gprof?

Also, what would we use for instrumentation? I'm guessing there's a library that would keep things simple (and efficient) for us.

IMO, such instrumentation is best inserted by the interpreter generator, see #5

👍

[gvanrossum]

Also, what would we use for instrumentation? I'm guessing there's a library that would keep things simple (and efficient) for us.

IIUC mostly we just have to keep track of our own counters. The interesting stuff is dumping those counters in some format to disk, at a convenient location, and tools to browse the counters. For Windows, ETW should be able to do this (it dumps via the OS kernel, not sure where the events end up being logged physically, but you can extract them to a file later). But we could also write our own Python script that extracts an array of counters and dumps them in a file indicated via an environment variable, sort of like we do for dxpairs.

[gvanrossum]

Curious whether Austin might be useful here. It's billed as a sampling profiler for Python programs, but it's pure C code.

[ericsnowcurrently]

Also, @gramster mentioned bprof to me and indicated that @jakebailey may have some good insight on it.

[jakebailey]

May have misheard; I use https://github.com/google/pprof (mainly for Go and Node), which does have a C/C++ library I haven't tried out (but is used for profiling chromium, for example).

[ericsnowcurrently]

Ah, I definitely misheard. 😄

[sweeneyde]

The Coz causal profiler looks interesting. According to the related paper, the authors used it to great effect in identifying a performance bottleneck in sqlite. Related StrangeLoop Conference talk Youtube Video

You build with -ldl and then add COZ_PROGRESS_NAMED("name for progress point") in a few spots in the c code (make a Python wrapper for this and insert it in benchmarks?), then apparently it points you to lines (not just the ones you marked IIUC), and it can make a graph of "optimizing this bit by x% improves overall performance by y%". They seem to be advertising it as uniquely useful for multithreaded apps, but it looks like they also talk about single-threaded uses. Disclaimer: I've never used this before.

[gvanrossum]

@brandtbucher tried coz last week and reported that it flagged lines for which we couldn't understand what would be wrong with them. We concluded that it's probably not useful for us (the example from the talk certainly sounds like it's more helpful for multi-threaded apps). If you're interested in proving us wrong, please give it a try and report back here!