Provide a standalone Rally installer

[pquentin]

Installing Rally is easy if you're familiar with Python. If you're not, things can get confusing quickly. Many of our users, including Kibana developers, could not care less about Python, and would prefer having a simple executable that embeds CPython and all dependencies. The gold standard here is Go, where binaries are how you ship your code. But we're not going to rewrite Rally in Go :) Instead, we want an installer like the one provided by awscli v2: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html. You just have to unzip and run a simple install script to use it.

Sharp edges

I asked a friend that maintains such an installer. He prefers not to be named, but here's what he had to share (thanks!):

Yeah, it's pyinstaller under the hood. That's been the most straightforward option.

I can give you the sharp edges we've found so far to hopefully save yourself some pain.

1. AV Vendors do not like when you upgrade PyInstaller. Every time we've done that, it's required going to Microsoft and other vendors to have the allowlist the new binary. So pick a recent version and plan to stay on it for a while.
2. Given that you're bundling a Python interpreter, you may need to upgrade and backport changes in the event of a CPython CVE. We learned the hard way you want this to be fast and easy.
3. Linux and Windows are pretty straight forward, but distributing the macOS binary had a somewhat steep learning curve. Mostly around signing, using the right base OS. Elastic may already have those pieces setup though.

(I think only 2 and 3 apply to us here.)

Another thing to watch out for: to provide both esrally and esrallyd, we'll need to use multipackage support: https://pyinstaller.readthedocs.io/en/stable/spec-files.html#multipackage-bundles.

Platforms

We don't support Windows, but we want to support developers on Linux and macOS, as well as running on Linux servers (including ARM). This means we would ideally like to provide packages for:

• Linux x86_64
• Linux aarch64
• macOS x86_64
• macOS arm64

However, since pyinstaller is not a cross-compiler, this means we'll need a macOS environment. A good way to automate this is to use GitHub Actions. Since GitHub Actions do not currently provide ARM-based Macs, that means we would only be providing a x86_64 executable. This could be good enough, since ARM-based Macs can run x86_64 executables with Rosetta 2. Not sure how we should go about signing the executable. We can probably start with ad-hoc signing.

We'll have a similar problem for Linux ARM, but at least it's easy to get our hands on a Graviton instance.

[pquentin]

PyOxidizer sounds like a more promising approach than PyInstaller, see https://hynek.me/til/python-portable-binaries/

[pquentin]

I started experimenting with PyOxidizer yesterday. It supports Python 3.8, 3.9 and 3.10, just like us. It's unclear how long we will have to wait to get 3.11. Many projects that have to deal with the low-level parts of CPython have difficulties supporting Python 3.11, but I'm not sure if it affects PyOxidizer in the same way. Issues I've faced so far:

• Rally uses __file__ in a few places which is not supported, see Stop requiring __file__ in Python packages indygreg/PyOxidizer#69 for details
• We depend on aiohttp which uses Cython, which required me to opt out of the static builds: Static builds don't work with Cython indygreg/PyOxidizer#567
• We depend on jsonschema that apparently uses pkgutil.get_data incorrectly but in a way that works on CPython: OxidizedFinder not loading file based resource data from folders that aren't Python packages indygreg/PyOxidizer#436

[pquentin]

Interesting pull request suggesting additional difficulties on macOS: splitgraph/sgr#656