Hoist Artifacts

Hoist

Hoist is the name for the deployment system used at Square, intended to be replaced by p2. One concept that will be carried along into p2 to make migration easier is the specification of a "deployable artifact", which is a .tar.gz file that hoist knows how to deploy. p2 uses "hoist artifact" as a type of launchable that may be included in a pod.

Hoist Artifact Specification

As mentioned above, a hoist artifact is a .tar.gz file. This file has a few contracts that will allow contents of the artifact to influence the deploy process. Most of this magic happens within a bin/ subdirectory in the root of the tarball.

An artifact man (and usually does) provide a file or directory named bin/launch. Any file that begins with bin/launch will be scheduled under runit to execute as a supervised process. If bin/launch is itself a file, we will expect a single runit service for this artifact (and therefore, the corresponding launchable). If bin/launch is a directory with 3 files inside it, we expect to end up with 3 runit services after a deploy.

Digest Verification

To ensure the contents of a hoist artifact are sound, you can provide a digest file that lists the SHA256 hash of every file inside the artifact. This is pretty easy to generate:

$ cd $ROOT_OF_ARTIFACT
$ find -xtype f -print0 | xargs -0 sha256sum > ../artifact.sum

You can specify the path to this digest file using the digest_location key under the corresponding launchable. When p2 unpacks the tarball, it will iterate over every single file in the tarball and compare its SHA256 hash to that of the digest. p2 will refuse to launch the artifact unless all the SHAs are the same and there are no leftover files (ie the list of files in the manifest corresponds exactly to the list of files in the tarball).

The digest is ignorant of symlinks. If you have one file in your artifact and two symlinks to it, all of them should have hashes. (That's the difference between find -type f, which skips symlinks, and find -xtype f, which follows them if they point to files.) p2 will attempt to read all three files. If one of the symlinks is broken, you will be unable to deploy because the symlink target can't be read. Make sure all your symlinks are relative and that they point to files that are actually in the artifact! (An easy way to check for absolute symlinks is tar -tzvf <tarball> | grep -- '-> /'.)

Signed Digests

Digest verification is a useful feature for ensuring tarball integrity, but for extra security, you may want to authenticate the digest itself. You can generate a digest signature like this:

$ gpg --no-armor --output my-digest.sum.sig --detach-sig my-digest.sum

The digest signature location can be specified via a digest_signature_location key under the launchable. The preparer will use the signature to determine who signed the manifest. The signer's public key must be on the preparer's keyring. If the signature is invalid, unknown or inaccessible, p2 will not attempt to verify the artifact, and it'll bail out of the deployment.