Simplify bottling for taps

[davidchall]

Without a BrewTestBot (or similar), it's labor intensive for tap maintainers to build bottles, publish them on Bintray, and update the formula with the bottle DSL. This could potentially be achieved on Travis CI using the brew test-bot and brew pull --bottle commands.

I've hacked together a partial solution by making changes only to Homebrew/brew-test-bot, but I'm looking for feedback from the experts to find a better solution (e.g. @MikeMcQuaid @sjackman).

I think there are two main points to discuss:

• best workflow without a TestBot user
• code refactoring in Homebrew/brew

Once we arrive at a solution, it can be added to brew tap-new and documented.

Suggested workflow

• Encrypted env vars: cannot upload bottles from forked PRs
• I suggest uploading bottles from master branch builds (i.e. test builds), since this is the point at which they are "accepted" into the tap
• In the absence of a TestBot user, I suggest using the owner's account as the TestBot user (creating testing-#{build_id} tags and branches)
• Exploit the new Travis CI stages, so that bottles are built and uploaded for each OS version in the first stage, and if all jobs complete successfully then a second stage runs brew pull --bottle

Code refactoring

Here are some variables that need to be edited to upload bottles and push formula changes:

• bintray-org (brew test-bot and brew pull --bottle)
• HOMEBREW_BINTRAY_USER (brew test-bot)
• HOMEBREW_BINTRAY_KEY (brew test-bot)
• git-name (brew test-bot)
• git-email (brew test-bot)
• test-bot-user (brew pull --bottle)
• BottleSpecification::DEFAULT_DOMAIN (to get correct root_url)

These are currently passed as cmdline arguments or env vars, which could instead be properties of the tap object (perhaps read from a .brew-tap.yml file in the tap repo). Of course, the HOMEBREW_BINTRAY_KEY will need to remain an encrypted env var.

These lines also hardcode where to find bottle patches. If we're going to support Travis CI push builds, then we'd need something similar.

[sjackman]

I've successfully used CircleCI to create macOS and Linux bottles for a tap:
https://github.com/Linuxbrew/homebrew-bioinformatics/blob/master/.circleci/config.yml
I created a user named LinuxbrewTestBot. If your tap were in an organization Foo, the test-bot user would be called FooTestBot and your Bintray organization would be called foo.

[MikeMcQuaid]

I suggest uploading bottles from master branch builds (i.e. test builds), since this is the point at which they are "accepted" into the tap

This would need to not just upload bottles but also push a commit with the bottle checksums.

In the absence of a TestBot user, I suggest using the owner's account as the TestBot user (creating testing-#{build_id} tags and branches)

This seems sensible 👍

These are currently passed as cmdline arguments or env vars, which could instead be properties of the tap object (perhaps read from a .brew-tap.yml file in the tap repo).

I think environment variables for these would be ideal for when they can't be inferred automatically. I agree automatic inferring would be ideal, though.

If your tap were in an organization Foo, the test-bot user would be called FooTestBot and your Bintray organization would be called foo.

I think the FooTestBot pattern makes sense in terms of overhead for orgs but not taps with a single user as maintainer.

---

As a general comment: thanks for documenting and experimenting with this @davidchall. I'd definitely like a generic process we can support for people to be able to use Travis CI and other CI providers to build bottles. I doubt this will be something that ever scales to official taps given the 60m build timeout but it should work otherwise and I'm open to reviewing and making changes to get us there.

[ilovezfs]

I'm not sure encouraging bottling in third-party taps is going to have a salutary impact on the third-party Homebrew ecosystem overall.

If there is a breaking upgrade to a library dependency of a bottle, then the bottle breaks. That means the software is broken for anyone who has already installed the bottle, and for anyone who is freshly installing the bottle. The bottle will remain broken until the tap owner gets around to building and publishing a new bottle. In the meantime anyone doing a fresh install will get a broken bottle and have to uninstall it and start over with a source install, assuming they can figure out what is going on.

By contrast, in a tap without bottles, preexisting installations will break when there is a breaking upgrade to a library dependency. But any fresh installations, since they will be source installs, will be fine, and there's no waiting on the tap owner to find out about and fix the problem.

So encouraging bottling in third-party taps will mean spreading the problems that have been endemic to the science tap more broadly, and triggering a constant cycle of break-notify-rebuild-republish-break-notify-rebuild-republish increasing the burden on tap owners and making the user experience more inconsistent.

[scpeters]

@ilovezfs I maintain the osrf/simulation tap and build bottles for it, and we have daily jenkins jobs that test the bottles, so that we know when to build new ones. For software that isn't quick to compile, even if the bottles fail 1% of the time, I think the average user experience is better.

Maybe I could complement my bottle-testing job with a job that searches github for homebrew-core pull requests that modify dependencies of our formulae so we aren't surprised when they get merged.

[ilovezfs]

we have daily jenkins jobs that test the bottles, so that we know when to build new ones.

I guess that proves the point.

[ilovezfs]

Maybe I could complement my bottle-testing job with a job that searches github for homebrew-core pull requests that modify dependencies of our formulae so we aren't surprised when they get merged.

subscribing to homebrew/core and using a mail filter is probably the simplest approach

[scpeters]

subscribing to homebrew/core and using a mail filter is probably the simplest approach

I'm doing that as we speak. It works for individuals but doesn't scale as well for teams wanting these notifications.

[scpeters]

we have daily jenkins jobs that test the bottles, so that we know when to build new ones.

I guess that proves the point.

It's true, and I agree that it's a hassle, but it's worth the hassle for me since our software takes a long time to compile. I guess I'm just saying you don't have to encourage bottling for third-party taps, but I hope you don't disable it, since we find it useful.

[ilovezfs]

Yeah, we wouldn't disable it :)

[sjackman]

If there is a breaking upgrade to a library dependency of a bottle, then the bottle breaks.

A solution that addresses some (but not all) of this breakage is…
after brew installing a bottle, automatically run brew linkage on that keg, and if it fails, fall back to installing from source. This handles renamed dynamic library files. It currently does not handle removed/renamed symbols in those libraries, though it could be extended to do that as well.

[ilovezfs]

That's pretty much papering over the problem with an inconsistent UX.

[scpeters]

I have a confession to make: I was running Yosemite on my development laptop until just a few weeks ago. I backdated my homebrew-core to a point that still had yosemite bottles for the packages I needed and set HOMEBREW_NO_AUTO_UPDATE=1 in my shell profile. I kept brew up to date with cd $(brew --repo) && git pull, and I could still install old yosemite bottles. I just had to remember to never brew up. This is not supported and no one should ask for help if they do it, but it seemed to work for me.

If 3rd-party taps could temporarily backdate homebrew-core, this would allow them to keep working until their bottles are rebuilt. I don't know how this would be implemented; I think it's possible, though it may not be desired functionality.

If this functionality was built, I could imagine brew doctor complaints if people have homebrew-core backdated.

[stek29]

Or bottles can include some metadata on what versions of dependent formulas (&their deps) they were built with, and brew install would check that metadata and fallback to source build if it doesn't match

[alyssais]

The metadata is there in INSTALL_RECEIPT.json inside the bottle, but I don’t think version checking is a good solution. Most version changes won’t require building from source, and rebuilding bottles for all dependents every version change would be unsustainable for us.

[stek29]

@alyssais placing metadata in separate file would allow checking it without downloading whole bottle. It might be placed as separate tag, so it doesn't break older brew.

Are dependent formulas rebotlled manually now?

Also, if it's just a version bump, only metadata on dependent formulas can be updated. Not ideal, but still better than rebuilding.

[stek29]

Or another option -- add some revision number to formula, which is bumped every time a breaking change requiring rebuild of dependent bottles occurs, and use that as a reference instead if version. Not sure on how to do that without surprising older brew versions though.

[alyssais]

Are dependent formulas rebotlled manually now?

Not by default, but they often are when the maintainers expect them to be broken.

Also, if it's just a version bump, only metadata on dependent formulas can be updated. Not ideal, but still better than rebuilding.

This would require a lot of manual testing if I’m understanding it right.

add some revision number

This is how we currently do this.

[MikeMcQuaid]

@stek Discourse is a better forum for questions and ideas if you've not personally written any code to improve this. Note we're not going to change the bottle file format.

[stek29]

@MikeMcQuaid I've not personally written any code to improve this, but I was planning to.
(Has anybody? It's an open issue which wasn't worked on yet, isn't it? it also has help wanted and is even listed in gsoc "ideas", so I expected it to be open for discussion relevant to the issue)

After looking through other issues, I got an impression that you've meant "don't leave comments unless you contributed to Homebrew and/or planning to work on this issue" -- if I'm right, then let me say that I did make some minor contributions to homebrew-core, and recently made a PR for brew :)

What I was suggesting is this:

• Bottle (as the tarball) format stays unchanged
• Bottle (as in Formula source) gets additional property specifying file with metadata for current bottle -- which would include copies of INSTALL_RECEIPTs of other tags
• Optional (to avoid changing all existing formulas) revision number, which is only bumped on breaking change is added to Formula (not the same as Formula.revision -- Formula.version / Formula.revision may change while keeping binary compatibility, while revision number I'm suggesting is bumped only if breaking changes requiring rebottling happen -- I'll refer to it as binary_revision)
• INSTALL_RECEIPT includes that "binary_revision" per required runtime dependency
• When installing from bottle:
  • Check for presence of metadata tag, if it exists get needed INSTALL_RECEIPT from it (if it doesn't, fallback to old behaviors (just install hoping that nothing breaks))
  • Check if binary_revisions of all installed Formulas are matching: if they are, fetch bottle and install it. If they're not, show a warning and install from source
• When checking formula's binary_revision, infer it from version / version+revision if it's not present
• Optionally require binary_revision for all new/updated formulas in future so eventually all formulas have it (unless they never update -- then they don't need it)
• When updating & rebottling a formula: add that new tag with all INSTALL_RECEIPTs to it
• When formula is updated and it's binary_revision is changed, it suggests maintainers to rebottle all dependent formulas. Actually, it could've been done automatically, but there's one thing I'm not sure about:

Problem: Dependencies of Dependencies. Let's say there's:

• A: depends on B
• B: depends on C
• C

Now, B and C can be some libraries, and so if C's binary_revision is changed, both A and B should be rebuilt.
However, if C is a library and B is some kind of language (I can't come up with any better analogy), then A probably should not be rebuilt.

There's an option -- when formula's binary_revision is changed, maintainers trigger rebuild on all dependent formulas, and optionally change their binary_revision's if "they [maintainers] expect them [dependencies of rebottled formula] to be broken".

[MikeMcQuaid]

@MikeMcQuaid I've not personally written any code to improve this, but I was planning to.

@stek29 Consider you just made two excellent PRs today: consider my comments revoked and apologised for ❤️

Optional (to avoid changing all existing formulas) revision number, which is only bumped on breaking change is added to Formula (not the same as Formula.revision -- Formula.version / Formula.revision may change while keeping binary compatibility, while revision number I'm suggesting is bumped only if breaking changes requiring rebottling happen -- I'll refer to it as binary_revision)

Bottles have a rebuild that's used if you're rebuilding the same version/revision but want a new bottle.

Bottle (as in Formula source) gets additional property specifying file with metadata for current bottle -- which would include copies of INSTALL_RECEIPTs of other tags

I'm not sure what this would be for? Can you elaborate? Checking all versions within a bottle and building from source if they don't match? There's no real way of detecting this automatically, unfortunately, and doing so manually would be a lot of work.

[stek29]

@MikeMcQuaid to avoid downloading whole bottle to only find out that it links to incompatible versions of formulas.

There's no real way of detecting this automatically, unfortunately, and doing so manually would be a lot of work

I'm suggesting adding a property "binary_revision" to formula which is manually bumped only in case of breaking changes which would require rebottling formulas depending on it.
If I understood @alyssais correctly, currently maintainers manually monitor each formula update and rebottle formulas depending on updated formula if they expect it to break.
Instead, they can bump formula's binary_revision and get all dependencies rebuilt.

However, as I noticed previously, "deps of deps" aren't that simple to handle.
I can only think of this: bump binary_revision manually on all rebottled formulas if needed, causing rebottling of their deps, do that on deps, and so on recursively.

I just discovered brew linkage, so I guess it could be used to simplify the work for maintainers.

brew linkage output is bundled in bottle metadata file along with binary_revision's.
When formula's binary_revision is bumped, maintainer runs a command, which:

• Triggers rebottling of all direct dependents of updated formula
• Goes through all dependents recursively, fetching metadata file, and checking if they link to formula which was bumped. If they do, suggests bumping that dependent's binary_revision, and marks that formula as bumped, so when going through other dependents it would suggests bumping for anything that linked to that formula.

[ilovezfs]

FYI we don't actually revision bump or want to revision bump and rebottle every recursive reverse dependency, only the ones that actually break without the revision bump.

[stek29]

@ilovezfs That's why it should "suggest" to bump it.

Also, could there be a case when A and B depend on C, but only A needs to be rebuilt when C is updated with some breaking changes?

[ilovezfs]

Yup, that happens all the time.

[stek29]

Ok, then what about providing both option to fully rebuild formula and to only rebuild metadata?

Again, A and B depend on C.
C is updated.

• If C update is non-breaking, binary_revision is not changed
• If C update is potentially breaking (breaks A but not B), binary_revision is changed, and:
  • A is fully rebuilt and it's metadata file is updated in result. Additionally, it's binary_revision is bumped, and the same process is done for A.
  • B only has it's metadata file updated to mark it tested and confirmed to be working with new binary_revision of C.

That means that after update all dependents are tested and either marked "just working" with new version, are rebuilt (but not marked changed so their dependents don't need to be rebuilt), or rebuilt and rebuild is marked as breaking, so their dependents need to be checked too.

[ilovezfs]

B only has it's metadata file updated to mark it tested and confirmed to be working with new binary_revision of C.

That means either

a) the INSTALL_RECEIPT, which is stored inside the prebuilt bottle, not apart from it, so cannot be updated without rebuilding the bottle
b) the formula file, which means you're maintaining "it works OK" state per dependency per version in the formula
c) some other files that don't exist yet

None of these sound like they can be handled automatically.

[stek29]

@ilovezfs c -- the new property/tag which is added to the bottle.

Also, they reference binary_version and not version/revision -- if there's some minor change which doesn't break anything, then binary_revision isn't bumped. If it breaks at least one formula in core, it may also likely break third-party taps, so binary_revision is bumped and all dependent formulas are reviewed (with assistance of automatic tool?..), and either rebuilt & binary_revision bumped, just rebuilt without bumping binary_revision (it's dependents aren't broken), or only metadata file is rebuilt to mark it compliant with current binary_revision without rebuilding.

It's certainly is not easier or prettier for maintainers, and it introduces an overhead of metadata file & rebuilding that metadata file from time to time, but I think that it would make bottle system more robust and make third-party bottles easier to use for end users and to make for third party tap maintainers.

If you don't think it's worth it -- I guess some other solution is needed :)

[MikeMcQuaid]

@MikeMcQuaid to avoid downloading whole bottle to only find out that it links to incompatible versions of formulas.

@stek29 I think it's not worth optimising for this case just to avoid downloading a single bottle, for what it's worth.

[scpeters]

Thanks @ilovezfs that makes sense. I was interpreting "vendoring" as requiring the use of the resource blocks, but forked formulae is a way of doing it.

Falling back to a build from source if the bottle breaks would be less effort / less performant, but I think that's where I'm currently leaning.

[sjackman]

Thanks @ilovezfs that makes sense. I was interpreting "vendoring" as requiring the use of the resource blocks, but forked formulae is a way of doing it.

That's also how I was interpreting your use of the word vendoring. This proposal is better than that.

[sjackman]

I'm okay rebuilding a bottle in a tap that depends on qt each time qt is updated in core. That's precisely what's done in Homebrew/core, and it works there. I want to avoid the short-term breakage in the interim before that bottle is rebuilt. Other less actively maintained taps may have other priorities, where vendoring is a better solution for them.

@sjackman I think you've literally been saying the opposite with the use-old-bottles proposition.

There's multiple possible technical solutions to this problem. One such solution is using old bottles. My current preferred solution is building from source until the tap has updated the bottle.

[ilovezfs]

any reason you haven't vendored all of your deps in the tap?

Ah. Yeah, that's why I said "in the tap" not "in the formula" but I guess that was not clear.

[ilovezfs]

My current preferred solution is building from source until the tap has updated the bottle.

That of course doesn't do anything for people who already have something installed when the breaking brew upgrade hits, unless there's some revdep-rebuild magic that kicks in for existing installations of the reverse dependencies, not just a source build fall back on new installs where a bottle is noticed to be broken on pour.

[sjackman]

Yes, good point.
brew upgrade could use brew linkage with the fast caching to detect that upgrading foo would break bar.
brew upgrade could rebuild bar from source.
brew upgrade could modify bar using install_name_tool to link to Cellar/foo/…/lib rather than opt/foo/lib.
brew upgrade could wait on upgrading foo until bar is rebottled, if that rebottling would happen in the near future, perhaps triggered automatically as Mike suggested. This option is not feasible if the rebottling of bar takes a long time or never happens.

[ilovezfs]

Some new brew magic that triggers a bar source build would be fine so that taps with absentee landords aren't totally broken. I still think best practice is bottling your own deps.

[scpeters]

Some new brew magic that triggers a bar source build would be fine so that taps with absentee landords aren't totally broken.

👍

I still think best practice is bottling your own deps.

If there was an easy way to rename a working core bottle, say boost, to osrf-boost without rebuilding, it would be much more attractive. Not sure how viable that is.

[wjwwood]

If there was an easy way to rename a working core bottle, say boost, to osrf-boost without rebuilding, it would be much more attractive. Not sure how viable that is.

That'd be nice, but it doesn't solve the proverbial framework integration issue, which is like "gazebo depends on osrf-boost which is at x.y and opencv depends on just boost which is at x.y+n and so gazebo and opencv can potentially not be linked together now". So then you end up basically needing to fork all of homebrew if you want to let your users combine your software arbitrarily with any other software in homebrew. Note, that's not a real example, just something I made up as an illustration, but something like this is realistic (imo) and has happened many times before to me.

[ilovezfs]

So then you end up basically needing to fork all of homebrew if you want to let your users combine your software arbitrarily with any other software in homebrew.

Yeah that would expressly need to not be a goal of the tap.

[wjwwood]

Yeah that would expressly need to not be a goal of the tap.

For this discussion, I agree. I didn't mean to get this issue off topic.

It's probably a somewhat unique requirement given that ROS integrates with some many other libraries at the user level while also using them internally, but it's also a big reason why we don't maintain a tap for ROS. That unfortunately means our users are more likely to avoid using macOS and Homebrew, and therefore also unlikely to contribute to Homebrew, though some do anyways.

[ilovezfs]

If there was an easy way to rename a working core bottle, say boost, to osrf-boost without rebuilding, it would be much more attractive. Not sure how viable that is.

It should be possible if you shadow the name instead of renaming. So

  depends_on "osrf/simulation/protobuf"
  depends_on "osrf/simulation/protobuf-c" => :build
  depends_on "osrf/simulation/zeromq"
  depends_on "osrf/simulation/cppzmq"

[ilovezfs]

(Note I wouldn't recommend trying that unless you're planning on expecting people to use tap-pin)

[sjackman]

Wouldn't that break formulae in Homebrew/core that depend on homebrew/core/protobuf, since they would see an older out-of-date version of protobuf?

[ilovezfs]

@sjackman yup.

[ilovezfs]

Is it possible to build downstream bottles for a 3rd party tap before the homebrew-core PR is merged?

@scpeters You should be able to pull the core PR into your own PR before we actually merge it to master.

[scpeters]

@scpeters You should be able to pull the core PR into your own PR before we actually merge it to master.

So brew pull --bottle should do it. Right. When I asked the question I had only ever used brew pull without --bottle (we modify bottle sha256's in our tap pull requests before merging). But it's clear in the maintainer docs; it sounds obvious now. Thanks for answering.

I'll try adding a core PR to our jenkins job parameters to see if this works.

[ilovezfs]

You'd need to brew pull without the --bottle since there won't be a published bottle yet, so you'd need to build the relevant dep yourself as well as part of the job.

[scpeters]

You'd need to brew pull without the --bottle since there won't be a published bottle yet, so you'd need to build the relevant dep yourself as well as part of the job.

Right when the pull request is made, there wouldn't be any published bottles, but they are uploaded once the jenkins CI jobs go green, right? If I was lucky with the timing to start the build after bottles had been uploaded, I think it might work? This is like an unofficial version of the "grace period" I discussed above. I'm not actually sure if that's how it works though.

[ilovezfs]

You could

brew pull --bottle --no-publish

and then manually download the bottles from Jenkins if they're already built and put them in your cache or upload them somewhere and set the root_url in the bottle block. They won't be on Bintray yet.

[scpeters]

They won't be on Bintray yet.

This was a little confusing to me, but I think I understand that they would be uploaded to bintray at this point but not published. So bintray credentials would be needed to download them.

You could brew pull --bottle --no-publish and then manually download the bottles from Jenkins if they're already built and put them in your cache or upload them somewhere and set the root_url in the bottle block.

Downloading the bottles from jenkins would take a little more work, crawling through the list of recent jenkins builds or getting the CI status from the GitHub API, but there's a limited number of recent builds, so they would potentially be deleted.

Is there room for a role in between contributor and maintainer (of homebrew-core) for 3rd-party tap maintainers who could have read-only access to unpublished bottles on bintray? I would consider this to be a privilege and that it wouldn't include an expectation of support from maintainers.

[sjackman]

I use CircleCI to build bottles for both Linux and macOS. You can download the bottle artifacts from CircleCI using their REST API. See https://github.com/Linuxbrew/homebrew-developer/blob/master/cmd/brew-pull-circle.rb#L50

You can then download the bottles form CircleCI and upload them to Bintray. I automated that step using a web hook and AWS Lambda. See https://github.com/Linuxbrew/linuxbrew-lambda

[scpeters]

I use Jenkins with machines in our office to build our bottles. In talking about unpublished homebrew-core bottles, I was referring to the unofficial "grace period" idea and trying to build bottles for a downstream tap before the homebrew-core pull request has been merged and bottles published to bintray.

[sjackman]

Ah, right. Thanks for the reminder, Steven.

[fredemmott]

I'm hoping to get HHVM's stuff in to a shape soon where I can send a PR so it's not third-party; for others, an alternative to full vendoring is to use static linking for dependencies. You can do this for everything, but what we've found best is to statically link dependencies that are particularly problematic (for example, ICU4C bumps the major version really often) and keep the stable stuff dynamically linked. Things still break occasionally but it's manageable.

[MikeMcQuaid]

I'm open to (and have been) accepting more specific PRs for this but closing this out as a bit too much of a meta-ask with no real completion requirements.