Merge Linuxbrew/brew into Homebrew/brew #612

sjackman · 2018-02-20T16:47:10Z

We aim to merge Linuxbrew/brew into Homebrew/brew by the end of 2018. Linux users will use Homebrew/brew. Linuxbrew/brew will remain as a fork to be used for development.

Submit relocatable kegs (keg_relocate.rb) to Homebrew/brew
- Submit patchelf to Homebrew/core Relocate bottles on Linux using patchelf Homebrew/brew#4382
Submit Superenv (extend/ENV/super.rb) to Homebrew/brew Add superenv for Linux Homebrew/brew#4185
Ensure Homebrew/brew on Linux can brew readall Homebrew/homebrew-core and/or Linuxbrew/homebrew-core (probably both)
- Figure out a solution for MacOS.version
Submit remaining patches to Homebrew/brew or revert in Linuxbrew/brew (to be determined)
See Merge Linuxbrew/brew into Homebrew/brew #612 (comment)
Merge Linuxbrew/homebrew-test-bot into Homebrew/homebrew-test-bot
Document what Linux platforms Homebrew/brew officially supports
Document Linux behaviour in documentation (manpage)

This list is not comprehensive and will be expanded as items arise.

Once these tasks are complete, we'll migrate from Linuxbrew/brew to Homebrew/brew in this order:

Move Linuxbrew CI to using Homebrew/brew
Move Linuxbrew master branch users to using Homebrew/brew's master branch
Move Linuxbrew stable/tag users to using Homebrew/brew's master branch

The text was updated successfully, but these errors were encountered:

MikeMcQuaid · 2018-02-20T17:04:23Z

Suggestion for ordering this migration:

Move Linuxbrew CI to using Homebrew/brew
Move Linuxbrew master branch users to using Homebrew/brew's master branch
Move Linuxbrew stable/tag users to using Homebrew/brew's master branch

Additional tasks:

Document what Linux platforms Homebrew/brew officially supports (likely a single distro and a lower level than Linuxbrew/brew). Instead or as well: consider what the bug reporting/fixing flow is going to look like
Document Linux behaviour in documentation e.g. manpage
Figure out a solution for MacOS.version
Ensure Homebrew/brew on Linux can brew readall Homebrew/homebrew-core and/or Linuxbrew/homebrew-core (probably both)

sjackman · 2018-02-20T18:20:59Z

These all look good. I've added them to the tracking issue above.

Document what Linux platforms Homebrew/brew officially supports (likely a single distro and a lower level than Linuxbrew/brew)

Linuxbrew builds its bottles on the most recent Ubuntu LTS platform, which is currently Ubuntu 16.04 LTS (Xenial Xerus) and in April will become Ubuntu 18.04 LTS (Bionic Beaver). Other distributions that I'd like to support are Debian, Fedora, and CentOS. The popularity of distribution and version being run on Google Analytics in the last 30 days, in order of popularity, are Ubuntu 16.04, Ubuntu 17.10, Fedora Core 27, Debian 8, Ubuntu 14.04, Ubuntu 12.04, CentOS 7, and Debian 9.

Rank	Distribution	Total Events	Unique Events
1.	Ubuntu 16.04.3 LTS	10,034(19.35%)	9,702(19.77%)
2.	Ubuntu 16.04.3 LTS, custom-prefix	7,264(14.01%)	6,959(14.18%)
3.	Ubuntu 17.10	5,833(11.25%)	5,595(11.40%)
4.	4.13.9-300.fc27.x86_64	1,397(2.69%)	1,305(2.66%)
5.	Debian GNU/Linux 8.10 (jessie), custom-prefix	1,030(1.99%)	887(1.81%)
6.	Ubuntu 14.04.5 LTS	843(1.63%)	812(1.65%)
7.	4.4.0-111-generic, CI	823(1.59%)	823(1.68%)
8.	Ubuntu 16.04.2 LTS	767(1.48%)	738(1.50%)
9.	"CentOS Linux release 7.4.1708 (Core) "	745(1.44%)	728(1.48%)
10.	Debian GNU/Linux 9.3 (stretch)	690(1.33%)	672(1.37%)

sjackman · 2018-02-20T18:31:34Z

Figure out a solution for MacOS.version

Ensure Homebrew/brew on Linux can brew readall Homebrew/homebrew-core and/or Linuxbrew/homebrew-core (probably both)

Current Homebrew/brew on Linux cannot tap Homebrew/core due to…

undefined method `full_version' for OS::Mac:Module
undefined method `version' for OS::Mac::Xcode:Module
undefined method `version' for OS::Mac:Module

@MikeMcQuaid How do you feel about defining these three methods as Version::NULL on Linux?

MikeMcQuaid · 2018-02-20T18:39:05Z

@MikeMcQuaid How do you feel about defining these three methods as Version::NULL on Linux?

@sjackman That seems reasonable 👍 .

Linuxbrew builds its bottles on the most recent Ubuntu LTS platform, which is currently Ubuntu 16.04 LTS (Xenial Xerus) and in April will become Ubuntu 18.04 LTS (Bionic Beaver).

Cool. I think we should stick with the latest Ubuntu LTS and ensure that our Linux brew tests job is running there, if possible.

sjackman · 2018-02-20T18:50:16Z

Cool. I think we should stick with the latest Ubuntu LTS and ensure that our Linux brew tests job is running there, if possible.

Use Docker on TravisCI? I haven't used it yet, but seems supported (https://docs.travis-ci.com/user/docker/). Docker support on CircleCI 2.0 is first class.

sjackman · 2018-02-20T19:09:14Z

@MikeMcQuaid How do you feel about defining these three methods as Version::NULL on Linux?

@sjackman That seems reasonable 👍 .

I've submitted PR Homebrew/brew#3820. There's also MacOS.prefer_64_bit?. That's the last one. Shall I rename this method to OS.prefer_64_bit? and submit PRs to Homebrew/brew and Homebrew/homebrew-core? I'll keep the old name as a deprecated alias.

MikeMcQuaid · 2018-02-21T08:53:14Z

I've submitted PR Homebrew/brew#3820. There's also MacOS.prefer_64_bit?. That's the last one. Shall I rename this method to OS.prefer_64_bit? and submit PRs to Homebrew/brew and Homebrew/homebrew-core? I'll keep the old name as a deprecated alias.

prefer_64_bit? has always felt a bit weird as a name to me. What about just OS.64_bit? or OS.is_64_bit? CC @ilovezfs in case you have other good naming thoughts.

Use Docker on TravisCI? I haven't used it yet, but seems supported (https://docs.travis-ci.com/user/docker/). Docker support on CircleCI 2.0 is first class.

What's the distro that the base Travis CI image is running?

ilovezfs · 2018-02-21T14:41:06Z

I guess OS.64_bit? seems fine. Apple is dropping 32-bit support altogether so we may also want a OS.64_bit_only? if that helps inform your naming thoughts at all.

sjackman · 2018-02-21T16:28:10Z

I liked OS.64_bit?

What's the distro that the base Travis CI image is running?

Ubuntu 12.04 LTS (Trusty Tahr), which is six years old now.

ilovezfs · 2018-02-21T16:34:22Z

I'm confused. https://docs.travis-ci.com/user/reference/overview/

Sudo-enabled #

A sudo enabled, full virtual machine per build, that runs Linux, either Ubuntu Precise 12.04 or Ubuntu Trusty 14.04.
Container-based #

A fast boot time environment in which sudo commands are not available. Running Linux Ubuntu Trusty 14.04

sjackman · 2018-02-21T17:11:23Z

You're right. I had forgotten that TravisCI offers both Precise and Trusty. Ubuntu 14.04 LTS (Trusty Tahr) is four years old. TravisCI does not offer the current Ubuntu 16.04 LTS (Xenial Xerus). Ubuntu 18.04 LTS (Bionic Beaver) will be released in April.

sjackman · 2018-02-23T19:31:47Z

Here's the list of essential (required) Debian packages. It's pretty safe to assume that these packages will be available on most Linux systems.
See https://www.reddit.com/r/debian/comments/6lnakq/what_are_the_essential_packages/

base-files              libattr1                libuuid1
base-passwd             libblkid1               login
bash                    libc-bin                lsb-base
bsdutils                libc6                   mawk
coreutils               libcomerr2              mount
dash                    libfdisk1               multiarch-support
debconf                 libgcc1                 ncurses-base
debianutils             liblzma5                ncurses-bin
diffutils               libmount1               passwd
dpkg                    libpam-modules          perl-base
e2fslibs                libpam-modules-bin      sed
e2fsprogs               libpam-runtime          sensible-utils
findutils               libpam0g                sysvinit-utils
gcc-6-base              libpcre3                tar
grep                    libselinux1             tzdata
gzip                    libsepol1               util-linux
hostname                libsmartcols1           zlib1g
init-system-helpers     libss2
libacl1                 libtinfo5

jamescostian · 2018-08-06T19:26:50Z

Is there anything that an outsider could do to help this merger? The todo list at the top seems like it may be out of date (e.g. "Figure out a solution for MacOS.version" seems to be resolved by a comment here and "Document what Linux platforms Homebrew/brew officially supports" seems to be resolved by another comment) so if anyone has a more up-to-date list of issues, I'd be happy to try and tackle some.

iMichka · 2018-08-06T19:32:11Z

I think we need to rework that list a little bit :)

sjackman · 2018-08-08T01:49:59Z

@jamescostian At this point, Homebrew/brew should be usable with Linuxbrew/core, so end-user testing would be helpful. Install Linuxbrew, and then change ~/.linuxbrew/Homebrew to use the Homebrew/brew remote rather than the default Linuxbrew/brew. If you run into any problems or have questions / need help, open a new issue at Linuxbrew/brew.

MikeMcQuaid · 2018-08-08T08:03:55Z

@sjackman That's pretty exciting. What are the main things that remain unmerged?

sjackman · 2018-08-14T01:26:54Z

MikeMcQuaid · 2018-08-14T08:01:45Z

HOMEBREW_BOTTLE_DEFAULT_DOMAIN is https://linuxbrew.bintray.com on Linux
Default core tap is https://github.com/Linuxbrew/homebrew-core on Linux
Find the host's GNU compiler e.g. /usr/bin/gcc-5 or /usr/bin/gcc44
Various fixes to stdenv for Linux (though superenv is the default)
Linuxbrew's bottles are built for glibc 2.23, so install the brewed glibc and gcc if the host's glibc is older than 2.23
depends_on :java installs the formula jdk on Linux
depends_on :x11 installs the formula linuxbrew/xorg/xorg on Linux
Ignore keg_only when :provided_by_macos, :provided_by_osx, :shadowed_by_macos on Linux
brew linkage checks for undesirable linking to host libraries on Linux
Default installation prefix is /home/linuxbrew/.linuxbrew on Linux
Treat cellar :any_skip_relocation as cellar :any on Linux, since relocation is always needed
Environment variable HOMEBREW_ARCH specifies the value of cc -march=…, useful when installing formulae on one machine to NFS, but then running those executables on a machine with an older CPU, e.g. HOMEBREW_ARCH=core2. Particularly useful for HPC systems with a newer head job submission node, but older compute nodes.

These all seem reasonable and mergeable 👍

Print a dot each minute during brew uses. This may no longer be necessary.

I think we've handled this with brew test-bot now for Travis CI. It's also a bunch faster with the newer Ruby we're on now.

maxim-belkin · 2018-08-14T09:21:39Z

Print a dot each minute during brew uses. This may no longer be necessary.

I think we've handled this with brew test-bot now for Travis CI. It's also a bunch faster with the newer Ruby we're on now.

For Linuxbrew, when something depends on linuxbrew/xorg/xorg (or some other package from linuxbrew/xorg) we end up with a longer-than-on-macOS list of dependencies. So, it's not directly related to the speed at which Ruby does things.

iMichka · 2018-09-06T05:37:47Z

depends_on :x11 installs the formula linuxbrew/xorg/xorg on Linux

I do not do that anymore in core. This got broken at one point, so I am using depends on "linuxbrew/xorg/xorg". I also noticed that it is often overkill to depend on the full xorg formula and went for a more fine-grained selection of dependencies.

sjackman · 2018-09-06T06:48:56Z

Thanks for the information, Michka! That should reduce a big part of the diff in superenv.

MikeMcQuaid · 2018-09-06T08:14:57Z

Various fixes to stdenv for Linux (though superenv is the default)

Sounds like from Homebrew/brew#4831 (comment) this can be removed for now.

depends_on :java installs the formula jdk on Linux
depends_on :x11 installs the formula linuxbrew/xorg/xorg on Linux

It would be interesting to explore whether openjdk and even xorg can be usefully built and used on macOS. It would be neat to make two big, open source cask dependencies not require manual installation.

sjackman · 2018-09-06T18:59:24Z

Various fixes to stdenv for Linux (though superenv is the default)

Sounds like from Homebrew/brew#4831 (comment) this can be removed for now.

Some fixes will be necessary, but I've struck out this item for now. It's not a priority, since Superenv is the default, and works great.

depends_on :java installs the formula jdk on Linux
depends_on :x11 installs the formula linuxbrew/xorg/xorg on Linux

It would be interesting to explore whether openjdk and even xorg can be usefully built and used on macOS. It would be neat to make two big, open source cask dependencies not require manual installation.

That would be pretty cool. I'm not myself the person to take up the torch of creating those formulae. It'd be awesome if someone did pick it up.

MikeMcQuaid · 2018-09-07T08:07:51Z

Some fixes will be necessary, but I've struck out this item for now. It's not a priority, since Superenv is the default, and works great.

👍

That would be pretty cool. I'm not myself the person to take up the torch of creating those formulae. It'd be awesome if someone did pick it up.

Agreed and same here 😁 (for now at least).

alebcay · 2018-09-13T22:36:22Z

That would be pretty cool. I'm not myself the person to take up the torch of creating those formulae. It'd be awesome if someone did pick it up.

I'm looking into making an openjdk formula, but it seems that the build process requires an existing Java compiler on the PATH in order to work (a bootstrap JDK).

sjackman · 2018-09-13T23:21:14Z

Homebrew has allowed using precompiled binaries in this case to bootstrap compiling the packaged compiler from source.

alebcay · 2018-09-15T21:06:56Z

What is the status of https://github.com/Linuxbrew/homebrew-core/pull/8977? I have an OpenJDK formula in the works (one that completely builds from the source tarballs pulled from the OpenJDK mercurial), but I see there's already a PR in progress.

iMichka · 2018-09-15T21:29:36Z

I still need to review it. If you have some comments for that new formula, they are welcome of course :)

alebcay · 2018-09-15T23:44:12Z

No, nothing in particular. I was just curious on which, if either, approach (from ojdkbuild or building ourselves) was preferable. It would save me some time so I could start working towards making the xorg formulas build from scratch. 😄

maxim-belkin · 2018-09-16T20:06:02Z

I could start working towards making the xorg formulas build from scratch

They build from scratch just fine. Some need to be update to account for a recent switch to real superenv, but otherwise...

sjackman · 2018-09-17T17:10:40Z

PR https://github.com/Linuxbrew/homebrew-core/pull/8977 installs prebuilt binaries of OpenJDK for Linux, which is okay in Linuxbrew/core, but not okay for Homebrew/core. The maintainers of Homebrew/core have expressed that they would like an openjdk formula that builds it from source. Would you be interested in submitting your formula upstream to Homebrew/core? Do you use or have access to a macOS machine?

sjackman · 2018-09-17T17:11:22Z

If so, the formula submitted upstream to Homebrew/core would need to support only macOS. Linuxbrew/core is a fork of Homebrew/core, and the formula here in Linuxbrew/core would support both macOS and Linux.

alebcay · 2018-09-18T14:32:53Z

Would you be interested in submitting your formula upstream to Homebrew/core? Do you use or have access to a macOS machine?

Yes, in that case, I'll take it over to the Homebrew/core side then.

sjackman · 2018-09-18T17:56:46Z

Excellent! That's great, Caleb.

MikeMcQuaid · 2018-10-31T09:59:41Z

Is there much more to do here for this to be merged?

sjackman · 2018-10-31T16:42:52Z

The substantial portions of the merger are complete. Homebrew/brew is now quite usable on Linux! Linuxbrew/test-bot ought to be merged into Homebrew/test-bot. Linuxbrew developers and beta testers need to give Homebrew/brew on Linux a spin. Then it'll be rolled out to CI on Brewsci/bio and Linuxbrew/core. Then it'll be rolled out to users with homebrew.devcmdrun=1, and finally to all users.

MikeMcQuaid · 2018-10-31T17:05:31Z

The substantial portions of the merger are complete.

What portions are left?

Linuxbrew developers and beta testers need to give Homebrew/brew on Linux a spin. Then it'll be rolled out to CI on Brewsci/bio and Linuxbrew/core. Then it'll be rolled out to users with homebrew.devcmdrun=1, and finally to all user

Cool, sounds good. Would be good to have this on CI within a few weeks if we want to hit the goal of rolling out to all users by the end of the year.

sjackman · 2018-10-31T23:05:19Z

eiennohito · 2018-11-27T11:43:52Z

Release without support for HOMEBREW_ARCH have broken my installation :(

sjackman · 2018-11-27T19:14:17Z

Hi, Arseny. That HOMEBREW_ARCH feature has been removed from Linuxbrew as part of the merging of Linuxbrew/brew into Homebrew/brew, because Homebrew did not have this feature, and due to bit rot, it hadn't actually been working in a while (specifically, it was implemented for --env=std, but never for --env=super, and Linuxbrew switched to --env=super over a year ago`). I recommend installing packages on a machine that has the oldest CPU architecture that you are using.

sjackman · 2018-12-18T20:14:30Z

@MikeMcQuaid Here's the list of supported distributions of Linux. Your thoughts? Shall I open a PR to create docs/Linux.md?

Edit: Moved this comment to Homebrew/brew#4758 (comment)

sjackman · 2019-01-07T19:28:30Z

The last remaining item is Merge Linuxbrew/homebrew-test-bot into Homebrew/homebrew-test-bot.
This issue has moved to Homebrew/brew#4758

sjackman · 2019-01-07T19:29:28Z

Homebrew/brew 1.9.0 will include beta support for Linux!
🥂 🎆

sjackman · 2019-01-11T16:10:57Z

Homebrew/brew 1.9.0 includes beta support for Linux and Windows 10!
https://brew.sh/2019/01/09/homebrew-1.9.0/

sjackman self-assigned this Feb 20, 2018

sjackman mentioned this issue Feb 20, 2018

bzip2 and zip dependencies_if_needed Homebrew/brew#3813

Merged

6 tasks

sjackman added the in progress label Feb 20, 2018

sjackman mentioned this issue Jul 5, 2018

use python flock firstly #737

Closed

6 tasks

sjackman mentioned this issue Aug 14, 2018

Use a more reliable command to check passwordless sudo Linuxbrew/install#36

Closed

jamescostian mentioned this issue Aug 15, 2018

Simplify setting PATH, MANPATH, and INFOPATH #812

Closed

sjackman mentioned this issue Aug 27, 2018

Official Linux support Homebrew/brew#4758

Closed

alebcay mentioned this issue Aug 29, 2018

Set HOMEBREW_BOTTLE_DEFAULT_DOMAIN based on operating system Homebrew/brew#4772

Merged

6 tasks

alebcay mentioned this issue Sep 5, 2018

std: use HOMEBREW_ARCH env var on Linux Homebrew/brew#4831

Merged

6 tasks

sjackman mentioned this issue Oct 23, 2018

relocation_formulae: Poured bottles depend on gcc [Linux] Homebrew/brew#5018

Merged

6 tasks

sjackman mentioned this issue Nov 2, 2018

Doctor check umask #876

Closed

6 tasks

sjackman closed this as completed Jan 7, 2019

Linuxbrew locked as resolved and limited conversation to collaborators Jan 7, 2019

sjackman removed the in progress label Jan 11, 2019

Merge Linuxbrew/brew into Homebrew/brew #612

Merge Linuxbrew/brew into Homebrew/brew #612

Comments

sjackman commented Feb 20, 2018 • edited Loading

MikeMcQuaid commented Feb 20, 2018

sjackman commented Feb 20, 2018 • edited Loading

sjackman commented Feb 20, 2018

MikeMcQuaid commented Feb 20, 2018

sjackman commented Feb 20, 2018 • edited Loading

sjackman commented Feb 20, 2018 • edited Loading

MikeMcQuaid commented Feb 21, 2018

ilovezfs commented Feb 21, 2018

sjackman commented Feb 21, 2018

ilovezfs commented Feb 21, 2018

sjackman commented Feb 21, 2018 • edited Loading

sjackman commented Feb 23, 2018

jamescostian commented Aug 6, 2018

iMichka commented Aug 6, 2018

sjackman commented Aug 8, 2018

MikeMcQuaid commented Aug 8, 2018

sjackman commented Aug 14, 2018 • edited Loading

MikeMcQuaid commented Aug 14, 2018

maxim-belkin commented Aug 14, 2018

iMichka commented Sep 6, 2018

sjackman commented Sep 6, 2018

MikeMcQuaid commented Sep 6, 2018

sjackman commented Sep 6, 2018

MikeMcQuaid commented Sep 7, 2018

alebcay commented Sep 13, 2018 • edited Loading

sjackman commented Sep 13, 2018

alebcay commented Sep 15, 2018 • edited Loading

iMichka commented Sep 15, 2018

alebcay commented Sep 15, 2018

maxim-belkin commented Sep 16, 2018

sjackman commented Sep 17, 2018

sjackman commented Sep 17, 2018

alebcay commented Sep 18, 2018

sjackman commented Sep 18, 2018

MikeMcQuaid commented Oct 31, 2018

sjackman commented Oct 31, 2018

MikeMcQuaid commented Oct 31, 2018

sjackman commented Oct 31, 2018 • edited Loading

Fixes for Linuxbrew/core CI

eiennohito commented Nov 27, 2018

sjackman commented Nov 27, 2018

sjackman commented Dec 18, 2018 • edited Loading

sjackman commented Jan 7, 2019 • edited Loading

sjackman commented Jan 7, 2019

sjackman commented Jan 11, 2019

sjackman commented Feb 20, 2018 •

edited

Loading

sjackman commented Feb 20, 2018 •

edited

Loading

sjackman commented Feb 20, 2018 •

edited

Loading

sjackman commented Feb 20, 2018 •

edited

Loading

sjackman commented Feb 21, 2018 •

edited

Loading

sjackman commented Aug 14, 2018 •

edited

Loading

alebcay commented Sep 13, 2018 •

edited

Loading

alebcay commented Sep 15, 2018 •

edited

Loading

sjackman commented Oct 31, 2018 •

edited

Loading

sjackman commented Dec 18, 2018 •

edited

Loading

sjackman commented Jan 7, 2019 •

edited

Loading