Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add compiler package script #57

Conversation

cosmicexplorer
Copy link
Contributor

The new script ./build-compiler-5.0.1.sh will package an archive named compiler.tar.gz for both MacOS and Linux when executed in a Linux environment. This archive currently contains the clang compiler and the C standard library, which is sufficient to compile the python_dist() test project -- this can be tested by running ./pants run testprojects/src/python/python_distribution/fasthello_with_install_requires:main_with_no_conflict.

Notes

  1. This script pulls down a binary release of clang+llvm from the llvm downloads page for MacOS, but we have to build the Linux package from a source release (downloaded from the same page). This means that the script should be executed in a Linux environment to produce both MacOS and Linux packages.
  2. LLVM really likes .tar.xz files for some reason, so xz is needed to unpack their archives -- you will need to run yum install -y xz if using the CentOS Dockerfile available in the pants codebase.
  3. This script executes the build-cmake-3.9.5.sh script already available in this repo to produce a cmake binary that can be used to build LLVM from source. This is necessary because LLVM requires cmake >= 3.4, which is otherwise not available in the CentOS Dockerfile from the pants codebase. Because build-cmake-3.9.5.sh only builds for Linux x86_64, this script only builds a compiler for 64-bit Linux hosts. If 32-bit support becomes necessary, I can probably modify both scripts to do so, but I haven't investigated whether this is supported for either cmake or LLVM yet.

Copy link
Member

@stuhood stuhood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks good. I've just updated the docker instructions a bit to synchronize them with pantsbuild/pants: #58. Please use those instructions to build the binaries in centos6 (rather than wheezy).

@@ -0,0 +1,141 @@
#!/bin/bash

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

compiler is too generic a name... clang, probably?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang is definitely better than compiler, but thinking about it now I might actually suggest to call it clang-llvm, or even clang+llvm, because the LLVM version (5.0.1) is what's used in the release download url from the LLVM downloads page, not the clang version. Also, the MacOS binary release download unpacks into an archive named clang+llvm-5.0.1-<platform>.tar.xz, so calling it clang+llvm would match that convention. It could be clang-llvm too, but I prefer clang+llvm because that makes it clear which part of clang+llvm-5.0.1.tar.gz is the package name and which part is the version.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you planning to package something in here other than clang itself?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the C and C++ standard library headers. The C standard library headers are already included -- I hadn't tested the C++ standard library headers because we don't have a python_dist() target with C++ code in testprojects/ yet in the pants repo. These would be provided with the compiler so that they are compatible with the compiler, and so we don't have to rely on headers existing in standard locations (which is a requirement for an internal consumer of python_dist() with native code).

I'm not planning to package anything that's not considered part of clang, however. It might make sense to just call it clang, by that logic. Since we're not packaging any other LLVM tools, it might be confusing/surprising to call it clang+llvm.

After concentrating on this for a bit, my vote goes to just calling it clang.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to build-clang-5.0.1.sh, producing clang.tar.gz in supportdir build-support/bin/clang in ad67ff4.

CORRESPONDING_CLANG_BIN_VERSION='5.0'
LLVM_RELEASE_BUILD_DIRNAME='llvm-tmp'
LLVM_PANTS_ARCHIVE_NAME='compiler.tar.gz'
COMPILER_SUPPORTDIR='build-support/bin/compiler'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to clang in ad67ff4.

Copy link
Contributor

@benjyw benjyw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mod stu's suggestions.

Copy link
Contributor

@illicitonion illicitonion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM modulo Stu's comments

@stuhood stuhood merged commit fee4cf0 into pantsbuild:gh-pages Mar 1, 2018
cosmicexplorer added a commit to cosmicexplorer/binaries that referenced this pull request Mar 21, 2018
cosmicexplorer added a commit to cosmicexplorer/binaries that referenced this pull request Mar 30, 2018
cosmicexplorer added a commit to cosmicexplorer/binaries that referenced this pull request Mar 30, 2018
cosmicexplorer added a commit to cosmicexplorer/binaries that referenced this pull request Nov 16, 2020
Add buildozer 0.6.0-80c7f0d45d7e40fa1f7362852697d4a03df557b3 (pantsbuild#56)

Synchronize docker instructions with pantsbuild/pants. (pantsbuild#58)

Add clang (pantsbuild#57)

Switch to a workflow without committed binaries (pantsbuild#59)

* Update the README to indicate that no new binaries should be committed here.
* Formalize the script location to improve maintainability.

Add Go 1.9.4 and 1.10 (pantsbuild#61)

Here we add `build.sh` scripts for Go 1.9.4 and 1.10, and a helper
script for generating the `build.sh` scripts. We also clarify in the
docs what platforms are supported (i386 and macOS pre 10.8 are not
supported going forward).

update some of the commands described in the README (pantsbuild#63)

add gcc 7.3.0 and clang 6.0.0 cross platform, binutils 2.30 for linux, edit clang 5.0.1 to conform to pantsbuild#59 (pantsbuild#62)

```
$ ./sync-s3.sh
upload: bin/clang/linux/x86_64/6.0.0/build-cmake.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-cmake.sh
upload: bin/clang/linux/x86_64/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build.sh
upload: bin/binutils/linux/x86_64/2.30/build.sh to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/build.sh
upload: bin/binutils/linux/x86_64/2.30/build-binutils.sh to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/build-binutils.sh
upload: bin/binutils/linux/x86_64/2.30/binutils.tar.gz to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/binutils.tar.gz
upload: bin/clang/mac/10.10/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/build-clang.sh
upload: bin/clang/mac/10.10/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/build.sh
upload: bin/clang/linux/x86_64/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.11/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/build-clang.sh
upload: bin/clang/mac/10.11/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/build.sh
upload: bin/clang/linux/x86_64/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-clang.sh
upload: bin/clang/linux/x86_64/6.0.0/build-clang-for-linux-with-cmake.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-clang-for-linux-with-cmake.sh
upload: bin/clang/mac/10.12/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/build-clang.sh
upload: bin/clang/mac/10.12/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/build.sh
upload: bin/clang/mac/10.10/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.12/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.13/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/build-clang.sh
upload: bin/clang/mac/10.13/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/build.sh
upload: bin/clang/mac/10.11/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.8/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/build.sh
upload: bin/clang/mac/10.8/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/build-clang.sh
upload: bin/clang/mac/10.13/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.9/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/build-clang.sh
upload: bin/clang/mac/10.9/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/build.sh
upload: bin/clang/mac/10.8/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/clang.tar.gz
upload: bin/gcc/linux/x86_64/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/build.sh
upload: bin/gcc/linux/x86_64/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/build-gcc.sh
upload: bin/clang/mac/10.9/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/clang.tar.gz
upload: bin/gcc/mac/10.10/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.10/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/build.sh
upload: bin/gcc/mac/10.11/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/build-gcc.sh
upload: bin/gcc/linux/x86_64/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.11/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/build.sh
upload: bin/gcc/mac/10.12/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.12/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/build.sh
upload: bin/gcc/mac/10.10/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.11/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.13/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.13/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/build.sh
upload: bin/gcc/mac/10.12/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.8/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.8/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/build.sh
upload: bin/gcc/mac/10.13/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.9/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.9/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/build.sh
upload: bin/gcc/mac/10.8/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.9/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/gcc.tar.gz
```

add xz package (pantsbuild#66)

cloc and isort are now fetched from `bin`, but without the platform-specific path component. (pantsbuild#68)

add --disable-rpath when configuring xz (pantsbuild#71)

This removes the need to set `DYLD_LIBRARY_PATH` or `LD_LIBRARY_PATH` in pantsbuild/pants#5936. The `xz` binary will use the version of `xz` and `liblzma` in the installation directory when invoked.

add xz wrapper script to set the library path (pantsbuild#72)

See pantsbuild#71, where we added `--disable-rpath` to `xz` to stop it from using a hardcoded library path at runtime. However, that still fails in Travis in pantsbuild/pants#5936 with the output:
```
INFO] Fetched llvm.tar.xz binary from: https://releases.llvm.org/6.0.0/clang+llvm-6.0.0-x86_64-linux-gnu-ubuntu-16.04.tar.xz .
INFO] Extracting /home/travis/.cache/pants/bin/llvm/linux/x86_64/6.0.0/llvm.tar.xz to /home/travis/.cache/pants/bin/llvm/linux/x86_64/6.0.0/llvm .
xz: /lib/x86_64-linux-gnu/liblzma.so.5: version `XZ_5.2' not found (required by xz)
```
While we had removed the query for the hardcoded path, we still need to actually set our own path. We had been doing that on the pants side -- instead, this PR adds a (very short) python script at the path of the real `xz` binary (a suggestion from @illicitonion in that PR) which sets `LD_LIBRARY_PATH` or `DYLD_LIBRARY_PATH` appropriately, not clobbering any existing values. This works with the current version of pantsbuild/pants#5936 for me on osx and linux.

statically link xz (pantsbuild#74)

See pantsbuild#71, pantsbuild#72, pantsbuild#73. Our `xz` is picking up travis's `xz` installation and using its (incompatible) `liblzma` shared library. On the suggestion of @stuhood in pantsbuild/pants#5936, we statically link `xz` here to avoid ever having this problem ever again. Proof:
*linux*:
```
> ldd ./xz-5.2.4-linux/xz-install/bin/xz
        linux-vdso.so.1 (0x00007ffc865cd000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f1fa385f000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f1fa34a3000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f1fa3cb4000)
```
*osx*:
```
> otool -L ./xz-5.2.4-linux/xz-install/bin/xz
./xz-5.2.4-linux/xz-install/bin/xz:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
```

add llvm 6.0.0 package. (pantsbuild#75)

See pantsbuild/pants#5991. This adds the binary I should have provided when we added the ability to download `BinaryTool`s from external urls, in the exact same way we do it within Pants.

Upgrade isort to 4.3.4. (pantsbuild#76)

This adds support for more python3 stdlib symbols.

Also introduce a build script.

Fixup isort path structure.

Node.js 10.7.0 and Yarnpkg 1.7.0 (pantsbuild#80)

Four small scripts, each named "build.sh" have been added.  These will build a specific version of Node.js or Yarnpkg in the current directory.

Symlinks from older macOS to 10.13 are included.

Add build scripts and symlinks for Node 8.11.3 and Yarnpkg 1.6.0 (pantsbuild#81)

Adding Node v10.13.0 build scripts for Mac 10.8-10.13 and Linux X64 (pantsbuild#86)

Add go 1.10.5 (pantsbuild#87)

Add build script for Thrift 0.11.0 (pantsbuild#83)

Add build scripts and symlinks for Yarnpkg 1.12.3 (pantsbuild#89)

Add updated go versions to pants binaries (pantsbuild#90)

* Add updated go versions to pants binaries

* Expanded the mac_versions into a map to encode the OSX min version requirements of the latest builds of go.

Add build scripts and symlinks for Yarnpkg 1.16.0 (pantsbuild#91)

* Add build scripts and symlinks for Yarnpkg 1.16.0

Add go 1.12.7 (pantsbuild#92)

* Add go 1.12.7

Upgrade to cloc 1.82 (pantsbuild#93)

Add cloc 1.80 (pantsbuild#94)

The current version of cloc (version 1.82) has a bug where the --ignored
flag is not working properly ( AlDanial/cloc#401 ).
The pants test for the cloc goal tests this behavior, so as long as this
bug is not fixed upstream, pants will fail this test. I've confirmed
that version 1.80 of cloc does not have this bug, so we should be able
to use it until upstream cloc fixes the bug in a new verson.

Add go 1.13.8 for linux and mac 10.11+
cosmicexplorer added a commit to cosmicexplorer/binaries that referenced this pull request Nov 16, 2020
Add buildozer 0.6.0-80c7f0d45d7e40fa1f7362852697d4a03df557b3 (pantsbuild#56)

Synchronize docker instructions with pantsbuild/pants. (pantsbuild#58)

Add clang (pantsbuild#57)

Switch to a workflow without committed binaries (pantsbuild#59)

* Update the README to indicate that no new binaries should be committed here.
* Formalize the script location to improve maintainability.

Add Go 1.9.4 and 1.10 (pantsbuild#61)

Here we add `build.sh` scripts for Go 1.9.4 and 1.10, and a helper
script for generating the `build.sh` scripts. We also clarify in the
docs what platforms are supported (i386 and macOS pre 10.8 are not
supported going forward).

update some of the commands described in the README (pantsbuild#63)

add gcc 7.3.0 and clang 6.0.0 cross platform, binutils 2.30 for linux, edit clang 5.0.1 to conform to pantsbuild#59 (pantsbuild#62)

```
$ ./sync-s3.sh
upload: bin/clang/linux/x86_64/6.0.0/build-cmake.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-cmake.sh
upload: bin/clang/linux/x86_64/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build.sh
upload: bin/binutils/linux/x86_64/2.30/build.sh to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/build.sh
upload: bin/binutils/linux/x86_64/2.30/build-binutils.sh to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/build-binutils.sh
upload: bin/binutils/linux/x86_64/2.30/binutils.tar.gz to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/binutils.tar.gz
upload: bin/clang/mac/10.10/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/build-clang.sh
upload: bin/clang/mac/10.10/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/build.sh
upload: bin/clang/linux/x86_64/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.11/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/build-clang.sh
upload: bin/clang/mac/10.11/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/build.sh
upload: bin/clang/linux/x86_64/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-clang.sh
upload: bin/clang/linux/x86_64/6.0.0/build-clang-for-linux-with-cmake.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-clang-for-linux-with-cmake.sh
upload: bin/clang/mac/10.12/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/build-clang.sh
upload: bin/clang/mac/10.12/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/build.sh
upload: bin/clang/mac/10.10/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.12/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.13/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/build-clang.sh
upload: bin/clang/mac/10.13/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/build.sh
upload: bin/clang/mac/10.11/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.8/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/build.sh
upload: bin/clang/mac/10.8/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/build-clang.sh
upload: bin/clang/mac/10.13/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.9/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/build-clang.sh
upload: bin/clang/mac/10.9/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/build.sh
upload: bin/clang/mac/10.8/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/clang.tar.gz
upload: bin/gcc/linux/x86_64/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/build.sh
upload: bin/gcc/linux/x86_64/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/build-gcc.sh
upload: bin/clang/mac/10.9/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/clang.tar.gz
upload: bin/gcc/mac/10.10/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.10/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/build.sh
upload: bin/gcc/mac/10.11/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/build-gcc.sh
upload: bin/gcc/linux/x86_64/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.11/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/build.sh
upload: bin/gcc/mac/10.12/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.12/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/build.sh
upload: bin/gcc/mac/10.10/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.11/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.13/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.13/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/build.sh
upload: bin/gcc/mac/10.12/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.8/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.8/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/build.sh
upload: bin/gcc/mac/10.13/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.9/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.9/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/build.sh
upload: bin/gcc/mac/10.8/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.9/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/gcc.tar.gz
```

add xz package (pantsbuild#66)

cloc and isort are now fetched from `bin`, but without the platform-specific path component. (pantsbuild#68)

add --disable-rpath when configuring xz (pantsbuild#71)

This removes the need to set `DYLD_LIBRARY_PATH` or `LD_LIBRARY_PATH` in pantsbuild/pants#5936. The `xz` binary will use the version of `xz` and `liblzma` in the installation directory when invoked.

add xz wrapper script to set the library path (pantsbuild#72)

See pantsbuild#71, where we added `--disable-rpath` to `xz` to stop it from using a hardcoded library path at runtime. However, that still fails in Travis in pantsbuild/pants#5936 with the output:
```
INFO] Fetched llvm.tar.xz binary from: https://releases.llvm.org/6.0.0/clang+llvm-6.0.0-x86_64-linux-gnu-ubuntu-16.04.tar.xz .
INFO] Extracting /home/travis/.cache/pants/bin/llvm/linux/x86_64/6.0.0/llvm.tar.xz to /home/travis/.cache/pants/bin/llvm/linux/x86_64/6.0.0/llvm .
xz: /lib/x86_64-linux-gnu/liblzma.so.5: version `XZ_5.2' not found (required by xz)
```
While we had removed the query for the hardcoded path, we still need to actually set our own path. We had been doing that on the pants side -- instead, this PR adds a (very short) python script at the path of the real `xz` binary (a suggestion from @illicitonion in that PR) which sets `LD_LIBRARY_PATH` or `DYLD_LIBRARY_PATH` appropriately, not clobbering any existing values. This works with the current version of pantsbuild/pants#5936 for me on osx and linux.

statically link xz (pantsbuild#74)

See pantsbuild#71, pantsbuild#72, pantsbuild#73. Our `xz` is picking up travis's `xz` installation and using its (incompatible) `liblzma` shared library. On the suggestion of @stuhood in pantsbuild/pants#5936, we statically link `xz` here to avoid ever having this problem ever again. Proof:
*linux*:
```
> ldd ./xz-5.2.4-linux/xz-install/bin/xz
        linux-vdso.so.1 (0x00007ffc865cd000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f1fa385f000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f1fa34a3000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f1fa3cb4000)
```
*osx*:
```
> otool -L ./xz-5.2.4-linux/xz-install/bin/xz
./xz-5.2.4-linux/xz-install/bin/xz:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
```

add llvm 6.0.0 package. (pantsbuild#75)

See pantsbuild/pants#5991. This adds the binary I should have provided when we added the ability to download `BinaryTool`s from external urls, in the exact same way we do it within Pants.

Upgrade isort to 4.3.4. (pantsbuild#76)

This adds support for more python3 stdlib symbols.

Also introduce a build script.

Fixup isort path structure.

Node.js 10.7.0 and Yarnpkg 1.7.0 (pantsbuild#80)

Four small scripts, each named "build.sh" have been added.  These will build a specific version of Node.js or Yarnpkg in the current directory.

Symlinks from older macOS to 10.13 are included.

Add build scripts and symlinks for Node 8.11.3 and Yarnpkg 1.6.0 (pantsbuild#81)

Adding Node v10.13.0 build scripts for Mac 10.8-10.13 and Linux X64 (pantsbuild#86)

Add go 1.10.5 (pantsbuild#87)

Add build script for Thrift 0.11.0 (pantsbuild#83)

Add build scripts and symlinks for Yarnpkg 1.12.3 (pantsbuild#89)

Add updated go versions to pants binaries (pantsbuild#90)

* Add updated go versions to pants binaries

* Expanded the mac_versions into a map to encode the OSX min version requirements of the latest builds of go.

Add build scripts and symlinks for Yarnpkg 1.16.0 (pantsbuild#91)

* Add build scripts and symlinks for Yarnpkg 1.16.0

Add go 1.12.7 (pantsbuild#92)

* Add go 1.12.7

Upgrade to cloc 1.82 (pantsbuild#93)

Add cloc 1.80 (pantsbuild#94)

The current version of cloc (version 1.82) has a bug where the --ignored
flag is not working properly ( AlDanial/cloc#401 ).
The pants test for the cloc goal tests this behavior, so as long as this
bug is not fixed upstream, pants will fail this test. I've confirmed
that version 1.80 of cloc does not have this bug, so we should be able
to use it until upstream cloc fixes the bug in a new verson.

Add go 1.13.8 for linux and mac 10.11+
cosmicexplorer added a commit to cosmicexplorer/binaries that referenced this pull request Nov 16, 2020
Add buildozer 0.6.0-80c7f0d45d7e40fa1f7362852697d4a03df557b3 (pantsbuild#56)

Synchronize docker instructions with pantsbuild/pants. (pantsbuild#58)

Add clang (pantsbuild#57)

Switch to a workflow without committed binaries (pantsbuild#59)

* Update the README to indicate that no new binaries should be committed here.
* Formalize the script location to improve maintainability.

Add Go 1.9.4 and 1.10 (pantsbuild#61)

Here we add `build.sh` scripts for Go 1.9.4 and 1.10, and a helper
script for generating the `build.sh` scripts. We also clarify in the
docs what platforms are supported (i386 and macOS pre 10.8 are not
supported going forward).

update some of the commands described in the README (pantsbuild#63)

add gcc 7.3.0 and clang 6.0.0 cross platform, binutils 2.30 for linux, edit clang 5.0.1 to conform to pantsbuild#59 (pantsbuild#62)

```
$ ./sync-s3.sh
upload: bin/clang/linux/x86_64/6.0.0/build-cmake.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-cmake.sh
upload: bin/clang/linux/x86_64/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build.sh
upload: bin/binutils/linux/x86_64/2.30/build.sh to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/build.sh
upload: bin/binutils/linux/x86_64/2.30/build-binutils.sh to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/build-binutils.sh
upload: bin/binutils/linux/x86_64/2.30/binutils.tar.gz to s3://binaries.pantsbuild.org/bin/binutils/linux/x86_64/2.30/binutils.tar.gz
upload: bin/clang/mac/10.10/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/build-clang.sh
upload: bin/clang/mac/10.10/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/build.sh
upload: bin/clang/linux/x86_64/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.11/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/build-clang.sh
upload: bin/clang/mac/10.11/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/build.sh
upload: bin/clang/linux/x86_64/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-clang.sh
upload: bin/clang/linux/x86_64/6.0.0/build-clang-for-linux-with-cmake.sh to s3://binaries.pantsbuild.org/bin/clang/linux/x86_64/6.0.0/build-clang-for-linux-with-cmake.sh
upload: bin/clang/mac/10.12/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/build-clang.sh
upload: bin/clang/mac/10.12/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/build.sh
upload: bin/clang/mac/10.10/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.10/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.12/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.12/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.13/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/build-clang.sh
upload: bin/clang/mac/10.13/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/build.sh
upload: bin/clang/mac/10.11/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.11/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.8/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/build.sh
upload: bin/clang/mac/10.8/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/build-clang.sh
upload: bin/clang/mac/10.13/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.13/6.0.0/clang.tar.gz
upload: bin/clang/mac/10.9/6.0.0/build-clang.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/build-clang.sh
upload: bin/clang/mac/10.9/6.0.0/build.sh to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/build.sh
upload: bin/clang/mac/10.8/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.8/6.0.0/clang.tar.gz
upload: bin/gcc/linux/x86_64/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/build.sh
upload: bin/gcc/linux/x86_64/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/build-gcc.sh
upload: bin/clang/mac/10.9/6.0.0/clang.tar.gz to s3://binaries.pantsbuild.org/bin/clang/mac/10.9/6.0.0/clang.tar.gz
upload: bin/gcc/mac/10.10/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.10/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/build.sh
upload: bin/gcc/mac/10.11/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/build-gcc.sh
upload: bin/gcc/linux/x86_64/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/linux/x86_64/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.11/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/build.sh
upload: bin/gcc/mac/10.12/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.12/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/build.sh
upload: bin/gcc/mac/10.10/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.10/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.11/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.11/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.13/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.13/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/build.sh
upload: bin/gcc/mac/10.12/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.12/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.8/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.8/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/build.sh
upload: bin/gcc/mac/10.13/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.13/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.9/7.3.0/build-gcc.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/build-gcc.sh
upload: bin/gcc/mac/10.9/7.3.0/build.sh to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/build.sh
upload: bin/gcc/mac/10.8/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.8/7.3.0/gcc.tar.gz
upload: bin/gcc/mac/10.9/7.3.0/gcc.tar.gz to s3://binaries.pantsbuild.org/bin/gcc/mac/10.9/7.3.0/gcc.tar.gz
```

add xz package (pantsbuild#66)

cloc and isort are now fetched from `bin`, but without the platform-specific path component. (pantsbuild#68)

add --disable-rpath when configuring xz (pantsbuild#71)

This removes the need to set `DYLD_LIBRARY_PATH` or `LD_LIBRARY_PATH` in pantsbuild/pants#5936. The `xz` binary will use the version of `xz` and `liblzma` in the installation directory when invoked.

add xz wrapper script to set the library path (pantsbuild#72)

See pantsbuild#71, where we added `--disable-rpath` to `xz` to stop it from using a hardcoded library path at runtime. However, that still fails in Travis in pantsbuild/pants#5936 with the output:
```
INFO] Fetched llvm.tar.xz binary from: https://releases.llvm.org/6.0.0/clang+llvm-6.0.0-x86_64-linux-gnu-ubuntu-16.04.tar.xz .
INFO] Extracting /home/travis/.cache/pants/bin/llvm/linux/x86_64/6.0.0/llvm.tar.xz to /home/travis/.cache/pants/bin/llvm/linux/x86_64/6.0.0/llvm .
xz: /lib/x86_64-linux-gnu/liblzma.so.5: version `XZ_5.2' not found (required by xz)
```
While we had removed the query for the hardcoded path, we still need to actually set our own path. We had been doing that on the pants side -- instead, this PR adds a (very short) python script at the path of the real `xz` binary (a suggestion from @illicitonion in that PR) which sets `LD_LIBRARY_PATH` or `DYLD_LIBRARY_PATH` appropriately, not clobbering any existing values. This works with the current version of pantsbuild/pants#5936 for me on osx and linux.

statically link xz (pantsbuild#74)

See pantsbuild#71, pantsbuild#72, pantsbuild#73. Our `xz` is picking up travis's `xz` installation and using its (incompatible) `liblzma` shared library. On the suggestion of @stuhood in pantsbuild/pants#5936, we statically link `xz` here to avoid ever having this problem ever again. Proof:
*linux*:
```
> ldd ./xz-5.2.4-linux/xz-install/bin/xz
        linux-vdso.so.1 (0x00007ffc865cd000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f1fa385f000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f1fa34a3000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f1fa3cb4000)
```
*osx*:
```
> otool -L ./xz-5.2.4-linux/xz-install/bin/xz
./xz-5.2.4-linux/xz-install/bin/xz:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
```

add llvm 6.0.0 package. (pantsbuild#75)

See pantsbuild/pants#5991. This adds the binary I should have provided when we added the ability to download `BinaryTool`s from external urls, in the exact same way we do it within Pants.

Upgrade isort to 4.3.4. (pantsbuild#76)

This adds support for more python3 stdlib symbols.

Also introduce a build script.

Fixup isort path structure.

Node.js 10.7.0 and Yarnpkg 1.7.0 (pantsbuild#80)

Four small scripts, each named "build.sh" have been added.  These will build a specific version of Node.js or Yarnpkg in the current directory.

Symlinks from older macOS to 10.13 are included.

Add build scripts and symlinks for Node 8.11.3 and Yarnpkg 1.6.0 (pantsbuild#81)

Adding Node v10.13.0 build scripts for Mac 10.8-10.13 and Linux X64 (pantsbuild#86)

Add go 1.10.5 (pantsbuild#87)

Add build script for Thrift 0.11.0 (pantsbuild#83)

Add build scripts and symlinks for Yarnpkg 1.12.3 (pantsbuild#89)

Add updated go versions to pants binaries (pantsbuild#90)

* Add updated go versions to pants binaries

* Expanded the mac_versions into a map to encode the OSX min version requirements of the latest builds of go.

Add build scripts and symlinks for Yarnpkg 1.16.0 (pantsbuild#91)

* Add build scripts and symlinks for Yarnpkg 1.16.0

Add go 1.12.7 (pantsbuild#92)

* Add go 1.12.7

Upgrade to cloc 1.82 (pantsbuild#93)

Add cloc 1.80 (pantsbuild#94)

The current version of cloc (version 1.82) has a bug where the --ignored
flag is not working properly ( AlDanial/cloc#401 ).
The pants test for the cloc goal tests this behavior, so as long as this
bug is not fixed upstream, pants will fail this test. I've confirmed
that version 1.80 of cloc does not have this bug, so we should be able
to use it until upstream cloc fixes the bug in a new verson.

Add go 1.13.8 for linux and mac 10.11+
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants