Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Travis finally released xenial (16.04). #6885

Merged
merged 18 commits into from
Dec 12, 2018
Merged

Conversation

OniOni
Copy link
Contributor

@OniOni OniOni commented Dec 7, 2018

Travis released Xenial (16.04) (see https://changelog.travis-ci.com/ubuntu-xenial-16-04-build-environment-is-here!-79690) This PR moves us to this new image.

The two major changes are:

  • Switch to Python 3.5 in CI.
    This coincides with prior discussions in Slack that we want to target Python 3.5 as the minimum Python 3 version. This is to give us access to native type hints and async/await syntax, along with general improvements such as fixing the shutil bug identified in Unblacklist JVM check_style Python 3 integration test thanks to using Python 3.5+ #6882.

  • Switch to openjdk over oraclejdk. This is mostly because travis only bakes version 10 and 11 of the oracle jdk and has removed all the tooling to get other versions of that jdk. While image comes with version 8 of the openjdk.

@OniOni OniOni changed the title Travis finally released xenial (16.04). [WIP] Travis finally released xenial (16.04). Dec 10, 2018
@OniOni OniOni changed the title [WIP] Travis finally released xenial (16.04). Travis finally released xenial (16.04). Dec 11, 2018
Copy link
Contributor

@Eric-Arellano Eric-Arellano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you Mathieu for pushing this through!

John and Stu, if you look at logs from most recent logs you’ll see how Python now resolves, including Pyenv installing 3.6 and 3.7. (I’m afk, sorry I can’t grab those lines directly)

Copy link
Member

@stuhood stuhood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks good. Would figure out whether the JAVA_HOME entry is actually necessary, and would wait for @jsirois to sanity check whether build-support/bin/travis-ci.sh is still resulting in relevant python(s).

.travis.yml Outdated
- sudo rm -rf /usr/lib/jvm/java-6-openjdk-amd64
- jdk_switcher use oraclejdk8
- PATH="/usr/lib/jvm/java-8-openjdk-amd64/jre/bin":$PATH
- JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64k
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a stray k on the end here, which makes me wonder whether this JAVA_HOME entry is actually being used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a very good point. I had removed it, believing it was not needed, but it was causing failures. I'll give it a spin without the JAVA_HOME, for sanity's sake.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, absence of JAVA_HOME causes a lot of failures.

@jsirois
Copy link
Contributor

jsirois commented Dec 12, 2018

The log output does show relevant pythons, and travis-ci.sh concocts this output by actually invoking the interpreters in question:

function list_pythons {
which -a python{,2,3} | sort -u | while read python_bin; do
${python_bin} <<EOF
import os
import sys
print('{} -> {}'.format(os.path.realpath(sys.executable),
'.'.join(map(str, sys.version_info[:3]))))
EOF
done
}

Picking a random shard I read:

Visible pythons are now:
/home/travis/virtualenv/python2.7.15/bin/python -> 2.7.15
/opt/python/2.7.15/bin/python2.7 -> 2.7.15
/opt/python/3.6.7/bin/python3.6 -> 3.6.7
/opt/python/3.7.1/bin/python3.7 -> 3.7.1
/usr/bin/python2.7 -> 2.7.12
/usr/bin/python3.5 -> 3.5.2

LGTM

We may no longer need the travis-ci.sh wrapper, but that can be for another change.

@Eric-Arellano Eric-Arellano merged commit 7da3ab5 into pantsbuild:master Dec 12, 2018
Eric-Arellano added a commit that referenced this pull request Dec 12, 2018
… Python 3.5+ (#6882)

## Problem
shutil.copytree was failing in CI for the `backend/jvm/tasks:checkstyle_integration` target due to a bug with Python 3.4.

```
shutil.Error: [('/home/travis/build/pantsbuild/pants/build-support/bin/native/src', '/tmp/tmpjdehtsq4/build-support/bin/native/src', "[Errno 21] Is a directory: '/home/travis/build/pantsbuild/pants/build-support/bin/native/src'"), ('/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/include/python2.7', '/tmp/tmpjdehtsq4/build-support/pants_dev_deps.venv/include/python2.7', "[Errno 21] Is a directory: '/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/include/python2.7'"), ('/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/config', '/tmp/tmpjdehtsq4/build-support/pants_dev_deps.venv/lib/python2.7/config', "[Errno 21] Is a directory: '/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/config'"), ('/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/encodings', '/tmp/tmpjdehtsq4/build-support/pants_dev_deps.venv/lib/python2.7/encodings', "[Errno 21] Is a directory: '/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/encodings'"), ('/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/lib-dynload', '/tmp/tmpjdehtsq4/build-support/pants_dev_deps.venv/lib/python2.7/lib-dynload', "[Errno 21] Is a directory: '/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/lib-dynload'")]
```

This error is confusing, because those are intentionally directories. Also, it doesn't reproduce on Python 2.7 and other Python 3 versions.

The bug is from the [std lib's handling of symbolic directories](https://bugs.python.org/issue21697). `shutil.copytree` treated all symbolic folders as files, which leads to the error `is a directory`. This was fixed in Python 3.5+.

## Solution
We upgraded Travis to Python 3.5 with #6885. This allows us to raise the floor of minimum Python 3 to 3.5, fixing this bug.
Eric-Arellano added a commit to Eric-Arellano/pants that referenced this pull request Dec 15, 2018
Eric-Arellano added a commit that referenced this pull request Dec 15, 2018
…'s config (#6939)

### Problem
In `test_conflict_via_config()`, we try to constraint Python to `< 2.7`. However, this fails to do anything when running on Python 3, because it's overriden by `ci.sh`'s environment variable `export PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS='["CPython>=3.5,<4"]'`.

This is expected behavior, that the environment variable overrides the command line argument. So, the issue is not with the source code, but this test should also not be failing.

### Solution
Set the integration test to be hermetic.

### Other related interpreter selection changes
* Allow Python 3.7 now that it's been fixed with #6363 
* In CI, require for Python 3 mode Python 3.5+, now that we use xenial (#6885)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants