-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
ARROW-6920: [Packaging] Build python 3.8 wheels
adds python3.8 wheels as far as I can tell python3.8 isn't available for Conda yet (conda-forge/python-feedstock#274), so that's will have to be added later Closes #5685 from sjhewitt/arrow-6920 and squashes the following commits: feac494 <Sutou Kouhei> Add a TODO comment 908a396 <Sutou Kouhei> Check Docker version for debug 43833a3 <Sutou Kouhei> Add Python 3.8 support to package metadata f3f9bfa <Krisztián Szűcs> Install tensorflow for 32 bit unicode width py27 cd9ae83 <Sutou Kouhei> Install TensorFlow for Python 2.7 manually 5293f2b <Sutou Kouhei> Pin base image and remove needless virtualenvs 344eecd <Sutou Kouhei> Use apache/arrow-dev again 99cad41 <Sutou Kouhei> Use the uploaded image f0c8e69 <Sutou Kouhei> Link to libpython only on Windows 260fa79 <Sutou Kouhei> Add a missing empty line 52761b4 <Sutou Kouhei> libpython.so may not exist 5e5977e <Sutou Kouhei> Don't link to other Python libraries on non Windows e047d54 <Sutou Kouhei> Don't use local function 10dac67 <Terence D. Honles> move plasma subprocesses to top level functions da300ac <Sutou Kouhei> Don't use TTY c4526d8 <Sutou Kouhei> Use more recent numpy 7ace0cb <Sutou Kouhei> Ensure using the latest image 655f88c <Sutou Kouhei> Use Azure Pipelines instead of Travis CI 2ef6213 <Sutou Kouhei> Use "call conda.bat activate/deactivate" 1c8427e <Sutou Kouhei> Use conda without call 99251ea <Sutou Kouhei> Use "conda activate/deactivate" f403b6f <Sutou Kouhei> Use the latest multibuild c818840 <Sutou Kouhei> Remove m suffix e9f1408 <Sutou Kouhei> Set up Python 3.8 2ab6732 <Simon Hewitt> ARROW-6920: Build python 3.8 wheels Lead-authored-by: Sutou Kouhei <kou@clear-code.com> Co-authored-by: Terence D. Honles <terence@honles.com> Co-authored-by: Krisztián Szűcs <szucs.krisztian@gmail.com> Co-authored-by: Simon Hewitt <si@sjhewitt.co.uk> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
- Loading branch information
1 parent
7102d7e
commit 6c6754a
Showing
18 changed files
with
234 additions
and
314 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,92 @@ | ||
# Licensed to the Apache Software Foundation (ASF) under one | ||
# or more contributor license agreements. See the NOTICE file | ||
# distributed with this work for additional information | ||
# regarding copyright ownership. The ASF licenses this file | ||
# to you under the Apache License, Version 2.0 (the | ||
# "License"); you may not use this file except in compliance | ||
# with the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
jobs: | ||
- job: linux | ||
pool: | ||
vmImage: ubuntu-latest | ||
timeoutInMinutes: 360 | ||
steps: | ||
- task: CondaEnvironment@1 | ||
inputs: | ||
packageSpecs: 'click github3.py jinja2 jira pygit2 ruamel.yaml setuptools_scm toolz' | ||
installOptions: '-c conda-forge' | ||
updateConda: false | ||
|
||
- script: | | ||
set -ex | ||
git clone --no-checkout {{ arrow.remote }} arrow | ||
git -C arrow fetch -t {{ arrow.remote }} {{ arrow.branch }} | ||
git -C arrow checkout FETCH_HEAD | ||
git -C arrow submodule update --init --recursive | ||
displayName: Clone arrow | ||
- script: | | ||
set -ex | ||
docker -v | ||
docker-compose -v | ||
cd arrow | ||
BUILD_IMAGE=centos-python-{{ wheel_tag }} | ||
docker-compose pull $BUILD_IMAGE | ||
# TODO(kou): Uncomment this when we resolve "ADD never use cache" problem. | ||
# docker-compose build $BUILD_IMAGE | ||
docker-compose run \ | ||
-e SETUPTOOLS_SCM_PRETEND_VERSION={{ arrow.no_rc_version }} \ | ||
-e PYTHON_VERSION="{{ python_version }}" \ | ||
-e UNICODE_WIDTH="{{ unicode_width }}" \ | ||
$BUILD_IMAGE | ||
displayName: Build wheel | ||
# auditwheel does always exit with 0 so it is mostly for debugging | ||
# purposes | ||
- script: | | ||
set -ex | ||
cd arrow | ||
docker run \ | ||
-v $(pwd):/arrow quay.io/pypa/{{ wheel_tag }}_x86_64 \ | ||
/bin/bash -c \ | ||
"auditwheel show /arrow/python/{{ wheel_tag }}/dist/*.whl" | ||
displayName: Audit wheel | ||
- script: | | ||
set -ex | ||
cd arrow | ||
test_args="" | ||
{%- if test_remove_system_libs %} | ||
test_args="${test_args} --remove-system-libs" | ||
{%- endif %} | ||
{%- for image in test_docker_images %} | ||
docker run \ | ||
--shm-size 2G \ | ||
-v $(pwd):/arrow \ | ||
-e WHEEL_TAG="{{ wheel_tag }}" \ | ||
{{ image }} \ | ||
/arrow/dev/tasks/python-wheels/manylinux-test.sh ${test_args} | ||
{%- endfor %} | ||
displayName: Test wheel | ||
- script: | | ||
set -ex | ||
python arrow/dev/tasks/crossbow.py \ | ||
--queue-path . \ | ||
--queue-remote {{ queue.remote_url }} \ | ||
upload-artifacts \ | ||
--pattern "arrow/python/{{ wheel_tag }}/dist/*" \ | ||
--sha {{ task.branch }} \ | ||
--tag {{ task.tag }} | ||
env: | ||
CROSSBOW_GITHUB_TOKEN: $(CROSSBOW_GITHUB_TOKEN) | ||
displayName: Upload packages as a GitHub release |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.