Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated the Dockerfile to switch to python 3.8 #176

Merged
merged 3 commits into from
Oct 1, 2020
Merged

Conversation

mraarif
Copy link
Contributor

@mraarif mraarif commented Aug 31, 2020

Description

Updated the Dockerfile to install python 3.8 and pip and use pip for python 3.8 to install dependencies.

Ticket Link

Link to the associated ticket
BOM-2001

Post-review

Squash commits into discrete sets of changes

@mraarif
Copy link
Contributor Author

mraarif commented Aug 31, 2020

Test these changes locally by building an image.

@mraarif mraarif requested a review from christopappas August 31, 2020 13:27
@mraarif
Copy link
Contributor Author

mraarif commented Aug 31, 2020

@jmbowman can you please tag the owning team here.

Dockerfile Outdated
apt-add-repository -y ppa:deadsnakes/ppa && apt-get update && \
apt-get upgrade -qy && apt-get install language-pack-en locales git python3.5 python3-pip \
python3-pip libmysqlclient-dev libssl-dev python3-dev python3.8-dev python3.8-distutils -qy && \
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my question over in the similar xqueue PR: https://github.com/edx/xqueue/pull/778/files#r480147263 . Just want to make sure this is the best pip installation approach before we start using it everywhere.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jmbowman we had to make this change because we faced

AttributeError: module 'platform' has no attribute 'linux_distribution'

while building a docker image to verify python 3.8 installation changes in Dockerfile.
I tried using ensurepip and got

/usr/bin/python3.8: No module named ensurepip

while building an image

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, as I just noted over on the xqueue PR, it doesn't look like there's a much cleaner solution to this until we upgrade to Ubuntu 20.04.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you also tag the owing team for review

@jmbowman
Copy link
Contributor

jmbowman commented Sep 2, 2020

@bbaker6225 Do you want to review this before we merge it?

@bbaker6225
Copy link
Contributor

Sure I'll give this a quick test

@bbaker6225
Copy link
Contributor

I'm running into the same issue from the license-manager PR on running make requirements. I'll try running make upgrade to see if that works, but if not pinning might be the way to go

@bbaker6225 bbaker6225 removed the request for review from christopappas September 2, 2020 19:08
@bbaker6225
Copy link
Contributor

No luck unfortunately. Pinning to avoid the setuptools errors might be the way to go

@mraarif mraarif force-pushed the BOM-2001 branch 6 times, most recently from 245e893 to 951ea3d Compare September 20, 2020 12:47
@mraarif
Copy link
Contributor Author

mraarif commented Sep 24, 2020

I was able to build the image and run make dev.up, make app-shell, and make requirements (inside the container) with these changes.

@bbaker6225 .. let me know if there's something else needed for the review to proceed.

@bbaker6225
Copy link
Contributor

Hi @mraarif , in attempting to rebuild my local enterprise catalog I run into this error:

ERROR: Command errored out with exit status 1: /venv/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bk6r76gx/psutil/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bk6r76gx/psutil/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-7446rbqr/install-record.txt --single-version-externally-managed --compile --install-headers /venv/include/site/python3.8/psutil Check the logs for full command output.
ERROR: Service 'worker' failed to build : The command '/bin/sh -c pip install -r /edx/app/enterprise_catalog/enterprise_catalog/requirements/production.txt' returned a non-zero code: 1

I pulled this branch for enterprise-catalog, ran make dev.down to remove the containers, and then followed the steps in docs/getting_started.rst to get the local set up working. However, this fails on make dev.provision with the above error. I've included the full trace below in case it's helpful.

➜  enterprise-catalog git:(BOM-2001) make dev.down
docker-compose down
Network devstack_default is external, skipping
➜  enterprise-catalog git:(BOM-2001) make dev.provision
bash ./provision-catalog.sh
Building worker
Step 1/35 : FROM ubuntu:xenial as app
 ---> c871c45b1573
Step 2/35 : LABEL maintainer="devops@edx.org"
 ---> Using cache
 ---> e295a1076210
Step 3/35 : RUN apt-get update && apt-get install -y software-properties-common && apt-get install -y curl && apt-add-repository -y ppa:deadsnakes/ppa && apt-get update && apt-get upgrade -qy && apt-get install language-pack-en locales git libmysqlclient-dev libssl-dev python3.8-dev python3.8-venv -qy && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> f948d96cf505
Step 4/35 : ENV VIRTUAL_ENV=/venv
 ---> Using cache
 ---> 14d2b712b927
Step 5/35 : RUN python3.8 -m venv $VIRTUAL_ENV
 ---> Using cache
 ---> ed2857b4f795
Step 6/35 : ENV PATH="$VIRTUAL_ENV/bin:$PATH"
 ---> Using cache
 ---> aff224a17006
Step 7/35 : RUN pip install pip==20.2.3 setuptools==50.3.0
 ---> Using cache
 ---> d4cc0a9ba4ac
Step 8/35 : RUN locale-gen en_US.UTF-8
 ---> Using cache
 ---> 55652057a00a
Step 9/35 : ENV LANG en_US.UTF-8
 ---> Using cache
 ---> 4d63c8051820
Step 10/35 : ENV LANGUAGE en_US:en
 ---> Using cache
 ---> b62f14f952f3
Step 11/35 : ENV LC_ALL en_US.UTF-8
 ---> Using cache
 ---> f0a203aa8bf0
Step 12/35 : EXPOSE 8160
 ---> Using cache
 ---> 71b81f49e399
Step 13/35 : EXPOSE 8161
 ---> Using cache
 ---> e21f9e563f14
Step 14/35 : RUN useradd -m --shell /bin/false app
 ---> Using cache
 ---> a6c068cf43f1
Step 15/35 : WORKDIR /edx/app/enterprise_catalog/enterprise_catalog
 ---> Using cache
 ---> 5ace0d5574c2
Step 16/35 : COPY requirements/ /edx/app/enterprise_catalog/enterprise_catalog/requirements/
 ---> Using cache
 ---> 6c9a2a7bb080
Step 17/35 : RUN pip install -r /edx/app/enterprise_catalog/enterprise_catalog/requirements/production.txt
 ---> Running in 53ab78063af5
Collecting algoliasearch==2.4.0
  Downloading algoliasearch-2.4.0-py2.py3-none-any.whl (31 kB)
Collecting amqp==2.6.1
  Downloading amqp-2.6.1-py2.py3-none-any.whl (48 kB)
Collecting billiard==3.6.3.0
  Downloading billiard-3.6.3.0-py3-none-any.whl (89 kB)
Collecting celery==4.4.7
  Downloading celery-4.4.7-py2.py3-none-any.whl (427 kB)
Collecting certifi==2020.6.20
  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting cffi==1.14.2
  Downloading cffi-1.14.2-cp38-cp38-manylinux1_x86_64.whl (410 kB)
Collecting chardet==3.0.4
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting coreapi==2.3.3
  Downloading coreapi-2.3.3-py2.py3-none-any.whl (25 kB)
Collecting coreschema==0.0.4
  Downloading coreschema-0.0.4.tar.gz (10 kB)
Collecting cryptography==3.1
  Downloading cryptography-3.1-cp35-abi3-manylinux2010_x86_64.whl (2.6 MB)
Collecting defusedxml==0.6.0
  Downloading defusedxml-0.6.0-py2.py3-none-any.whl (23 kB)
Collecting django-cors-headers==3.5.0
  Downloading django_cors_headers-3.5.0-py3-none-any.whl (11 kB)
Collecting django-crum==0.7.7
  Downloading django_crum-0.7.7-py2.py3-none-any.whl (4.7 kB)
Collecting django-extensions==3.0.7
  Downloading django_extensions-3.0.7-py2.py3-none-any.whl (221 kB)
Collecting django-model-utils==4.0.0
  Downloading django_model_utils-4.0.0-py2.py3-none-any.whl (29 kB)
Collecting django-rest-swagger==2.2.0
  Downloading django_rest_swagger-2.2.0-py2.py3-none-any.whl (495 kB)
Collecting django-simple-history==2.11.0
  Downloading django_simple_history-2.11.0-py2.py3-none-any.whl (45 kB)
Collecting django-waffle==2.0.0
  Downloading django_waffle-2.0.0-py2.py3-none-any.whl (43 kB)
Collecting django==2.2.16
  Downloading Django-2.2.16-py3-none-any.whl (7.5 MB)
Collecting djangorestframework-xml==2.0.0
  Downloading djangorestframework_xml-2.0.0-py2.py3-none-any.whl (6.0 kB)
Collecting djangorestframework==3.11.1
  Downloading djangorestframework-3.11.1-py3-none-any.whl (911 kB)
Collecting drf-jwt==1.17.1
  Downloading drf_jwt-1.17.1-py2.py3-none-any.whl (18 kB)
Collecting edx-auth-backends==3.1.0
  Downloading edx_auth_backends-3.1.0-py2.py3-none-any.whl (29 kB)
Collecting edx-celeryutils==0.5.2
  Downloading edx_celeryutils-0.5.2-py2.py3-none-any.whl (25 kB)
Collecting edx-django-release-util==0.4.4
  Downloading edx_django_release_util-0.4.4-py2.py3-none-any.whl (40 kB)
Collecting edx-django-utils==3.8.0
  Downloading edx_django_utils-3.8.0-py2.py3-none-any.whl (53 kB)
Collecting edx-drf-extensions==6.1.2
  Downloading edx_drf_extensions-6.1.2-py2.py3-none-any.whl (57 kB)
Collecting edx-opaque-keys==2.1.1
  Downloading edx-opaque-keys-2.1.1.tar.gz (61 kB)
Collecting edx-rbac==1.3.2
  Downloading edx_rbac-1.3.2-py2.py3-none-any.whl (38 kB)
Collecting edx-rest-api-client==1.9.2
  Downloading edx_rest_api_client-1.9.2-py3-none-any.whl (10 kB)
Collecting future==0.18.2
  Downloading future-0.18.2.tar.gz (829 kB)
Collecting gevent==20.6.2
  Downloading gevent-20.6.2-cp38-cp38-manylinux2010_x86_64.whl (6.1 MB)
Collecting greenlet==0.4.16
  Downloading greenlet-0.4.16-cp38-cp38-manylinux1_x86_64.whl (48 kB)
Collecting gunicorn==20.0.4
  Downloading gunicorn-20.0.4-py2.py3-none-any.whl (77 kB)
Collecting idna==2.10
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting importlib-metadata==1.7.0
  Downloading importlib_metadata-1.7.0-py2.py3-none-any.whl (31 kB)
Collecting itypes==1.2.0
  Downloading itypes-1.2.0-py2.py3-none-any.whl (4.8 kB)
Collecting jinja2==2.11.2
  Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting jsonfield2==3.0.3
  Downloading jsonfield2-3.0.3-py3-none-any.whl (6.4 kB)
Collecting kombu==4.6.11
  Downloading kombu-4.6.11-py2.py3-none-any.whl (184 kB)
Collecting langcodes==2.1.0
  Downloading langcodes-2.1.0.tar.gz (5.0 MB)
Collecting marisa-trie==0.7.5
  Downloading marisa-trie-0.7.5.tar.gz (270 kB)
Collecting markupsafe==1.1.1
  Downloading MarkupSafe-1.1.1-cp38-cp38-manylinux1_x86_64.whl (32 kB)
Collecting mysqlclient==2.0.1
  Downloading mysqlclient-2.0.1.tar.gz (87 kB)
Collecting newrelic==5.18.0.148
  Downloading newrelic-5.18.0.148-cp38-cp38-manylinux2010_x86_64.whl (684 kB)
Collecting oauthlib==3.1.0
  Downloading oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
Collecting openapi-codec==1.3.2
  Downloading openapi-codec-1.3.2.tar.gz (6.3 kB)
Collecting pbr==5.5.0
  Downloading pbr-5.5.0-py2.py3-none-any.whl (106 kB)
Collecting psutil==5.7.2
  Downloading psutil-5.7.2.tar.gz (460 kB)
Collecting pycparser==2.20
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting pycryptodomex==3.9.8
  Downloading pycryptodomex-3.9.8-cp38-cp38-manylinux1_x86_64.whl (13.7 MB)
Collecting pyjwkest==1.4.2
  Downloading pyjwkest-1.4.2.tar.gz (41 kB)
Collecting pyjwt[crypto]==1.7.1
  Downloading PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)
Collecting pymongo==3.11.0
  Downloading pymongo-3.11.0-cp38-cp38-manylinux2014_x86_64.whl (530 kB)
Collecting python-dateutil==2.8.1
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting python-memcached==1.59
  Downloading python_memcached-1.59-py2.py3-none-any.whl (16 kB)
Collecting python3-openid==3.2.0
  Downloading python3_openid-3.2.0-py3-none-any.whl (133 kB)
Collecting pytz==2020.1
  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting pyyaml==5.3.1
  Downloading PyYAML-5.3.1.tar.gz (269 kB)
Collecting redis==3.5.3
  Downloading redis-3.5.3-py2.py3-none-any.whl (72 kB)
Collecting requests-oauthlib==1.3.0
  Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting requests==2.24.0
  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting rest-condition==1.0.3
  Downloading rest_condition-1.0.3.tar.gz (4.9 kB)
Collecting rules==2.2
  Downloading rules-2.2.tar.gz (53 kB)
Collecting semantic-version==2.8.5
  Downloading semantic_version-2.8.5-py2.py3-none-any.whl (15 kB)
Collecting simplejson==3.17.2
  Downloading simplejson-3.17.2-cp38-cp38-manylinux2010_x86_64.whl (137 kB)
Collecting six==1.15.0
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting slumber==0.7.1
  Downloading slumber-0.7.1.tar.gz (10 kB)
Collecting social-auth-app-django
  Cloning https://github.com/python-social-auth/social-app-django.git (to revision c00d23c2b45c3317bd35b15ad1b959338689cef8) to /tmp/pip-install-bk6r76gx/social-auth-app-django
Collecting social-auth-core==3.2.0
  Downloading social_auth_core-3.2.0-py3-none-any.whl (315 kB)
Collecting sqlparse==0.3.1
  Downloading sqlparse-0.3.1-py2.py3-none-any.whl (40 kB)
Collecting stevedore==1.32.0
  Downloading stevedore-1.32.0-py2.py3-none-any.whl (43 kB)
Collecting uritemplate==3.0.1
  Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting urllib3==1.25.10
  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting vine==1.3.0
  Downloading vine-1.3.0-py2.py3-none-any.whl (14 kB)
Collecting zipp==1.2.0
  Downloading zipp-1.2.0-py2.py3-none-any.whl (4.8 kB)
Collecting zope.event==4.4
  Downloading zope.event-4.4-py2.py3-none-any.whl (7.6 kB)
Collecting zope.interface==5.1.0
  Downloading zope.interface-5.1.0-cp38-cp38-manylinux2010_x86_64.whl (243 kB)
Requirement already satisfied: setuptools in /venv/lib/python3.8/site-packages (from gevent==20.6.2->-r /edx/app/enterprise_catalog/enterprise_catalog/requirements/production.txt (line 38)) (50.3.0)
Using legacy 'setup.py install' for coreschema, since package 'wheel' is not installed.
Using legacy 'setup.py install' for edx-opaque-keys, since package 'wheel' is not installed.
Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
Using legacy 'setup.py install' for langcodes, since package 'wheel' is not installed.
Using legacy 'setup.py install' for marisa-trie, since package 'wheel' is not installed.
Using legacy 'setup.py install' for mysqlclient, since package 'wheel' is not installed.
Using legacy 'setup.py install' for openapi-codec, since package 'wheel' is not installed.
Using legacy 'setup.py install' for psutil, since package 'wheel' is not installed.
Using legacy 'setup.py install' for pyjwkest, since package 'wheel' is not installed.
Using legacy 'setup.py install' for pyyaml, since package 'wheel' is not installed.
Using legacy 'setup.py install' for rest-condition, since package 'wheel' is not installed.
Using legacy 'setup.py install' for rules, since package 'wheel' is not installed.
Using legacy 'setup.py install' for slumber, since package 'wheel' is not installed.
Using legacy 'setup.py install' for social-auth-app-django, since package 'wheel' is not installed.
Installing collected packages: urllib3, idna, certifi, chardet, requests, algoliasearch, vine, amqp, billiard, kombu, pytz, celery, pycparser, cffi, itypes, markupsafe, jinja2, coreschema, uritemplate, coreapi, six, cryptography, defusedxml, sqlparse, django, django-cors-headers, django-crum, django-extensions, django-model-utils, djangorestframework, openapi-codec, simplejson, django-rest-swagger, django-simple-history, django-waffle, djangorestframework-xml, pyjwt, drf-jwt, oauthlib, python3-openid, requests-oauthlib, social-auth-core, social-auth-app-django, edx-auth-backends, jsonfield2, future, edx-celeryutils, pyyaml, edx-django-release-util, pbr, stevedore, psutil, newrelic, edx-django-utils, python-dateutil, semantic-version, pycryptodomex, pyjwkest, pymongo, edx-opaque-keys, rest-condition, edx-drf-extensions, edx-rbac, slumber, edx-rest-api-client, zope.interface, zope.event, greenlet, gevent, gunicorn, zipp, importlib-metadata, marisa-trie, langcodes, mysqlclient, python-memcached, redis, rules
    Running setup.py install for coreschema: started
    Running setup.py install for coreschema: finished with status 'done'
    Running setup.py install for openapi-codec: started
    Running setup.py install for openapi-codec: finished with status 'done'
    Running setup.py install for social-auth-app-django: started
    Running setup.py install for social-auth-app-django: finished with status 'done'
    Running setup.py install for future: started
    Running setup.py install for future: finished with status 'done'
    Running setup.py install for pyyaml: started
    Running setup.py install for pyyaml: finished with status 'done'
    Running setup.py install for psutil: started
    Running setup.py install for psutil: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /venv/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bk6r76gx/psutil/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bk6r76gx/psutil/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-7446rbqr/install-record.txt --single-version-externally-managed --compile --install-headers /venv/include/site/python3.8/psutil
         cwd: /tmp/pip-install-bk6r76gx/psutil/
    Complete output (44 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/psutil
    copying psutil/_psbsd.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/__init__.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_psosx.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_pslinux.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_pswindows.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_common.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_pssunos.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_psposix.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_compat.py -> build/lib.linux-x86_64-3.8/psutil
    copying psutil/_psaix.py -> build/lib.linux-x86_64-3.8/psutil
    creating build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/__init__.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_linux.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_osx.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_windows.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_testutils.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_process.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_unicode.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_bsd.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_memleaks.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_posix.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_misc.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_system.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/runner.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_aix.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/__main__.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_contracts.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_connections.py -> build/lib.linux-x86_64-3.8/psutil/tests
    copying psutil/tests/test_sunos.py -> build/lib.linux-x86_64-3.8/psutil/tests
    running build_ext
    building 'psutil._psutil_linux' extension
    creating build/temp.linux-x86_64-3.8
    creating build/temp.linux-x86_64-3.8/psutil
    x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=572 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/venv/include -I/usr/include/python3.8 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-3.8/psutil/_psutil_common.o
    unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
    C compiler or Python headers are not installed on this system. Try to run:
    sudo apt-get install gcc python3-dev
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /venv/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bk6r76gx/psutil/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bk6r76gx/psutil/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-7446rbqr/install-record.txt --single-version-externally-managed --compile --install-headers /venv/include/site/python3.8/psutil Check the logs for full command output.
ERROR: Service 'worker' failed to build : The command '/bin/sh -c pip install -r /edx/app/enterprise_catalog/enterprise_catalog/requirements/production.txt' returned a non-zero code: 1

@bbaker6225
Copy link
Contributor

Just tried on master and I can confirm that the same provisioning process succeeds using python 3.5 there

@mraarif
Copy link
Contributor Author

mraarif commented Sep 30, 2020

@bbaker6225 you should be able to run make dev.provision now

@mraarif
Copy link
Contributor Author

mraarif commented Sep 30, 2020

I tested the changes by running make docker_build, make dev.up, make dev.provision, make app-shell and make requirements

@bbaker6225
Copy link
Contributor

Great, I'll give it another run shortly

@bbaker6225
Copy link
Contributor

bbaker6225 commented Sep 30, 2020

Thanks for the updates, everything works great except for this error in the worker logs:

Traceback (most recent call last):
  File "/venv/bin/celery", line 8, in <module>
    sys.exit(main())
  File "/venv/lib/python3.8/site-packages/celery/__main__.py", line 16, in main
    _main()
  File "/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 322, in main
    cmd.execute_from_commandline(argv)
  File "/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 499, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/venv/lib/python3.8/site-packages/celery/bin/base.py", line 305, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 491, in handle_argv
    return self.execute(command, argv)
  File "/venv/lib/python3.8/site-packages/celery/bin/celery.py", line 415, in execute
    return cls(
  File "/venv/lib/python3.8/site-packages/celery/bin/worker.py", line 221, in run_from_argv
    *self.parse_options(prog_name, argv, command))
  File "/venv/lib/python3.8/site-packages/celery/bin/base.py", line 428, in parse_options
    self.parser = self.create_parser(prog_name, command)
  File "/venv/lib/python3.8/site-packages/celery/bin/base.py", line 440, in create_parser
    description=self._format_description(self.description),
  File "/venv/lib/python3.8/site-packages/celery/bin/base.py", line 462, in _format_description
    text.fill_paragraphs(text.dedent(description), width))
  File "/venv/lib/python3.8/site-packages/celery/utils/text.py", line 58, in fill_paragraphs
    return sep.join(fill(p, width) for p in s.split(sep))
  File "/venv/lib/python3.8/site-packages/celery/utils/text.py", line 58, in <genexpr>
    return sep.join(fill(p, width) for p in s.split(sep))
  File "/usr/lib/python3.8/textwrap.py", line 391, in fill
    return w.fill(text)
  File "/usr/lib/python3.8/textwrap.py", line 363, in fill
    return "\n".join(self.wrap(text))
  File "/usr/lib/python3.8/textwrap.py", line 354, in wrap
    return self._wrap_chunks(chunks)
  File "/usr/lib/python3.8/textwrap.py", line 248, in _wrap_chunks
    raise ValueError("invalid width %r (must be > 0)" % self.width)
ValueError: invalid width -2 (must be > 0)

I can't find many results on google, but one I did find suggested it might be a local only issue with logging to the console. Do you have any ideas?

@mraarif
Copy link
Contributor Author

mraarif commented Oct 1, 2020

@bbaker6225 seems like this GitHub issue: celery/celery#5761 has the same error I think

@bbaker6225
Copy link
Contributor

Hm you're right, looks like adding

environment:
  - COLUMNS=80

might fix it

@bbaker6225
Copy link
Contributor

I'm changing the worker environment in docker-compose.yml to

    environment:
      CELERY_ALWAYS_EAGER: 'false'
      CELERY_BROKER_TRANSPORT: redis
      CELERY_BROKER_HOSTNAME: edx.devstack.redis:6379
      CELERY_BROKER_VHOST: 0
      CELERY_BROKER_PASSWORD: password
      DJANGO_SETTINGS_MODULE: enterprise_catalog.settings.devstack
      COLUMNS: 80

we'll see if that works

@bbaker6225
Copy link
Contributor

@mraarif yep that worked. I've updated the PR with it and will merge it once tests pass

@bbaker6225 bbaker6225 merged commit 48f05b9 into master Oct 1, 2020
@bbaker6225 bbaker6225 deleted the BOM-2001 branch October 1, 2020 17:09
irfanuddinahmad pushed a commit that referenced this pull request Jul 24, 2024
* Updated the Dockerfile to switch to python 3.8

* removed unused curl installation

* Add columns:80 as env variable

Co-authored-by: Brandon Baker <bbaker@edx.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants