feat: build manylinux compatible builds #1527

narendasan · 2022-12-05T15:59:11Z

Description

Start building wheels that a manylinux compatible

Fixes #1402

Type of change

Please delete options that are not relevant and/or add your own.

New feature (non-breaking change which adds functionality)

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

…nylinux

Signed-off-by: Naren Dasan <narens@nvidia.com>

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

Signed-off-by: Naren Dasan <narens@nvidia.com>

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

Signed-off-by: Naren Dasan <narens@nvidia.com>

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

…nylinux

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

peri044

Can you add a short README (for us developers) which contains the instruction on how to build such manylinux compatible builds with Circle CI?

peri044 · 2022-12-14T16:09:40Z

py/ci/build_whl.sh

+#build_py37
+#build_py38
+#build_py39
+#build_py310
 #build_py311
-build_libtorchtrt_pre_cxx11_abi
+#build_libtorchtrt_pre_cxx11_abi
 #build_libtorchtrt


Can this be removed ?

peri044 · 2022-12-14T16:30:31Z

py/ci/build_whl.sh

@@ -2,40 +2,66 @@

 # Example usage: docker run -it -v$(pwd)/..:/workspace/TRTorch build_trtorch_wheel /bin/bash /workspace/TRTorch/py/build_whl.sh


Replace trtorch references here

peri044 · 2022-12-14T16:35:36Z

py/ci/build_whl.sh

+    ${PY_DIR}/bin/python -m pip install -r requirements.txt
+    ${PY_DIR}/bin/python -m pip install setuptools wheel auditwheel
+    ${PY_DIR}/bin/python setup.py bdist_wheel --release --ci
+    LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${PY_PKG_DIR}/torch/lib:${PY_PKG_DIR}/tensorrt/:${CUDA_HOME}/lib64:${CUDA_HOME}/lib64/stubs ${PY_DIR}/bin/python -m auditwheel repair $(cat ${PROJECT_DIR}/py/ci/soname_excludes.params) --plat manylinux_2_17_x86_64 dist/torch_tensorrt-*-${PY_BUILD_CODE}-linux_x86_64.whl


Can you let me know what is happening here with this soname_excludes ?

peri044 · 2022-12-14T16:36:57Z

py/ci/build_whl.sh

+    ${PY_DIR}/bin/python -m pip install -r requirements.txt
+    ${PY_DIR}/bin/python -m pip install setuptools wheel auditwheel
+    ${PY_DIR}/bin/python setup.py bdist_wheel --release --ci
+    LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${PY_PKG_DIR}/torch/lib:${PY_PKG_DIR}/tensorrt/:${CUDA_HOME}/lib64:${CUDA_HOME}/lib64/stubs ${PY_DIR}/bin/python -m auditwheel repair  $(cat ${PROJECT_DIR}/py/ci/soname_excludes.params) --plat manylinux_2_17_x86_64 dist/torch_tensorrt-*-${PY_BUILD_CODE}-linux_x86_64.whl
 }

 #build_py311() {


Should we remove this and add later ?

py/ci/build_whl.sh

.circleci/config.yml

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

peri044 · 2022-12-17T22:51:51Z

status :
If I don't have torch and tensorrt installed and I run pip install -i https://test.pypi.org/simple/ torch-tensorrt, I get the error

     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.3/20.3 MB 7.1 MB/s eta 0:00:00
  Downloading https://test-files.pythonhosted.org/packages/3a/1e/bbd08b7b2ef7f96cd4351054deab66457834f4e1436ccb169037e71a39e7/torch-tensorrt-0.0.0.tar.gz (8.9 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [13 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-h8cldzod/torch-tensorrt_f8be91da1be844129bb09646e54b371f/setup.py", line 125, in <module>
          raise RuntimeError(open("ERROR.txt", "r").read())
      RuntimeError:

If they are installed, I don't and it works

peri044

Linter fails with some weird error. The changes LGTM.
Tested with pip install --extra-index-url https://test.pypi.org/simple/ torch-tensorrt and it works fine.

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

feat: build manylinux compatible builds

7c946ff

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

facebook-github-bot added the cla signed label Dec 5, 2022

github-actions bot added the component: api [Python] Issues re: Python API label Dec 5, 2022

narendasan and others added 2 commits December 5, 2022 12:17

chore: pickup builds from wheelhouse

c808089

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

ci: Parallelize packaging builds

31d0a0a

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

narendasan force-pushed the manylinux branch from 950b48a to 31d0a0a Compare December 5, 2022 17:32

chore: split builds

ac3c596

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

narendasan force-pushed the manylinux branch from 82fb1bc to ac3c596 Compare December 5, 2022 17:42

narendasan added 2 commits December 5, 2022 09:59

feat: do testing with release packages

35b97dc

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

chore: fix splitting

59df0fb

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

github-actions bot added the component: tests Issues re: Tests label Dec 7, 2022

narendasan and others added 18 commits December 6, 2022 23:45

Update config.yml

a920e21

chore: install auditwheel

6891c24

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

Merge branch 'manylinux' of https://github.com/NVIDIA/TRTorch into ma…

50e3193

…nylinux

Auditwheel

3e787c7

Signed-off-by: Naren Dasan <narens@nvidia.com>

refactor: auditwheel

a7c17b7

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

chore: put builds in right place

5b64012

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

fix

f68e93d

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

chore: more fixes for ci

3a49585

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

chore: remove old build task

eeb4e1c

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

fix

81f3f8e

Signed-off-by: Naren Dasan <narens@nvidia.com>

ci things

d67eb13

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

""

c8a8267

Signed-off-by: Naren Dasan <narens@nvidia.com>

""

74f9506

Signed-off-by: Naren Dasan <narens@nvidia.com>

""

96620d5

Signed-off-by: Naren Dasan <narens@nvidia.com>

Precise install

ac96016

feat: Dont reship every .so

e3202e9

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

Merge branch 'manylinux' of https://github.com/NVIDIA/TRTorch into ma…

18dd56c

…nylinux

fix: address invalid classifier

478d2a1

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

peri044 requested changes Dec 14, 2022

View reviewed changes

chore: better ignores

a780c45

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

narendasan added 3 commits January 30, 2023 17:09

Ship libgomp

b567f3a

Dont need to ship libgomp, shipped with torch

fabe47d

Pull in deps for testing

fc56c3d

peri044 approved these changes Feb 15, 2023

View reviewed changes

narendasan merged commit 3d6a1ba into release/1.3 Feb 15, 2023

narendasan added a commit that referenced this pull request Feb 15, 2023

feat: build manylinux compatible builds (#1527)

57504d9

Signed-off-by: Naren Dasan <naren@narendasan.com> Signed-off-by: Naren Dasan <narens@nvidia.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: build manylinux compatible builds #1527

feat: build manylinux compatible builds #1527

narendasan commented Dec 5, 2022

peri044 left a comment

peri044 Dec 14, 2022

peri044 Dec 14, 2022

peri044 Dec 14, 2022

peri044 Dec 14, 2022

peri044 commented Dec 17, 2022

peri044 left a comment

		@@ -2,40 +2,66 @@

		# Example usage: docker run -it -v$(pwd)/..:/workspace/TRTorch build_trtorch_wheel /bin/bash /workspace/TRTorch/py/build_whl.sh

feat: build manylinux compatible builds #1527

feat: build manylinux compatible builds #1527

Conversation

narendasan commented Dec 5, 2022

Description

Type of change

peri044 left a comment

Choose a reason for hiding this comment

peri044 Dec 14, 2022

Choose a reason for hiding this comment

peri044 Dec 14, 2022

Choose a reason for hiding this comment

peri044 Dec 14, 2022

Choose a reason for hiding this comment

peri044 Dec 14, 2022

Choose a reason for hiding this comment

peri044 commented Dec 17, 2022

peri044 left a comment

Choose a reason for hiding this comment