Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pytorch CUDA Upgrade to 11.7 and Decommsion 11.3 and 10.2 #1042

Open
31 of 34 tasks
atalman opened this issue Jun 1, 2022 · 2 comments
Open
31 of 34 tasks

Pytorch CUDA Upgrade to 11.7 and Decommsion 11.3 and 10.2 #1042

atalman opened this issue Jun 1, 2022 · 2 comments

Comments

@atalman
Copy link
Contributor

atalman commented Jun 1, 2022 •
edited
Loading

This issue will track the current progress on upgrading CUDA 11.7 support, and decommission legacy CUDA version

Cuda Support Matrix as of Pytorch 1.12

CUDA CUDNN additional details
10.2 7.6.5.32 Legacy CUDA Release, to be decommissioned issue
11.3 8.3.2.44 Stable CUDA Release
11.6 8.3.2.44 Latest CUDA Release

Pre CUDA 11.7 Upgrade

This issue is required to move CUDA 11.6 to Stable version. And we want to address it before CUDA 11.7.

  • Follow Up on the usage for cudatoolkit across pytorch projects pytorch#69691 Conda-forge dependency for 11.6 for cudatoolkit. In short Since CUDA 11.5, cudatoolkit is only available on conda-forge channel. We should migrate from cudatoolkit to cuda and abandon usage of conda-forge from pytorch, torchvision and torchaudio. This work should be scheduled and addressed as soon as we cut release 1.12 for pytorch and all domain libraries.

Decommission CUDA 10.2

This can be done in parallel to CUDA 11.7 upgrade. We want to ultimately address it before 11.7, but can also be done in parallel.

Upgrade CUDA 11.7

As per https://github.com/pytorch/builder/blob/main/CUDA_UPGRADE_GUIDE.MD

  • Installing to conda-builder and libtorch containers
    • Push pytorch/conda-builder
    • Push the libtorch image
  • Add setup to manywheels
    • Push pytorch/manylinux-builder
  • Update MAGMA
    • Push magma-cuda117 to conda
    • Add magma for windows into our S3
  • Add Windows builder for 11.7
    • Check if driver needs to be updated
    • Add fixes that had to come up
  • Include CUDA 11.7 into our nightly matrix
    • Update conda build_pytorch.sh script and add conda binaries
    • Windows
    • Linux
    • MacOS
    • Add fixes that had to come up
  • Create 11.7 CI
    • Windows
    • Linux + add MAGMA to CI conda
  • Add 11.7 to torchvision CI
  • Add 11.7 to torchaudio CI

Past Issues to be Resolved by upgrade (needs to be retested)

Post CUDA 11.7 Upgrade

Target End State

CUDA 11.6 - Stable, CUDA 11.7 - Latest Experimental
CUDA 10.2 and CUDA 11.3 Decommissioned

BE tasks for Meta Team

cc @ptrblck @malfet @seemethere @ezyang @pytorch/pytorch-dev-infra @ngimel
@atalman atalman mentioned this issue Jun 2, 2022
Open
This was referenced Jun 3, 2022
Merged
Merged
@ptrblck
Copy link
Contributor

ptrblck commented Jun 3, 2022

For 11.7:
Created two PRs to add the docker and magma builds.

CC @crcrpar
Could you check the launch bounds for torch.mode in 11.7, please?

CC @IvanYashchuk
Adding you for potentially needed MAGMA fixes

This was referenced Jun 8, 2022
Merged
Merged
This was referenced Jul 8, 2022
Closed
Closed
@LeHenschel LeHenschel mentioned this issue Jul 8, 2022
Closed
pytorchmergebot pushed a commit to pytorch/pytorch that referenced this issue Jul 13, 2022
@ptrblck @pytorchmergebot
Add linux cuda 11.7 workflows (#81089)
12c30a8 
Ref pytorch/builder#1042

CC @atalman
Pull Request resolved: #81089
Approved by: https://github.com/atalman
facebook-github-bot pushed a commit to pytorch/pytorch that referenced this issue Jul 14, 2022
@ptrblck @facebook-github-bot
Add linux cuda 11.7 workflows (#81089) (#81089)
32da950 
Summary:
Ref pytorch/builder#1042

CC atalman

Pull Request resolved: #81089
Approved by: https://github.com/atalman

Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/12c30a8250d1a00a0ac2e03d70ec909c76d922c3

Reviewed By: DanilBaibak

Differential Revision: D37847365

Pulled By: DanilBaibak

fbshipit-source-id: 6a412c452643938c7e3ec9a45360eb739044aa55
pytorchmergebot pushed a commit to pytorch/pytorch that referenced this issue Jul 18, 2022
@ptrblck @pytorchmergebot
Add CUDA 11.7 workflows (#81095)
99c464a 
Ref pytorch/builder#1042

CC @atalman

Pull Request resolved: #81095
Approved by: https://github.com/atalman
facebook-github-bot pushed a commit to pytorch/pytorch that referenced this issue Jul 19, 2022
@ptrblck @facebook-github-bot
Add CUDA 11.7 workflows (#81095) (#81095)
e13e8d7 
Summary:
Ref pytorch/builder#1042

CC atalman

Pull Request resolved: #81095
Approved by: https://github.com/atalman

Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/99c464ae26ca4dbdb39e136da0f15c50d245f2d3

Reviewed By: jeanschmidt

Differential Revision: D37940360

Pulled By: jeanschmidt

fbshipit-source-id: 7789c02b0c706eb3079ffc786a2197ec7d6f6449
@atalman atalman mentioned this issue Jul 28, 2022
Open
@atalman atalman mentioned this issue Sep 30, 2022
Closed
@bryantbiggs
Copy link
Contributor

I believe this is now complete, correct @atalman ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@atalman @bryantbiggs @ptrblck