Flip default on weights_only #137602

mikaylagawarecki · 2024-10-09T16:16:17Z

Stack from ghstack (oldest at bottom):

cc @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames @rec

[ghstack-poisoned]

pytorch-bot · 2024-10-09T16:16:21Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/137602

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 27c7380 with merge base 73fde0d ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: e75d4173cf10156143ade29d60b2cd8ac346a9f2 Pull Request resolved: #137602

[ghstack-poisoned]

ghstack-source-id: f9d26b50df5e20dc7beaedf203e64324bf408884 Pull Request resolved: #137602

[ghstack-poisoned]

mikaylagawarecki · 2024-10-16T16:15:42Z

.github/ci_commit_pins/xla.txt

@@ -1 +1 @@
-2eb4a60ed14a38260b85b0c765161f0ce45be6d1
+f71c02d1f457d58371e013632efb016c01bd1866


This was intentional to include pytorch/xla#8259

kit1980 · 2024-10-16T17:11:34Z

I'm not sure if it's OK to use different default in fbcode/OSS.
cc @malfet

kit1980 · 2024-10-16T17:13:04Z

Also this will break a lot of people.
Maybe it's time to break.

malfet

Tentatively LGTM, but please change the signatures a bit further. And not IS_FBCODE feels a bit suspicious, perhaps better behavior would be to add weights_only=False to respective callsights?

malfet · 2024-10-16T17:19:04Z

.github/ci_commit_pins/xla.txt

@@ -1 +1 @@
-2eb4a60ed14a38260b85b0c765161f0ce45be6d1
+f71c02d1f457d58371e013632efb016c01bd1866


This is not intended, is it?

This is intended, see #137602 (comment)

malfet · 2024-10-16T17:22:07Z

torch/serialization.py

@@ -1159,7 +1160,7 @@ def load(
    map_location: MAP_LOCATION = None,
    pickle_module: Any = None,
    *,
-    weights_only: Optional[bool] = None,
+    weights_only: Optional[bool] = not IS_FBCODE,


if you are changing it to hard boolean, please remove Optional and delete the logic associated with opt-in checks

Suggested change

weights_only: Optional[bool] = not IS_FBCODE,

weights_only: bool = not IS_FBCODE,

kit1980 · 2024-10-16T17:30:15Z

And just FYI users can use TorchFix to find and update call sites where weights_only is not explicitly set: https://github.com/pytorch-labs/torchfix?tab=readme-ov-file#tor102-torchload-without-weights_only-parameter-is-unsafe

albanD

A couple more pieces need to be added to the CI and we should announce this one as a bc-breaking change broadly for sure!
But yes let's do it!

Summary: Some fixes for pytorch/pytorch#137602 Reviewed By: xuzhao9 Differential Revision: D64628614 Pulled By: mikaylagawarecki

Summary: Some fixes for pytorch/pytorch#137602 Pull Request resolved: #2514 Reviewed By: xuzhao9 Differential Revision: D64628614 Pulled By: mikaylagawarecki fbshipit-source-id: edebf25cc6648919d5673a3baeaffdac26e5b91f

[ghstack-poisoned]

ghstack-source-id: 97e14307bf0b69477e818622d05d436e9a160934 Pull Request resolved: #137602

After #137602, the default `weights_only` has been set to True. This test is failing in trunk slow jobs atm benchmark_utils/test_benchmark_utils.py::TestBenchmarkUtils::test_collect_callgrind [GH job link](https://github.com/pytorch/pytorch/actions/runs/11672436111/job/32502454946) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/1aa71be56c39908893273bd9558b127159e1ef3a) Pull Request resolved: #139810 Approved by: https://github.com/kit1980

Prevent same global from being added multiple times Pull Request resolved: pytorch#139303 Approved by: https://github.com/janeyx99 ghstack dependencies: pytorch#138936, pytorch#139221, pytorch#139433, pytorch#139541, pytorch#137602

…139810) After pytorch#137602, the default `weights_only` has been set to True. This test is failing in trunk slow jobs atm benchmark_utils/test_benchmark_utils.py::TestBenchmarkUtils::test_collect_callgrind [GH job link](https://github.com/pytorch/pytorch/actions/runs/11672436111/job/32502454946) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/1aa71be56c39908893273bd9558b127159e1ef3a) Pull Request resolved: pytorch#139810 Approved by: https://github.com/kit1980

cc mikaylagawarecki albanD ghstack-source-id: 6fc7434a259f92b0fca8875b20ac22624ecf1a03 Pull Request resolved: #2558

…139810) After pytorch#137602, the default `weights_only` has been set to True. This test is failing in trunk slow jobs atm benchmark_utils/test_benchmark_utils.py::TestBenchmarkUtils::test_collect_callgrind [GH job link](https://github.com/pytorch/pytorch/actions/runs/11672436111/job/32502454946) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/1aa71be56c39908893273bd9558b127159e1ef3a) Pull Request resolved: pytorch#139810 Approved by: https://github.com/kit1980

cc mikaylagawarecki albanD ghstack-source-id: 6fc7434a259f92b0fca8875b20ac22624ecf1a03 Pull Request resolved: #2558 (cherry picked from commit 165163a)

Starting from version 2.4 PyTorch introduces a stricter check for the objects which can be loaded with torch.load(). Starting from version 2.6 loading with weights_only=True requires allowlisting of such objects. This commit adds allowlist of some numpy objects used to load model checkpoints. Usage is restricted by context manager. User can still additionally call torch.serialization.add_safe_globals() to add other objects into the safe globals list. Accelerate library also stepped into same problem and addressed it with PR-3036. Fixes: huggingface#34631 See: pytorch/pytorch#137602 See: https://pytorch.org/docs/stable/notes/serialization.html#torch.serialization.add_safe_globals See: huggingface/accelerate#3036 Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>

Starting from version 2.4 PyTorch introduces a stricter check for the objects which can be loaded with torch.load(). Starting from version 2.6 loading with weights_only=True requires allowlisting of such objects. This commit adds allowlist of some numpy objects used to load model checkpoints. Usage is restricted by context manager. User can still additionally call torch.serialization.add_safe_globals() to add other objects into the safe globals list. Accelerate library also stepped into same problem and addressed it with PR-3036. Fixes: #34631 See: pytorch/pytorch#137602 See: https://pytorch.org/docs/stable/notes/serialization.html#torch.serialization.add_safe_globals See: huggingface/accelerate#3036 Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>

Pull Request resolved: pytorch#137602 Approved by: https://github.com/malfet, https://github.com/albanD ghstack dependencies: pytorch#138936, pytorch#139221, pytorch#139433, pytorch#139541

Prevent same global from being added multiple times Pull Request resolved: pytorch#139303 Approved by: https://github.com/janeyx99 ghstack dependencies: pytorch#138936, pytorch#139221, pytorch#139433, pytorch#139541, pytorch#137602

…139810) After pytorch#137602, the default `weights_only` has been set to True. This test is failing in trunk slow jobs atm benchmark_utils/test_benchmark_utils.py::TestBenchmarkUtils::test_collect_callgrind [GH job link](https://github.com/pytorch/pytorch/actions/runs/11672436111/job/32502454946) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/1aa71be56c39908893273bd9558b127159e1ef3a) Pull Request resolved: pytorch#139810 Approved by: https://github.com/kit1980

Starting from version 2.4 PyTorch introduces a stricter check for the objects which can be loaded with torch.load(). Starting from version 2.6 loading with weights_only=True requires allowlisting of such objects. This commit adds allowlist of some numpy objects used to load model checkpoints. Usage is restricted by context manager. User can still additionally call torch.serialization.add_safe_globals() to add other objects into the safe globals list. Accelerate library also stepped into same problem and addressed it with PR-3036. Fixes: huggingface#34631 See: pytorch/pytorch#137602 See: https://pytorch.org/docs/stable/notes/serialization.html#torch.serialization.add_safe_globals See: huggingface/accelerate#3036 Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>

Flip default on weights_only

4839d5a

[ghstack-poisoned]

This was referenced Oct 9, 2024

Remove dependency on numpy for serialization for XLA/open registration devices without numpy #137444

Closed

Remove numpy dependency for maia serialization #137600

Closed

mikaylagawarecki added a commit that referenced this pull request Oct 9, 2024

Flip default on weights_only

25fcbad

ghstack-source-id: e75d4173cf10156143ade29d60b2cd8ac346a9f2 Pull Request resolved: #137602

mikaylagawarecki added release notes: python_frontend python frontend release notes category topic: bc breaking topic category labels Oct 9, 2024

Update on "Flip default on weights_only"

a141fb5

[ghstack-poisoned]

mikaylagawarecki added a commit that referenced this pull request Oct 9, 2024

Flip default on weights_only

6bd4ece

ghstack-source-id: f9d26b50df5e20dc7beaedf203e64324bf408884 Pull Request resolved: #137602

JackCaoG mentioned this pull request Oct 15, 2024

Set weights_only to False for xser.load pytorch/xla#8259

Merged

Update on "Flip default on weights_only"

12ac158

[ghstack-poisoned]

mikaylagawarecki requested a review from a team as a code owner October 16, 2024 16:08

pytorch-bot bot added the ciflow/inductor label Oct 16, 2024

mikaylagawarecki requested a review from albanD October 16, 2024 16:15

mikaylagawarecki commented Oct 16, 2024

View reviewed changes

malfet approved these changes Oct 16, 2024

View reviewed changes

albanD approved these changes Oct 16, 2024

View reviewed changes

mikaylagawarecki added ciflow/trunk Trigger trunk jobs on your pull request ci-no-td Do not run TD on this PR labels Oct 17, 2024

mikaylagawarecki mentioned this pull request Oct 18, 2024

Fixes to prep for weights_only default flip pytorch/benchmark#2514

Closed

Update on "Flip default on weights_only"

9062584

[ghstack-poisoned]

mikaylagawarecki requested a review from kulinseth as a code owner October 22, 2024 16:14

mikaylagawarecki added a commit that referenced this pull request Oct 22, 2024

Flip default on weights_only

bcb59c6

ghstack-source-id: 97e14307bf0b69477e818622d05d436e9a160934 Pull Request resolved: #137602

dvrogozh mentioned this pull request Nov 6, 2024

safe_globals are needed to resume training on upcoming PyTorch 2.6 huggingface/transformers#34631

Closed

dvrogozh mentioned this pull request Nov 6, 2024

Add safe_globals to resume training on PyTorch 2.6 huggingface/transformers#34632

Merged

vmoens added a commit to pytorch/rl that referenced this pull request Nov 13, 2024

[BugFix] Fix tests failing because of pytorch/pytorch#137602

98ae8c0

cc mikaylagawarecki albanD ghstack-source-id: 6fc7434a259f92b0fca8875b20ac22624ecf1a03 Pull Request resolved: #2558

vmoens added a commit to pytorch/rl that referenced this pull request Nov 13, 2024

[BugFix] Fix tests failing because of pytorch/pytorch#137602

165163a

cc mikaylagawarecki albanD ghstack-source-id: 6fc7434a259f92b0fca8875b20ac22624ecf1a03 Pull Request resolved: #2558

ClowDragon mentioned this pull request Nov 26, 2024

torch.load error pytorch/opacus#690

Closed

github-actions bot deleted the gh/mikaylagawarecki/269/head branch December 5, 2024 02:14

mikaylagawarecki mentioned this pull request Dec 5, 2024

I think it is necessary to set torch.load with wetigh-only=True by default #142126

Closed

dvrogozh mentioned this pull request Dec 7, 2024

[XPU] model works with 2.5.1 while break with nightly build #142123

Closed

chang-l mentioned this pull request Jan 6, 2025

[TestFailure] Resolving exceptions thrown across multiple GraphBolt tests. dmlc/dgl#7852

Merged

4 tasks

joshuacwnewton mentioned this pull request Jan 28, 2025

PyTorch 2.6 has changed the default value of weights_only to True, breaking the loading of some nnunetv2 models MIC-DKFZ/nnUNet#2681

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flip default on weights_only #137602

Flip default on weights_only #137602

mikaylagawarecki commented Oct 9, 2024 •

edited

Loading

pytorch-bot bot commented Oct 9, 2024 •

edited

Loading

mikaylagawarecki Oct 16, 2024

kit1980 commented Oct 16, 2024

kit1980 commented Oct 16, 2024

malfet left a comment

malfet Oct 16, 2024

mikaylagawarecki Oct 16, 2024

malfet Oct 16, 2024

kit1980 commented Oct 16, 2024

albanD left a comment

		@@ -1 +1 @@
		2eb4a60ed14a38260b85b0c765161f0ce45be6d1
		f71c02d1f457d58371e013632efb016c01bd1866

	weights_only: Optional[bool] = not IS_FBCODE,
	weights_only: bool = not IS_FBCODE,

Flip default on weights_only #137602

Flip default on weights_only #137602

Conversation

mikaylagawarecki commented Oct 9, 2024 • edited Loading

pytorch-bot bot commented Oct 9, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/137602

✅ No Failures

mikaylagawarecki Oct 16, 2024

Choose a reason for hiding this comment

kit1980 commented Oct 16, 2024

kit1980 commented Oct 16, 2024

malfet left a comment

Choose a reason for hiding this comment

malfet Oct 16, 2024

Choose a reason for hiding this comment

mikaylagawarecki Oct 16, 2024

Choose a reason for hiding this comment

malfet Oct 16, 2024

Choose a reason for hiding this comment

kit1980 commented Oct 16, 2024

albanD left a comment

Choose a reason for hiding this comment

mikaylagawarecki commented Oct 9, 2024 •

edited

Loading

pytorch-bot bot commented Oct 9, 2024 •

edited

Loading