Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimized set_initialized_submodules. #35493

Merged
merged 1 commit into from
Jan 21, 2025

Conversation

LagPixelLOL
Copy link
Contributor

Fixed an insanely slow function set_initialized_submodules, when loading DeepSeek V3, since it has so many experts, the amount of state_dict_keys are huge and also there are a lot of named_modules.

loaded_keys = {k.replace(f"{module_name}.", "") for k in state_dict_keys if k.startswith(f"{module_name}.")}

This line, especially k.startswith(f"{module_name}."), gets called at least 1 million times, which is insanely slow, it takes at least 10 minutes runtime.

My optimization reduces the runtime to less than 10 seconds by not iterating through state_dict_keys and deleting module_name from the start for every named_module, instead, it prepends the module_name before the module.state_dict() keys.

cc: @ArthurZucker

Copy link
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arf sorry for my super late review. This does make sense, don't know why we were re-computing the set so many time.
Thanks 🤗

@ArthurZucker ArthurZucker merged commit 568941b into huggingface:main Jan 21, 2025
23 checks passed
bursteratom pushed a commit to bursteratom/transformers that referenced this pull request Jan 31, 2025
elvircrn pushed a commit to elvircrn/transformers that referenced this pull request Feb 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants