Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[shardformer] write an shardformer example with bert finetuning #4111

Closed
wants to merge 0 commits into from
Closed

[shardformer] write an shardformer example with bert finetuning #4111

wants to merge 0 commits into from

Conversation

flybird11111
Copy link
Contributor

📌 Checklist before creating the PR

  • I have created an issue for this PR for traceability
  • The title follows the standard format: [doc/gemini/tensor/...]: A concise description
  • I have added relevant tags if possible for us to better distinguish different PRs

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge

e.g. fixed #1234, closed #1234, resolved #1234

#4110

📝 What does this PR do?

Summarize your work here.
if you have any plots/diagrams/screenshots/tables, please attach them here.

write an shardformer example with bert finetuning

💥 Checklist before requesting a review

  • I have linked my PR to an issue (instruction)
  • My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
  • I have performed a self-review of my code
  • I have added thorough tests.
  • I have added docstrings for all the functions/methods I implemented

⭐️ Do you enjoy contributing to Colossal-AI?

  • 🌝 Yes, I do.
  • 🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

@flybird11111 flybird11111 added the example example-related issuer or pull request label Jun 28, 2023
@FrankLeeeee FrankLeeeee linked an issue Jun 28, 2023 that may be closed by this pull request
[shardformer] write an shardformer example with bert finetuning #4110
Closed
FrankLeeeee
FrankLeeeee reviewed Jun 30, 2023
View reviewed changes
examples/language/bert/test_ci.sh Outdated
@@ -6,3 +6,5 @@ pip install -r requirements.txt
for plugin in "torch_ddp" "torch_ddp_fp16" "gemini" "low_level_zero"; do
torchrun --standalone --nproc_per_node 4 finetune.py --target_f1 0.86 --plugin $plugin --model_type "bert"
done

torchrun --standalone --nproc_per_node=1 shardformer_benchmark.py
Copy link
Contributor

@FrankLeeeee FrankLeeeee Jun 30, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to test the sharding, please use more than 1 GPU and set a small number of iteration so that it can finish running within 1-2 minutes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FrankLeeeee FrankLeeeee FrankLeeeee left review comments

@ver217 ver217 Awaiting requested review from ver217

Assignees
No one assigned
Labels
example example-related issuer or pull request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[shardformer] write an shardformer example with bert finetuning
2 participants
@flybird11111 @FrankLeeeee