Use float on cpu if fp16 when filling w/ -inf in w2v2 #5
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
taylanbil
force-pushed
the
fix-fp16-indexput
branch
from
c851c9c to
75ea85e
Compare
bilgeacun
approved these changes
Dec 10, 2020
bilgeacun
added a commit
that referenced
this pull request
Feb 6, 2021
input shape temp update clean up dataset updates clean up move tensor idx to matrix op inside apply_mask use tensor operators to replace tensor indexing, passed consistency test verification Minor improvements Fix bucketpadlendataset Moved mask matrices creation to dataset prep. Remove dynamism, apply mask correctly, add some guardrails, some cleanups. Send device data to cpu b4 logging. Fix data bucketing for RawAudioDataset, refactor bucketing functions, fix filling w/ -inf in wav2vec2, minor cleanups Sample size computeation during data prep to reduce atens, dont call item in log_scalar, minor cleanups Remove extra validation atens, clean up marking step and sending to cpu. Correct loss computation for w2v2 criterion + refactor index_put Fix bug in index_put + fix integer division Dont call float on extra logs, clean up comment. Correct accuracy computation, refactor xla tensor check. Adjust loss computation so it works w/ binary cross entropy. Remove sending log outputs back to cpu after allreduce. Dont sample padded states when sampling negatives + correct mi in loss computation. Fixing config issues after rebase Fix bug in negatives from everywhere Fixing config issue for TPU after rebase Taylans changes on top of rebase Use float on cpu if fp16 when filling w/ -inf in w2v2 (#5) * Use float on cpu if fp16 when filling w/ -inf in w2v2 * xla -> self.xla * make logging_output_can_be_summed a regular method instead of staticmethod. Make tpu codepath work w/ hydra. (#6) * Make tpu codepath work w/ hydra. * Share and pass down model related args to rawaudiodataset correctly. * fp16 bug fix on non-xla devices (self._inftensor change) * use index_put to avoid dynamicity in model's fwd. * Get rid of some unnecessary warnings for tpus to clean up stderr. * Send logging outputs to cpu b4 logging to reduce atens. * Util function to move cpu tensors to tpu. * Use the util function to handle dummy batches to avoid crash at the end of epoch in distributed training. * fixing configs for precompute mask indices Co-authored-by: Bilge Acun <acun@fb.com>
bilgeacun
added a commit
that referenced
this pull request
Feb 11, 2021
input shape temp update clean up dataset updates clean up move tensor idx to matrix op inside apply_mask use tensor operators to replace tensor indexing, passed consistency test verification Minor improvements Fix bucketpadlendataset Moved mask matrices creation to dataset prep. Remove dynamism, apply mask correctly, add some guardrails, some cleanups. Send device data to cpu b4 logging. Fix data bucketing for RawAudioDataset, refactor bucketing functions, fix filling w/ -inf in wav2vec2, minor cleanups Sample size computeation during data prep to reduce atens, dont call item in log_scalar, minor cleanups Remove extra validation atens, clean up marking step and sending to cpu. Correct loss computation for w2v2 criterion + refactor index_put Fix bug in index_put + fix integer division Dont call float on extra logs, clean up comment. Correct accuracy computation, refactor xla tensor check. Adjust loss computation so it works w/ binary cross entropy. Remove sending log outputs back to cpu after allreduce. Dont sample padded states when sampling negatives + correct mi in loss computation. Fixing config issues after rebase Fix bug in negatives from everywhere Fixing config issue for TPU after rebase Taylans changes on top of rebase Use float on cpu if fp16 when filling w/ -inf in w2v2 (#5) * Use float on cpu if fp16 when filling w/ -inf in w2v2 * xla -> self.xla * make logging_output_can_be_summed a regular method instead of staticmethod. Make tpu codepath work w/ hydra. (#6) * Make tpu codepath work w/ hydra. * Share and pass down model related args to rawaudiodataset correctly. * fp16 bug fix on non-xla devices (self._inftensor change) * use index_put to avoid dynamicity in model's fwd. * Get rid of some unnecessary warnings for tpus to clean up stderr. * Send logging outputs to cpu b4 logging to reduce atens. * Util function to move cpu tensors to tpu. * Use the util function to handle dummy batches to avoid crash at the end of epoch in distributed training. * fixing configs for precompute mask indices Co-authored-by: Bilge Acun <acun@fb.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.