You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Documenting this feature request from @mf-silva as supporting per-sample L2 gradient norms for LayerNorm allows estimating importance scores for data points on LLM architectures which often have LayerNorm.
A good starting point to implement this is to take a look at the custom first-order extension example in the docs.
The text was updated successfully, but these errors were encountered:
Documenting this feature request from @mf-silva as supporting per-sample L2 gradient norms for
LayerNorm
allows estimating importance scores for data points on LLM architectures which often haveLayerNorm
.A good starting point to implement this is to take a look at the custom first-order extension example in the docs.
The text was updated successfully, but these errors were encountered: