Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Destination flipping for add!!(x, y) #346

Open
oxinabox opened this issue May 4, 2021 · 1 comment
Open

Destination flipping for add!!(x, y) #346

oxinabox opened this issue May 4, 2021 · 1 comment
Labels
inplace accumulation for things relating to inplace accumulation of gradients

Comments

@oxinabox
Copy link
Member

oxinabox commented May 4, 2021

Right now add!!(x, y) will only ever mutate x, never y.
because of the logically linear nature of AD, y being a differential (i.e. the output of pullback) means it is used exactly once .
Which means we are infact free to mutate it, also since this will be it's only use. (same also says why we are always ok to mutate x)
See google-research/dex-lang#401 (comment)
(and to a lesser extent https://twitter.com/oxinabox_frames/status/1387722380978622468)

So we could pick either x or y depending on which was a valid inplacable_destination.
And maybe also always chose Array if one of them is an Array (since that is the best destination if that has been allocated)

This would solve at least the basic cases of #345

@oxinabox oxinabox added the inplace accumulation for things relating to inplace accumulation of gradients label May 4, 2021
@oxinabox
Copy link
Member Author

oxinabox commented May 4, 2021

Argument agaist this is that if y comes from identity_pullback(y)=y
and was provided by the user then the user might be using it for something else also.

Kind feel like that shouldn't be our problem though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
inplace accumulation for things relating to inplace accumulation of gradients
Projects
None yet
Development

No branches or pull requests

1 participant