[Prim] add batch_norm composite rule #49894
Conversation
|
你的PR提交成功，感谢你对开源项目的贡献! |
| value_table[new_out.name] = new_out | ||
| to_bind[orig_out.name] = new_out.name | ||
| to_bind_rev[new_out.name] = orig_out.name | ||
| if new_out is not None: |
There was a problem hiding this comment.
This may cause some problem, not safe
There was a problem hiding this comment.
remove origin if it's not none
| broadcast_to(reshape(bias, stats_shape), x_hat.shape), | ||
| ) | ||
|
|
||
| batch_mean_ = assign(batch_mean) |
cyber-pioneer
force-pushed
the
composite_debug
branch
2 times, most recently
from
7d7fa8b
to
b3675ee
Compare
cyber-pioneer
force-pushed
the
composite_debug
branch
3 times, most recently
from
51fef9d
to
16d1225
Compare
| multiply(momentum, run_mean), | ||
| multiply( | ||
| subtract(ones(run_mean.shape, run_mean.dtype), momentum), | ||
| reshape(batch_mean, run_mean.shape), |
| multiply(momentum, run_var), | ||
| multiply( | ||
| subtract(ones(run_var.shape, run_var.dtype), momentum), | ||
| reshape(batch_var, run_var.shape), |
| reduce_axes, | ||
| keepdim=True, | ||
| ) | ||
| x_hat = divide( |
cyber-pioneer
force-pushed
the
composite_debug
branch
5 times, most recently
from
2187998
to
3035b49
Compare
| @@ -448,6 +448,30 @@ def _test_use_sync(value): | |||
| __sync_stat_with_flag(value) | |||
|
|
|||
|
|
|||
| # ops in forward_blacklisk will not be replaced by composite ops. | |||
| prim_config = {"forward_blacklist": []} | |||
There was a problem hiding this comment.
or merge this into backward blacklist
There was a problem hiding this comment.
later forward and backward blacklist will be united
| 1 if i in reduce_axes else s for i, s in enumerate(x.shape) | ||
| ) | ||
|
|
||
| batch_mean = zeros(run_mean.shape, run_mean.dtype) |
There was a problem hiding this comment.
Why make batched_mean as zeros?
There was a problem hiding this comment.
This is init value of batched_mean. If training, zeros will be replaced by actual bach value. If testing, batched_mean is unuseful.
| fetch_list=[y], | ||
| ) | ||
| paddle.disable_static() | ||
| core._set_prim_forward_enabled(False) |
There was a problem hiding this comment.
_set_prim_all_enabled(False)
| data_format, | ||
| use_global_stats, | ||
| ) | ||
| gradients = paddle.grad(res, x) |
There was a problem hiding this comment.
in forward fn, output of bn will be mupliplied by random vector, this is equal to feed random v.
cyber-pioneer
force-pushed
the
composite_debug
branch
4 times, most recently
from
a03cba0
to
6e2d5b0
Compare
move composite test case remove unuseful var add composite op blacklist
cyber-pioneer
force-pushed
the
composite_debug
branch
from
6e2d5b0
to
ef1f8a1
Compare
cyber-pioneer
changed the title
PR types
New features
PR changes
Others
Describe
Add batch_norm composite rule and support batch_norm replaced by prime ops.
Composite rule
mean = reduce_sum(x) / nhw
variance = reduce_sum(x * x) / nhw - mean * mean
std_variance_inv = rsqrt(variance + epsilon), shape = [c]
y = scale * (x - mean) * std_variance_inv + bias, shape = [n, c, h, w]
moving_mean = moving_mean * momentum + (1.0 - momentum) * mean, shape = [c]
moving_variance = moving_variance * momentum + (1.0 - momentum) * variance, shape = [c]