Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PaddlePaddle hackathon] + ADD CELU #36088

Merged
merged 7 commits into from
Oct 13, 2021

Conversation

JunnYu
Copy link
Member

@JunnYu JunnYu commented Sep 26, 2021

PR types

New features

PR changes

OPs

Describe

@paddle-bot-old
Copy link

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@JunnYu JunnYu changed the title Add celu use c++ [PaddlePaddle hackathon] + ADD CELU Sep 26, 2021
@JunnYu
Copy link
Member Author

JunnYu commented Sep 28, 2021

  • PR-CI-Coverage中python覆盖100%,c++代码覆盖有点低,好多REGISTER_OP都没覆盖到,不清楚为什么- -。
  • PR-CI-CPU-Py2 没有approve。

@@ -1996,6 +2080,33 @@ class ELUDoubleGradKernel
}
};

template <typename DeviceContext, typename Functor>
class CELUDoubleGradKernel
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果代码实现与ELU相同,是否有必要重复代码,是否可以直接沿用ELU的代码?

Copy link
Member Author

@JunnYu JunnYu Oct 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

代码实现大致与ELU相同,但是存在些许不同:
(1)前向计算多了个alpha
(2)反向传播计算梯度时候,由于多了alpha因此,计算的梯度方式有些许不同之处。

   CELU的
    // dx = dout , if alpha > 0 and x > 0
    // dx = dout * (x/alpha).exp(), if alpha > 0 and x <= 0
    // dx = dout , if alpha < 0 and x > 0
    // dx = dout * (x/alpha).exp(), if alpha < 0 and x <=0

   ELU的
   // dx = dout, if alpha > 0 and x > 0
   // dx = dout * alpha * x.exp(), if alpha > 0 and x <= 0
   // dx = dout * (1 + alpha * x.exp()), if alpha <= 0 and x > 0
   // dx = 0, if alpha <= 0 and x <=0

python/paddle/nn/functional/activation.py Outdated Show resolved Hide resolved
python/paddle/nn/__init__.py Show resolved Hide resolved
python/paddle/nn/layer/activation.py Show resolved Hide resolved
python/paddle/nn/layer/activation.py Show resolved Hide resolved
python/paddle/nn/layer/activation.py Show resolved Hide resolved
python/paddle/nn/layer/activation.py Show resolved Hide resolved
python/paddle/nn/layer/activation.py Show resolved Hide resolved
@JunnYu
Copy link
Member Author

JunnYu commented Oct 11, 2021

@zhiboniu 同样的,这个我之前也想过使用纯Python实现,然后发现paddle的有些算子不支持float16,如paddle.where,于是我才想到用c++的来实现这个算子。#36076

Copy link
Contributor

@zhiboniu zhiboniu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@TCChenlong
Copy link
Contributor

还需要在 docs 中补充对应的中文文档~

Copy link
Contributor

@XiaoguangHu01 XiaoguangHu01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@TCChenlong TCChenlong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jeff41404 jeff41404 merged commit d7064f0 into PaddlePaddle:develop Oct 13, 2021
@JunnYu JunnYu deleted the add_celu_use_c++ branch October 20, 2021 06:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants