Wrapper functions for NNlib

[matsueushi]

Related to #614. I found some of activation functions recently added to NNlib are imcompatible with GPU, so I defined @cufunc wrappers for them. I also updated the existing definitions to make them consistent with NNlib (https://github.com/FluxML/NNlib.jl/blob/master/src/activation.jl).

I skipped rrelu because rand is used within its definition
https://github.com/FluxML/NNlib.jl/blob/ac5101b2f4b4afc8cc01968e5c8dadaa0eaa862a/src/activation.jl#L92-L95 and couldn't figure out how it can be handled.

[maleadt]

Thanks!
bors try

[bors]

try

Build succeeded

• ci/gitlab/trying

[CarloLucibello]

Are all those definitions needed here? e.g. swish(x) = x * σ(x) won't work out of the box? (assuming it is defined like that in NNlib. If not, it should)

[matsueushi]

Yes, if you remove @cufunc of swish, you will see warnings

┌ Warning: calls to Base intrinsics might be GPU incompatible
│   exception =
│    You called exp(x::T) where T<:Union{Float32, Float64} in Base.Math at special/exp.jl:75, maybe you intended to call exp(x::Float64) in CUDAnative at /root/.julia/packages/CUDAnative/hwB4d/src/device/cuda/math.jl:100 instead?
│    Stacktrace:
│     [1] exp at special/exp.jl:75
│     [2] #28 at /root/.julia/packages/GPUArrays/GLRnH/src/host/broadcast.jl:64
└ @ CUDAnative ~/.julia/packages/CUDAnative/hwB4d/src/compiler/irgen.jl:113
┌ Warning: calls to Base intrinsics might be GPU incompatible
│   exception =
│    You called exp(x::T) where T<:Union{Float32, Float64} in Base.Math at special/exp.jl:75, maybe you intended to call exp(x::Float64) in CUDAnative at /root/.julia/packages/CUDAnative/hwB4d/src/device/cuda/math.jl:100 instead?
│    Stacktrace:
│     [1] exp at special/exp.jl:75
│     [2] #28 at /root/.julia/packages/GPUArrays/GLRnH/src/host/broadcast.jl:64
└ @ CUDAnative ~/.julia/packages/CUDAnative/hwB4d/src/compiler/irgen.jl:113

I added all the activation functions in NNlib to test first and checked GPU compatibility by running tests with a blank "nnlib.jl", so unnecessary wrappers are not included. I guess if exp or tanh is used in a definition, we need to define its wrapper.

[maleadt]

These are just warning, if the existing Base implementations work we can whitelist them in CUDAnative.jl instead of adding an implementation here. https://github.com/JuliaGPU/CUDAnative.jl/blob/75313e555ef7383f5da4cd71840eed92f76ccc9a/src/compiler/irgen.jl#L48

[matsueushi]

Then I will remove wrappers for σ, elu, swith, selu and celu. The implementation of exp seems to work fine, but if log1p or tanh is used, a wrapper is still needed.

[maleadt]

bors r+

[maleadt]

@matsueushi so only exp needs whitelisting in CUDAnative?

[bors]

Build failed

• ci/gitlab/staging

[matsueushi]

@maleadt Yes, other functions showed errors, not warnings.