Builtin cosine_similarity

[rwightman]

Hello, curious if there is any reason why the projection function uses self defined cosine sim vs something like below

    def _projection(self, p, grad, perturb, delta, wd_ratio, eps):
        wd = 1
        expand_size = [-1] + [1] * (len(p.shape) - 1)
        for view_func in [self._channel_view, self._layer_view]:
            g_view = view_func(grad)
            p_view = view_func(p.data)

            cosine_sim = F.cosine_similarity(g_view, p_view, dim=1, eps=eps).abs_()

            if cosine_sim.max() < delta / math.sqrt(p_view.size(1)):
                p_n = p.data / p_view.norm(dim=1).add_(eps).view(expand_size)
                perturb -= p_n * view_func(p_n * perturb).sum(dim=1).view(expand_size)
                wd = wd_ratio

                return perturb, wd

        return perturb, wd

[rwightman]

Also, curious why your cosine similarity has an absolute value, I scanned the paper but it wasn't immediately obvious...

[bhheo]

Hi

There is no special reason to use self defined cosine sim
I just couldn't think of PyTorch cosine similarity.
I will replace it in the next code release.

I missed the absolute value operation in the paper and it is a mistake.
In if statement of eq (11), we want to check whether the cosine similarity is close to zero or not.
So, it should be

I will change it in the next ArXiv version

Thanks for the valuable advices.

[bhheo]

Cosine similarity is changed to built-in operation in #7.
Thanks for your contribution.

[SanghyukChun]

@rwightman
The fixed version is just released as adamp==0.3.0(https://github.com/clovaai/AdamP/releases/tag/v0.3.0).
Thanks for your contribution again!