feat(losses): add hinge_embedding_loss to ivy experimental API.

[akshatvishu]

PR Description

The test are failing locally due to error while using @with_supported_device_and_dtypes decorator.
For more info regarding the same, please read this discussion at ivy discord.

Update(19 Oct 2023): The above issue is resolved(ref:link)
and the test are also passing locally for all backend but had to turn test_gradients=False otherwise tests fail for JAX and paddle backends.

Related Issue

Closes #26974

Checklist

• Did you add a function?
• Did you add the tests?
• Did you run your tests and are your tests passing?
• Did pre-commit not fail on any check?
• Did you follow the steps we provided?

[github-actions]

Thank you for this PR, here is the CI results:

---

This pull request does not result in any additional test failures. Congratulations!

[akshatvishu]

Another key thing to note is that although I have chosen ground_truth_backend=torch, there is a subtle difference between the implementation of the hinge embedding loss between PyTorch and PaddlePaddle native API's and I 've gone with the PaddlePaddle approach within my composition for Jax, NumPy and Tensorflow backends for IVY experimental API due to paddle one being much faithful the mathematical definition of Hinge_Loss

Pytorch:

def hinge_embedding_loss(self, target, margin, size_average):
    zeros = at.zeros_like(self)
    margin_clamp = (margin - self).clamp_min_(0)
    output_margin = at.where(target != 1, margin_clamp, zeros)
    output_self = at.where(target != -1, self, zeros)
    output = (output_margin + output_self).sum()

Paddle:

def hinge_embedding_loss(input, label, margin=1.0, reduction='mean', name=None):
    zero_ = paddle.zeros([1], dtype=input.dtype)
    loss = paddle.where(label == 1.0, input, zero_) + paddle.where(
        label == -1.0, paddle.nn.functional.relu(margin - input), zero_
    )

In the PyTorch implementation, if the target is not 1, the input value is clamped to a minimum of 0 after subtracting from the margin. If the target is not -1, the input value is directly taken.

In the PaddlePaddle implementation, for the target/label of 1, the input value is taken directly. For the target/label of -1, the rectified linear unit (ReLU) function is applied to the difference of the margin and input.

However, both implementations do not handle cases where the label/target is neither -1 nor 1. In such scenarios, depending on the input value, the loss might behave differently between the two implementations.

Reference:

• https://github.com/PaddlePaddle/Paddle/blob/release/2.5/python/paddle/nn/functional/loss.py#L3333
• https://github.com/cornell-brg/hb-pytorch/blob/084cf4d483e4adc2451d6875eb7cafd91d7ea47b/aten/src/ATen/native/Loss.cpp#L53

[akshatvishu]

Hi @MahmoudAshraf97, can you please take a look at this once ?

[Sarvesh-Kesharwani]

Hi @akshatvishu
please resolve the merge conflicts

[Sarvesh-Kesharwani]

LGTM!