[WIP] Implement SSL-EY

[jameschapman19]

No description provided.

[guarin]

Oh wow, thanks for the PR! This looks interesting, will have a look ASAP.

[guarin]

Just to make sure, is this already ready for review?

[jameschapman19]

Hi,

Still WIP but close and will let you know.

Need to run the test suite locally. Will also run all the benchmarks I am able to (I have written the scripts in the right format though). I put a PR in as I thought it might run the CI and help me fix things up before review.

In many ways the pitch of the original work is VICReg without any hyperparameters in the objective so it plugs in quite easily.

James

[guarin]

Awesome, let use know once it is ready :)

[guarin]

Btw. we can also run some benchmarks on our side if that helps :)

[codecov]

Codecov Report

Attention: Patch coverage is 91.52542% with 5 lines in your changes missing coverage. Please review.

Project coverage is 85.58%. Comparing base (534cdf4) to head (f88efd0).
Report is 110 commits behind head on master.

Files	Patch %	Lines
lightly/loss/ssley_loss.py	87.09%	4 Missing ⚠️
lightly/utils/dist.py	93.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1443      +/-   ##
==========================================
+ Coverage   85.53%   85.58%   +0.05%     
==========================================
  Files         135      137       +2     
  Lines        5655     5703      +48     
==========================================
+ Hits         4837     4881      +44     
- Misses        818      822       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[guarin]

The failing codecov upload is unrelated to the PR :)
Could you run make format to apply the autoformatting? See https://github.com/lightly-ai/lightly?tab=readme-ov-file#development for more info.

[guarin]

This import doesn't work as ssley_transform doesn't exist yet. I think it would make sense to add a ssley_transform.py module and create a SSLEYTransform which inherits from VICRegTransform. That way one doesn't have to remember that ssley uses the vicreg transform.

[guarin]

Fixed typed errors in this module because it is used in ssley_loss

[guarin]

@jameschapman19 I fixed merged conflicts and some typing issues. I also added a SSLEYProjectionHead changed the examples to use the SSLEYTransform. I'll now try training the model on imagenet and then try to get this PR merged :)