Make image processors more general

[NielsRogge]

What does this PR do?

This PR undoes #26965 and instead makes image processors more general, by removing the hardcoded "shortest_side" arguments, allowing to pass the following to image processors:

size = {"height": ..., "width": ...}
size = {"shortest_edge": ...}

Ideally it should also allow for the following (cc @amyeroberts)

size = {"longest_edge": ...}
size = {"shortest_edge": ..., "longest_edge": ...}

Image processors should not be limited to only {"shortest_edge": ...} for instance, as CLIPImageProcessor is at the moment. The size argument is now always a dictionary containing one of these 4 possibilities; hence it would be great to support them all.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[amyeroberts]

Thanks for opening this PR!

Ideally it should also allow for the following (cc @amyeroberts)

size = {"longest_edge": ...}
size = {"shortest_edge": ..., "longest_edge": ...}

As discussed on slack, enabling all image processors to accept {"height": h, "width": w} is a change I'm very much pro. Enabling all to accept {"shortest_edge": x, "longest_edge": y} isn't something I think we should as this behaviour isn't well-defined and varies between models.

[NielsRogge]

cc @ydshieh this is failing for KOSMOS-2 - it would be great to revert that change for KOSMOS-2 (i.e. remove the "use_square_size" attribute) since this is introduced for legacy behaviour of size (which used to be an integer and is now a dictionary).

[ArthurZucker]

It did but it's also very convenient not to have to use copied from when the only difference was this, so reverting IMO is not the best solution. Let's rather adapt / make it compatible

[ydshieh]

Reverting #26965 means we well have to add Kosmos2ImageProcessor (while currently it is CLIPImageProcessor with use_square_size=True). Probably it will go without problem, but still kind of breaking changes.

[NielsRogge]

@ydshieh would it also be possible to just set size to {"height": ..., "width" ...}? With this PR, image processors are made more general, making sure square sizes are supported

[ydshieh]

Hi, could you give more details on

set size to {"height": ..., "width" ...}

(where should I set this, for example.)

Do you mean on the config file?

[NielsRogge]

Yes I meant in the preprocessor_config.json. Given that the model is only one month old, we could still update them. Alternatively, I've added a backwards compatibility in this PR, only for CLIPImageProcessor. Let me know what you think works best.

[ydshieh]

We can keep your backwards compatibility code, and I could try to open Hub PRs to update them.
Let's also hear what the core maintainers say of course.

[ArthurZucker]

Looks like a better and more general solution so in favor, I'll let @amyeroberts review when she comes back unless it's urgent! 😉

[NielsRogge]

It's required to merge #27718

[amyeroberts]

Thanks for adding!

The test file will need to be removed before merging - otherwise looks good!

[ydshieh]

Merge now as full (slow) CI on 4 models touched by this PR pass

[ArthurZucker]

cc @younesbelkada we have a regression on this