Rotate page

[Rob192]

This is part of the solutions discussed in #225 and address the point "Integrate the feature in the predictor while being disabled by default" of the "Page-level orientation" part.

It implements the function rotate_page and tracks the 'page_angles'

[codecov]

Codecov Report

Merging #488 (66fad67) into main (e14645a) will increase coverage by 0.17%.
The diff coverage is 96.29%.

@@            Coverage Diff             @@
##             main     #488      +/-   ##
==========================================
+ Coverage   96.15%   96.32%   +0.17%     
==========================================
  Files         114      117       +3     
  Lines        4447     4523      +76     
==========================================
+ Hits         4276     4357      +81     
+ Misses        171      166       -5     

Flag	Coverage Δ
unittests	96.32% <96.29%> (+0.17%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
doctr/utils/visualization.py	92.03% <0.00%> (ø)
doctr/utils/geometry.py	98.00% <96.66%> (-0.79%)	⬇️
doctr/models/_utils.py	95.34% <100.00%> (+1.30%)	⬆️
doctr/models/predictor/pytorch.py	100.00% <100.00%> (ø)
doctr/models/predictor/tensorflow.py	100.00% <100.00%> (ø)
doctr/utils/common_types.py	100.00% <100.00%> (ø)
...dels/detection/differentiable_binarization/base.py	89.44% <0.00%> (-2.49%)	⬇️
doctr/datasets/cord.py	97.43% <0.00%> (ø)
doctr/datasets/funsd.py	96.87% <0.00%> (ø)
doctr/datasets/sroie.py	94.87% <0.00%> (ø)
... and 17 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e14645a...66fad67. Read the comment docs.

[fg-mindee]

Thanks a lot to jump back into this 🙏

I added a few comments but for me, there is only one major part missing:

• with this PR, we rotate the page firsthand before any DL model forward
• the process is almost identical, but the localizations of words have to be remapped into the original pages (meaning if we detect that the page has a 30° orientation, we forward its straightened version, if we detect a straight bounding box in this, we have to reproject it correctly so that the localization refers to the original page)

Let me know what you think :)

[fg-mindee]

@Rob192 also, if you don't minde merging the main branch, there seems to be some conflicts 😅

[Rob192]

@Rob192 also, if you don't minde merging the main branch, there seems to be some conflicts 😅
Yes ! you guys are mooving too fast for me 😮

[Rob192]

Hello,

Thanks for the review ! My objective is now to use geometry.rotate_image to rotate the documents before going into the predictor and then use the same function to rotate every box, compare to the center of the document previously used.

However, I am feeling a bit confused regarding the expand functionnality you added in several functions of Doctr. I hoped you could help me clarify. I think you goal was to do the same for every function, i.e. given a image of shape (10,20), the rotation by an angle of 90° with expand=True would give a result image of shape (20,10). This is not the case for all the function in utils.geometry :

import numpy as np
import matplotlib.pyplot as plt

import torch
import tensorflow as tf

from doctr.utils import geometry
from doctr.transforms.functional.tensorflow import rotate as rotate_tf
from doctr.transforms.functional.pytorch import rotate as rotate_to

_, axes = plt.subplots(3, 1)

img = np.ones((32, 64, 3), dtype=np.float32)
rotated = geometry.rotate_image(img, 45., expand=True)
print(rotated.shape)
axes[0].imshow(rotated)

input_t = torch.ones((3, 32, 64), dtype=torch.float32)
boxes = np.array([
    [15, 20, 35, 30]
])
r_img, r_boxes = rotate_to(input_t, boxes, angle=45., expand=True)
print(r_img.shape)
axes[1].imshow(r_img.permute((1,2,0)))

input_t = tf.ones((32, 64, 3), dtype=tf.float32)
boxes = np.array([
    [15, 20, 35, 30]
])
r_img, r_boxes = rotate_tf(input_t, boxes, angle=45., expand=True)
print(r_img.shape)
axes[2].imshow(r_img)

Do you think I could modify geometry.rotate_image so that expand=True has the same behaviour as the two other functions in doctr.transforms.functional ? More precisely I would add a expand_type parameter that could take values in ['crop', 'keep_original_shape', 'keep_destination_shape']

[fg-mindee]

Thanks for the edits! Regarding your question about rotate_image, it's still an ongoing implementation so we can add optional features in it. But to keep the PR short, perhaps this should be implemented in the predictor directly? (we can refactor in another PR once we get this working)

[Rob192]

Hello @fg-mindee !
Here are my modifications to remap the localizations of the words back into the original pages. Here are some details of the choices I did :

• I created a new rotate_image function inside doctr/models/_utils.py per your recommandation. I removed the last bit of the function to keep the size of the original image. I introduced a mask functionality to crop the padding at the end of the + then - rotation.
• I had to modify rotate_boxes to follow the expand that is done in rotate_image. The calculation for the rotation of the boxes was not right and I fixed that. There is also a mask functionality in rotate_boxes
  I hope this makes sense to you guys

[fg-mindee]

Thanks again Rob!
I think there are still a few edges cases we are not covering but this is taking a very solid shape!

[fg-mindee]

Still have a few comments, but the major one being about the reprojection of rotated bounding boxes

[fg-mindee]

There is one merge conflict to resolve, minor tweaks in the unittest and we can merge!
Sorry about this 🙃

[Rob192]

OK @fg-mindee ! Now I got it for the testing !

[fg-mindee]

Looks good to me, thanks a lot @Rob192 ! @charlesmindee would you mind taking a look at it as well before we merge? 🙏

[charlesmindee]

Thanks for the PR !

[fg-mindee]

Sorry Robin, a minor thing I missed in my last review to adjust 😅
That shouldn't change much but especially on this feature it's important to have a good understanding of each argument influence on the final output!

[fg-mindee]

There seems to be a failing test: you might have to pass preserve_origin_shape=True where I added the comment 👍

[fg-mindee]

Making that last argument non-optional and we're good!

[fg-mindee]

All good 🙌 Thanks a lot!

[Rob192]

Yeah ! Thanks a lot for your support in reviewing this !