How Dandere2x Works
This is targeted mostly towards a more general audience, so I'm simplifying things a lot.
We'll refer to this as 'the video', which is comprised of two frames. This was taken out of 'Kimi no Na Wa', a movie that pioneered me into the anime genre.
Here's some stuff you can identify looking at video.mkv
- The waiter's body moves and so does the customer's body.
- The rest of the scene stays static.
Imagine you were tasked with drawing the first frame in 'video.mkv'. This frame could be previewed here
Assume it takes you 5 seconds to draw the frame by hand (we're just making up numbers for intuition). In a sense, waifu2x has to 'redraw' each image you give it. So when you think 'upscaling an image', think that a computer has to manually re-draw the image by hand.
For naming sakes, we call frame1's upscaling drawing_1
Now assume you had to draw frame2.
Recall from Statement 1, that most of the scene is the same between the two images. Would it be necessary to redraw the entire frame? Could we take drawing_1, and only draw over the parts that were different? Could we produce drawing_2 while using part of our drawing in drawing_1? Yes, we can.
This is the core motivation of Dandere2x.
Let's assume that drawing_1 has already been produced from frame_1. To re-iterate, drawing_1 is the upscaled version of frame_1.
Because of Statement 1, it's possible to identify the parts of frame_2 that stay the same in frame_1.
Convince yourself that the black parts of the image are the parts of the waiter's body and the customer's body that moves.
We can call this image the interpolated_frame_2. The word interpolated means 'insert (something of a different nature) into something else.', so in a sense, we're inserting parts of frame_1 into frame_2.
If it wasn't for the black parts, interpolated_frame_2 would be identical to frame_2. All we need to do is look at which parts of frame_2 are missing from interpolated_frame_2, and draw those, then stitch them in.
I've taken the liberty of putting all the missing black boxes into a single image.
interpolated_frame_2_missing
This image contains all the missing black parts! This image contains less stuff to draw than drawing the entire frame_2.
Since we've already drawn frame_1 into drawing_1, and we know what parts are missing in interpolated_frame_2, and we put all the missing blocks in interpolated_frame_2 into its own image, we can draw interpolated_frame_2_missing, and just stitch those drawings into interpolated_frame_2.
A visual intuition for this is:
(we make up numbers to give intuition)
Assume that it takes 5 seconds to draw frame_1 into drawing_1.
With this assumption, it would also take 5 seconds to draw frame_2 into drawing_2.
However, if we were to just use interpolated_frame_2 and only draw interpolated_frame_2_missing, we can assume that it takes 1 second to draw interpolated_drawing_2_missing
Drawing frame1 and frame2 separately
5 seconds + 5 seconds = 10 seconds.
Drawing frame1 and using interpolated_frame_2 and drawing interpolated_drawing_2_missing
5 seconds + 1 seconds = 6 seconds
We can see that using interpolated_frame_2 is much faster.