What can i solve the face box?

[skeletonNN]

Great Job! When i infer the video, it can be found that there is a box for face. Can you solve it? Thank you!

[Inferencer]

I can solve it, working on something that stops the box jittering first this will help when I code the box removal.
For a quick fix while I'm working on it you can manually mask using a video editing tool

(note: It's good to watch the first few mins of the davinci tut even if you don't have it as it will explain difference in length between the input & output although I will later automate that)

Davinci Tut
or
After Effects Tut

[skeletonNN]

Can I use the python code?

…

---- Replied Message ---- | From | ***@***.***> | | Date | 05/08/2024 01:16 | | To | Inferencer/LipSick ***@***.***> | | Cc | skeleton ***@***.***>, Author ***@***.***> | | Subject | Re: [Inferencer/LipSick] What can i solve the face box? (Issue #8) | I can solve it, working on something that stops the box jittering first this will help when I code the box removal. For a quick fix while I'm working on it you can manually mask using a video editing tool (note: It's good to watch the first few mins of the davinci tut even if you don't have it as it will explain difference in length between the input & output although I will later automate that) Davinci Tut or After Effects Tut — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[Inferencer]

when it's working I'll upload it, could be end of this week or it could be end of next week

[skeletonNN]

Thank You!

[skeletonNN]

One more thing! I find the color is different from the original pic!

I think the problem is due to the Ditnet. Can you some suggestions?

[Inferencer]

I always assumed this was due to the pre-trained model being trained on low quality dark lighting datasets but I guess it could be possible to attempt to match the lighting which could affect the color I will certainly do some experiments to find out

[Inferencer]

Crop_radius_example.1.mp4

The custom crop radius has now been uploaded in the windows/ main branch so after you calculate the crop_radius in the crop radius tab and set the recommended value in the main tab you will get less box flickering so that's stage one done which will help with box removal.

I need to make this more user friendly though

[skeletonNN]

I always assumed this was due to the pre-trained model being trained on low quality dark lighting datasets but I guess it could be possible to attempt to match the lighting which could affect the color I will certainly do some experiments to find out

I understand it should conform to the lighting color information in the reference image, but it seems that this information has not been learned.

[Inferencer]

I always assumed this was due to the pre-trained model being trained on low quality dark lighting datasets but I guess it could be possible to attempt to match the lighting which could affect the color I will certainly do some experiments to find out

I understand it should conform to the lighting color information in the reference image, but it seems that this information has not been learned.

Another theory is as its currently using 5 reference frames.. (I will change that soon to use an optional single frame up to 5 frames)
the frames are being layered on top of each-other at the learning stage which is decreasing quality, I have found when using a single frame as reference although the lip-sync is less slightly expressive that teeth & skin quality increase dramatically this could possibly be affecting color too.

This is an example of bad skin and teeth then after there is an example of using a single reference frame for all frames, although the target was doing weird chin movements in both videos I was more focused on the teeth at the time

BadSkin.mp4

Teeth.mp4

As you can see the skin color matching improves, for clarification this was a single reference frame and the reference frame was up-scaled with GFPGAN so my current theory is its using too many reference frames and also teeth related, based on past experiments when the teeth are dim so is the skin, the worse quality the teeth the worse quality & color the skin.

[skeletonNN]

I always assumed this was due to the pre-trained model being trained on low quality dark lighting datasets but I guess it could be possible to attempt to match the lighting which could affect the color I will certainly do some experiments to find out

I understand it should conform to the lighting color information in the reference image, but it seems that this information has not been learned.

Another theory is as its currently using 5 reference frames.. (I will change that soon to use an optional single frame up to 5 frames) the frames are being layered on top of each-other at the learning stage which is decreasing quality, I have found when using a single frame as reference although the lip-sync is less slightly expressive that teeth & skin quality increase dramatically this could possibly be affecting color too.

This is an example of bad skin and teeth then after there is an example of using a single reference frame for all frames, although the target was doing weird chin movements in both videos I was more focused on the teeth at the time

BadSkin.mp4
Teeth.mp4
As you can see the skin color matching improves, for clarification this was a single reference frame and the reference frame was up-scaled with GFPGAN so my current theory is its using too many reference frames and also teeth related, based on past experiments when the teeth are dim so is the skin, the worse quality the teeth the worse quality & color the skin.

Excellent conclusion, I will try it.

[skeletonNN]

In some cases, I use silent audio to get them to shut up and the mouth can close completely. But the generated frame video is constantly shaking due to the presence of the frame.

default.mp4

[Inferencer]

I created a decent detector for the best crop radius but just found that the new crop radius is not correctly implemented into the code so it's defaulting to the normal shaking regardless of crop radius, will fix soon to prevent shaking.

[Inferencer]

Correct crop radius detection has now been implemented in the advanced accordion, I Believe even without using custom crop radius it defaults to most common crop radius this should now prevent box flickering so I can now move into testing box removal methods. Hope this helps