This repository has been archived by the owner on Aug 29, 2023. It is now read-only.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Face upscaling implementation
app.py
Added an interface block that lists *.onnx files in the upscaler_models folder with radio buttons and a way to pass upscaler string to refacer.
refacer.py
reface function is modified so it creates a ESRGAN or GFPGAN objects if the upscaler string has corresponding values and makes self.upscale_en = True.
process_faces functions' code is modified so we can use face_swapper.get(frame, ... , paste_back=False) returning both extracted processed face and M affine matrix for scalar transform, that is then passed to the new paste_upscale.
paste_upscale function code is from INSwapper, heavily modified to remove unused code and fix jagged edge borders when the faces are partially on the edge of the frame.
gfpgan_onnx.py
Added a GFPGAN implementation based on xuanandsix's GFPGAN-onnxruntime-demo.
You can easily convert the models with the provided scripts. Converted GFPGANv1.4 model provides best identity compared to 1.3 and 1.2, which is crucial for such a small 128px face.
esrgan_onnx.py
Added a ESRGAN implementation based on my fork of Sg4Dylan's ESRGAN-ONNX.
In my tests the best quality and speed-wise model is converted 2x_DigitalFlim_SuperUltraCompact_nf24-nc8_289k_net_g, the original found in Model Database.
With that model the performance is almost the same as without upscaling.
It works with regular and compact ESRGAN models, it might even work with RealESRGAN, I haven't tested it.
how to use
Place downloaded *.onnx weights into upscaler_models folder, make sure GFPGAN models contain GFPGAN in the filename. Select model in the radio buttons interface block, None uses native INSwapper paste_back processing without upscale.
WARNING: big model weights warmup at first frames, so refacer kind of hangs for a moment. Then it works fine until you change models/restart.
notes
Tested on Windows with CPU/CUDA.
Keep in mind that only 128px face is upscaled, not the whole picture.
Both implementations are as simple as they get, they just convert a int8 numpy.array image to float32 tensor and process it (with 2 different normalizations).
ESRGAN-ONNX repo has a license conflict with Apache 2.0 and MIT being kind of incompatible, so the code in the PR written from scratch.
Personally I recommend using chaiNNer to convert ESRGAN models to *.onnx but it can be done with the code just using torch.
fp16 conversion don't have any performance advantages, at least on CUDA.
PS
Face masking still works so-so, big area around the face depending on an angle looks worse than the face itself and the rest of the image. I tried MODnet, but it has poor temporal consistency and sometimes misses face parts between frames.
If anyone can recommend any other matting/masking NN's that can be run on ONNX, I'll be grateful!