Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation - Official Pytorch Implementation

[Arxiv] [Video]

Evaluation code for Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation. Finally ported to pyTorch!

Recent Updates

2020.05.06: Added myBinder version for quick testing of the model

2020.04.30: Initial pyTorch release

What's in this release?

The original pix2vertex repo was composed of three parts

• A lua based network to perform the image to depth + correspondence maps trained on synthetic facial data
• A Matlab based non-rigid ICP scheme for converting the output maps to a full 3D Mesh
• A Matlab based shape-from-shading scheme for adding fine mesoscopic details

This repo currently contains our image-to-image network with weights and model ported from the lua version and a simple python postprocessing scheme.

• The released network was trained on a combination of synthetic images and unlabeled real images for some extra robustness :)

Installation

    $ pip install pix2vertex

    $ git clone https://github.com/matansel/pix2vertex.pytorch.git
    $ cd pix2vertex.pytorch
    $ python setup.py install

Usage

 import pix2vertex as p2v
 from imageio import imread
 image = imread(<some image file>)
 
 depth, cropped = p2v.reconstruct(image)
 
 p2v.vis_depth_interactive(depth)

Pretrained Model

Models can be downloaded from these links:

• pix2vertex model
• dlib landmark predictor, note that the dlib model has its own license.

or simply by running download.sh

TODOs

• Port Torch model to pyTorch
• Release an inference notebook (using K3D)
• Add requirements
• Pack as wheel?
• Ported to MyBinder
• Port the Shape-from-Shading method used in our matlab paper
• Write a short blog about the revised training scheme

Citation

If you use this code for your research, please cite our paper Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation:

@article{sela2017unrestricted,
  title={Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation},
  author={Sela, Matan and Richardson, Elad and Kimmel, Ron},
  journal={arxiv},
  year={2017}
}