Plenoptic Face Reconstruction
Description
Plenoptic cameras
[Ng05,Perwass12] are capable of imaging the scene from
different perspectives. The information of the different perspectives is stored
on a single image sensor which allows to acquire dynamic scenes and perform 3D
reconstruction easily [Monteiro16]. The acquisition on a single sensor has the
advantages described previously but it also limits the spacing between the
viewpoints of the different perspectives which limits the field of view and the
3D reconstruction. Thus, in this work we plan to work with a stereo of
plenoptic cameras.
3D face models are
widely used for several purposes, such as biometric systems, face verification,
facial expression recognition, 3D visualization, and so on. However, the
reconstructed models of the face generated from the data of the optical setups
used are quite noisy due to the lack of texture and the thin structures present
in the face. In this work, we want to improve the reconstruction using a stereo
of plenoptic cameras complemented with view interpolation [Vagharshakyan17].
Additionally, one wants to use symmetric priors and methods like the one
presented in [Lourenço18] to improve the thin and textureless
regions reconstruction.
Objectives
- Study plenoptic
camera
- Acquire real
dataset for face reconstruction
- Perform view
interpolation (https://videos-rennes.inria.fr/video/r1S20C_Lr 27:42 a 28:06)
and 3D face reconstruction (https://videos-rennes.inria.fr/video/HJ5KVkrUH
43:20 a 43:47)
References:
[Ng05] Ng, Ren.
Digital light field photography. Stanford, CA: stanford
university, 2006.
[Perwass12] Perwass, C., & Wietzke, L.
(2012, February). Single lens 3D-camera with extended depth-of-field. In Human
Vision and Electronic Imaging XVII (Vol. 8291, p. 829108). International
Society for Optics and Photonics.
[Monteiro16]
Monteiro, Nuno Barroso, Joao Pedro Barreto, and José Gaspar. "Dense lightfield disparity estimation using total variation
regularization." International Conference on Image Analysis and
Recognition. Springer, Cham, 2016.
[Lourenço18]
Lourenco, Rui, et al. "Silhouette enhancement in light field disparity
estimation using the structure tensor." 2018 25th IEEE International
Conference on Image Processing (ICIP). IEEE, 2018.
[Vagharshakyan17] Vagharshakyan, Suren, Robert Bregovic,
and Atanas Gotchev. "Light field reconstruction
using Shearlet transform." IEEE transactions on
pattern analysis and machine intelligence 40.1 (2017): 133-147.
More information:
Conventional cameras give information about the
total amount of light that reaches each position of the sensor. During image
formation, there is some information of the light field that is lost, for
example: (i) the direction of the ray and; (ii) the
contribution of each ray to the total amount of light captured in an image.
Plenoptic cameras like Lytro
and Raytrix preserve this information. The additional
information allows retrieving 3D information from the scene using a single
image. Reconstruction methods for plenoptic images are based on identifying
line features on epipolar plane images or on shearing
the lightfield according to some customized metric
for feature matching. These algorithms are computationally demanding and are
still not capable of giving accurate depth estimations for low-textured and
occluded regions.
On the other hand, depth sensors are active
sensors that project a structured light pattern into the scene. This allows
obtaining depth measurements regardless of the texture of the scene.
Nonetheless, standard depth sensors give depth estimates with lower accuracy in
the near range (less than 1.5 m).
References:
[Dansereau13] Dansereau, Donald G., Oscar
Pizarro, and Stefan B. Williams. "Decoding, calibration and rectification
for lenselet-based plenoptic cameras." Computer
Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on. IEEE, 2013.
[Herrera12] Herrera, Daniel, Juho Kannala, and Janne Heikkilä. "Joint depth
and color camera calibration with distortion correction." IEEE
Transactions on Pattern Analysis and Machine Intelligence 34.10 (2012):
2058-2064.
Place for conducting
the work-proposal:
ISR / IST
More MSc
dissertation proposals on Computer and Robot Vision in:
http://users.isr.tecnico.ulisboa.pt/~jag/msc/msc_2018_2019.html