Non-linear SfM

Category: Uncategorized / Tags: no tag / Add Comment

Non-linear optimization for non-rigid structure from motion recovery

We use bundle adjustment to estimate the structure, motion and deformation of our model by minimizing a meaningful geometric cost function based on the BHB [1] non-rigid model:

\arg \min_{R_i,l_{ik},S_k} \sum_{i,j} \left\| x_{ij} - \hat{x}_{ij} \right\|^2 = \left\| x_{ij} - R_i \left( \sum_k l_{ik} S_{kj} \right) \right\|^2

The initialization of the minimization method is given by the factorization of the measurement matrix W with an approach essentially similar to the Brand method (See [2] for an accurate description and this page for a general introduction on factorization)

A video (avi, 10.3MB) shows the subject face performing rigid motion and a set of face deformations. The video presents two views (front and side) of the reconstructed face, the image point tracks used to create the measurement matrix and the three axes representing the object relative orientation.
Parameters Plot Nobundle The following plots represent the parameters (configuration weights and rotation angles) extracted with the factorization method used as initialisation for the non-linear optimization step. Note the extremely noisy behavior of the parameters.
Parameters Plot bundle The parameters behavior after the bundle adjustment optimization are shown in the following figure. The parameters values reflects the real deformation and motion components of the non-rigid object.


[1] C. Bregler, A. Hertzmann, H.Biermann. Recovering Non-Rigid 3D Shape from Image Streams. IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), June 2000.

[2] M. E. Brand. Morphable 3D Models from Video. IEEE Computer Vision and Pattern Recognition (CVPR), December 2001.

References on non-rigid factorization:

  • A. Del Bue, F. Smeraldi, and L. Agapito, "Non-rigid structure from motion using ranklet–based tracking and non-linear optimization," Image and Vision Computing, vol. 25, iss. 3, pp. 297-310, 2007.
    @Article{DelBue:etal:IVC2007,
      author = {A. {Del Bue} and F. Smeraldi and L. Agapito},
      title = {Non-rigid structure from motion using ranklet--based tracking and non-linear optimization},
      journal = {Image and Vision Computing},
      year = {2007},
      volume = {25},
      number = {3},
      pages = {297--310},
      month = {March}
    }
  • M. Paladini, A. Del Bue, S. M. s, M. Dodig, J. Xavier, and L. Agapito, "Factorization for Non-Rigid and Articulated Structure using Metric Projections," in Proc. IEEE Conference on Computer Vision and Pattern Recognition, Miami, Florida, 2009, pp. 2898-2905.
    @conference{paladini:etal:2009, title={{Factorization for Non-Rigid and Articulated Structure using Metric Projections}},
      author={M. Paladini and A. {Del Bue} and M. Sto{\v{s}}ic and M. Dodig and J. Xavier and L. Agapito},
      year={2009},
      pages={2898--2905},
      booktitle={Proc. IEEE Conference on Computer Vision and Pattern Recognition, Miami, Florida}
    }

Google Scholar

Comments are closed.