In surveillance, information security, and access control applications, face recognition and identification from a video sequence is an important problem. The problems facing video-based face recognition are: a) poor quality of video (especially outdoors) and the large changes in illumination and pose, (b) detecting and localizing a face, which is often very small, from the background clutter, which may vary from frame to frame. This paper presents a new technique to model and recognize human faces in video sequences. Each registered person is represented by a lowdimensional appearance manifold in the ambient image space. The complex nonlinear appearance manifold expressed as a collection of subsets (named pose manifolds), and the connectivity among them. Each pose manifold is approximated by an affine plane. To construct this representation, exemplars are sampled from videos, and these exemplars are clustered with a K-means algorithm; each cluster is represented as a plane computed through principal component analysis (PCA). The connectivity between the pose manifolds encodes the transition probability between images in each of the pose manifold and is learned from a training video sequences. A maximum a posteriori formulation is presented for face recognition in test video sequences by integrating the likelihood that the input image comes from a particular pose manifold and the transition probability to this pose manifold from the previous frame. To recognize faces with partial occlusion, we introduce a weight mask into the process. Extensive experiments demonstrate that the proposed algorithm outperforms existing frame-based face recognition methods with temporal voting schemes.

1. B. Knight and A. Johnston. The role of movement in face recognition. Visual Cognition, 4:265 274, 1997.-
2. L. Gu, S.Z. Li, and H.J. Zhang. Learning probabilistic distribution model for multi-view face detection. In Proc. of IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, 2001.
3. Y.M. Li, S.G. Gong, and H. Liddell. Modeling faces dynamically across views and over time. In Proc. of International Conf. on Computer Vision, pages I: 554–559, 2001.
4. A. Azarbayejani, T. Starner, B. Horowitz, and A.P. Pentland. Visually controlled graphics. IEEE Trans. on Pattern Analysis and Machine Intelligence, 15:602–605, June 1993.
5. D. Terzopoulos and K. Waters. Analysis and synthesis of facial image sequences using physical and anatomical models. IEEE Trans. on Pattern Analysis and Machine Intelligence, 15:569– 579, June 1993.
6. A.L. Yuille, D.S. Cohen, and P.W. Hallinan. Feature extraction from faces using deformable templates. International Journal of Computer Vision, 8:99–111, August 1992
7. T.S. Jebara and A.P. Pentland. Parameterized structure from motion for 3d adaptive feedback tracking of faces. In Proc. of IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, pages 144–150, 1997.
8. C. Bregler, A. Hertzmann, and H. Biermann. Recovering nonrigid 3D shape from image streams. InProc. of IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, pages II:690– 696, 2000.
9. S. Romdhani, V. Blanz, and T. Vetter. Face identification by fitting a 3d morphable model using linearshape and texture error functions. In Proc. of European Conference on Computer Vision, page IV: 3 ff, 2002.
10. S. Gong, S. McKenna, and A. Psarrou. Dynamic Vision: From Images to Face Recognition. Singapore: World Scientific, 2000
11. H. Wechsler, V. Kakkad, J. Huang, S. Gutta, and V. Chen. Automatic video-based person authentication using the rbf network. In Proc. of Intl. Conf. on Audio, Video and Biometric Person Authentication, pages 85–92, 1997.
12. S. Zhou, V. Krueger, and R. Chellappa. Probabilistic recognition of human faces from video. Computer Vision and Image Understanding, 91:214–245, July-August 2003.
13. K.C. Lee, J. Ho, M.H. Yang, and D.J. Kriegman. Video-based face recognition using probabilistic appearance manifolds. In Proc. of IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, pages I: 313–320, 2003

Face Recognition Probabilistic Appearance Manifolds