現今，人臉表情合成在人機互動扮演很重要的角色。然而，因為人臉表情合成牽涉很多因素，例如臉部特徵的非剛體移動，表情變化產生的皺紋和光線因素，導致合成上的困難。此外，文化因素及個人性格，使得即使是相同表情，也有很大的差異 (例如：有些人笑偏向張嘴，有些人偏向閉嘴)。這些因素使得人臉合成變成一個很大的挑戰。
當人臉表情變化時，人臉可以分成因表情轉變而產生變化的部位，及不產生變化的部位。這些部位能改善人臉表情合成的結果。我們提出利用 GentleBoost自動的分辨這兩種部位，並建立新穎的人臉合成系統來合成包含細部紋理和正確的人臉結構的表情影像。系統架構上分成兩個階段，全域階段和區域階段。首先，我們利用表情變化的部位來衡量不同表情的相似度，並修改 K-mean 分群演算法來對人臉表情做分群。在全域階段，結合因素分析 (Factor Analysis) 和組合空間 (combined space) 的概念，來分析正常表情和其它表情的關係。然後，其它表情的人臉結構可以被合成出來。在區域階段，馬可夫隨機域 (Markov Random Fields) 將人臉結構納入考慮來合成表情變化部位的細部紋理。
實驗結果顯示我們提出的兩階段的合成系統比起其他作者的方法能合成更細部的紋理和更自然的表情。此外，我們能合成不曾出現在正常表情的虛擬紋理，例如牙齒。

Nowadays, facial expression synthesis plays an important role in human-computer interaction. However, facial expression involves many factors. For example, non-rigid motion of facial features, wrinkles due to skin deformation, and illumination. In addition, different cultures and personalities make the same expression have a large variation (e.g., some people smile with their open mouth, but some with closed mouth). These factors make facial synthesis become a challenge task.
Human face in generally is composed of two types of patches. One is visually invariant when changing facial expression, while the other one is variant. In this study, two types of patches are defined as expression-invariant patches and expression-variant patches, respectively. Facial expression synthesis system is dedicated to expression-variant patches, which can improve synthesis results. GentleBoost algorithm is proposed to automatically classify all patches as two types, which is the basis of our novel synthesis system. Our system is a two-step approach, global model step and local model step, which synthesize facial expression images with local detailed texture by constraining a global facial structure. First, facial expression types are clustered by a modified K-mean algorithm that the similarity measurement is based on the expression-variant patches. At global model step, factor analysis is applied for each facial expression type with the concept of combined space in order to find models the relationship between neutral expression and a specific facial expression. Then, global facial structure with a specific expression can be synthesized by the known relationship. At local model step, a Markov random field is used to synthesize local detailed textures, which are constrained by the global structure, in theses expression-variant patches.
Experimental results show that our system can synthesize more detailed texture and more natural expression compared with other methods by applying the proposed two-step approach being dedicated to expression-variant patches. In addition, our synthesized results can contain virtual texture which can’t emerge in neutral input images such as teeth.

[1] B. Abboud, and F. Davoine, “Appearance Factorization Based Facial Expression Recognition and Synthesis”, International Conference on Pattern Recognition, Vol. 4, pp. 163-166, 2004.
[2] A. B. Ashraf, S. Lucey, and T. Chen, “Learning patch correspondences for improved viewpoint invariant face recognition”, IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8, 2008.
[3] D. Beymer, A. Shashua, and T. Poggio, “Example Based Image Analysis and Synthesis”, AI Memo 1431, Massachusetts Institution of Technology, 1993.
[4] C. Bishop, “Pattern Recognition and Machine Learning”, Springer, 2006.
[5] X. Chai, S. Shan, X. Chen, and W. Gao. "Local Linear Regression (LLR) for Pose Invariant Face Recognition", International Conference on Automatic Face and Gesture Recognition, pp. 1716-1725, 2006.
[6] S. C. Chang, K. Aizawa, H. Harashima, and T. Takebe. “Analysis and Synthesis of Facial Image Sequences in Model-Based Image Coding”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 4, pp. 257 – 275, 1994.
[7] B. Choe, and H. S. Ko, “Analysis and Synthesis of Facial Expressions with Hand-Generated Muscle Actuation Basis”, ACM SIGGRAPH courses, No. 21, 2006.
[8] T. F. Cootes, G. J. Edwards, and C. J. Taylor, “Active Appearance Models”, IEEE Transaction on pattern analysis and machine intelligence, Vol. 23, No. 6, pp. 681-685, 2001.
[9] Z. Deng, U. Neumann, J. P. Lewis, T. Y. Kim, M. Bulut, and S. Narayanan, “Expressive Facial Animation Synthesis by Learning Speech Coarticulation and Expression Spaces”, IEEE Transactions on Visualization and Computer Graphics, Vol. 12, No. 6, pp. 1523-1534, 2006.
[10] J. Friedman, T. Hastie, and R. Tibshirani, “Additive Logistic Regression: A Statistical View of Boosting”, Annals of statistics, Vol. 28, No. 2, pp. 337–374, 2000.
[11] Z. Ghahramani, and G. E. Hinton, “The EM Algorithm for Mixtures of Factor Analyzers”, Technical Report CRG-TR-96-1, 1997
[12] L. Huang, and C. Su, "Facial Expression Synthesis Using Manifold Learning and Belief Propagation", SoftComput, Vol. 10, No. 12, pp. 1193-1200, 2006.
[13] T. Kanade, J. F. Cohn, and Y. Tian, “Comprehensive Database for Facial Expression Analysis”, International Conference on Automatic Face and Gesture Recognition, pp. 46-53, 2000.
[14] C. S. Lee, and A. Elgammal, “Nonlinear Shape and Appearance Models for Facial Expression Analysis and Synthesis”, International Conference on Pattern Recognition, 2006, Vol. 1, pp. 497-502, 2006.
[15] Z. Liu, Y. Shan, and Z. Zhang, “Expressive Expression Mapping with Ratio Images”, SIGGRAPH, pp. 271-276, 2001.
[16] U. Mohammed, S. J. D. Prince, and J. Kautz, "Visio-lization: Generating Novel Facial Images", SIGGRAPH, No. 57, 2009.
[17] J. Y. Noh, and U. Neumann, “Expression Cloning”, ACM SIGGRAPH courses, No. 22, 2006.
[18] F. I. Parke, “Computer Generated Animation of Faces”, Proceedings of the ACM annual conference, Vol. 1, pp. 451-457, 1972.
[19] F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D. H. Salesin, “Synthesizing Realistic Facial Expressions from Photographs”, SIGGRAPH, pp. 75-87, 1998.
[20] H. Pyun, Y. Kim, W. Chae, H. W. Kang, and S. Y. Shin, “An Example-Based Approach for Facial Expression Cloning”, Proceedings of the ACM SIGGRAPH, pp.167-176, 2003.
[21] K. R. Thorisson, “Gandalf: An Embodied Humanoid Capable of Real-Time Multimodal Dialogue with People”, ACM International Conference on Autonomous Agents, pp. 536-537, 1997.
[22] A. Torralba, K. P. Murphy, and W. T. Freeman. "Sharing Visual Features for Multiclass and Multiview Object Detection", IEEE Transaction on pattern analysis and machine intelligence, Vol 29, pp. 854- 896, 2007.
[23] T. Vetter, and T. Poggio, “Linear Object Classes and Image Synthesis from a Single Example Image”, IEEE Transaction on pattern analysis and machine intelligence, Vol. 19, No. 7, pp. 733-742, 1997.
[24] P. Viola, and M. Jones, “Robust Real-time Object Detection”, International Journal of Computer Vision, Vol. 57, No. 2, pp. 137–154, 2004.
[25] H. Wang, and N. Ahuja, “Facial Expression Decomposition”, IEEE International Conference on Computer Vision, Vol. 2, pp. 958-965, 2003.
[25] S. Wang, X. D. Gu, and H. Qin, “Automatic Non-rigid Registration of 3D Dynamic Data for Facial Expression”, IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8, 2008.
[27] P. Xie, Y. Chen, and J. Liu, “Realistic Facial Animation Synthesis and Transfer Based on Flexible Expression Ratio Image”, IEEE International Conference on Multimedia and Expo, pp. 1207-1210, 2007.
[28] L. Xiong, N. Zheng, Q. You, and J. Liu, “Facial Expression Sequence Synthesis Based on Shape and Texture Fusion Model”, IEEE International Conference on Image Processing, Vol. 4, pp. 473-476, 2007.
[29] P. Yang, Q. Liu, X. Cui, and D. N. Metaxas, “Facial Expression Recognition Using Encoded Dynamic Features”, IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8, 2008.
[30] P. Yang, Q. Liu, X. Cui, and D. N. Metaxas, “Boosting Coded Dynamic Features for Facial Action Units and Facial Expression Recognition”, IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-6, 2007.
[31] J. Yedidia, W. T. Freeman, and Y. Weiss, "Understanding Belief Propagation and its Generalizations", MITSUBISHI Electric Research Lab. TR-2001-22 , 2002.
[32] L. Zalewski, and S. Gong, “Synthesis and Recognition of Facial Expressions in Virtual 3D Views”, International Conference on Automatic Face and Gesture Recognition, pp. 493-498, 2004.
[33] Q. Zhang, Z. Liu, G. Quo, D. Terzopoulos, and H. Y. Shum, “Geometry-driven Photorealistic Facial Expression Synthesis”, IEEE Transactions on Visualization and Computer Graphics, Vol. 12, No. 1, pp. 48-60, 2006.