近年來隨著深度學習快速發展，Deepfake造假能力也越趨逼真，導致在現今網路多媒體蓬勃發展的環境下，其在社交工程中造成的影響甚钜，不僅濫用此偽造技術生成難以識別的虛假內容會造成難以估量的危害，驗證內容的真實性同樣是重要的課題，其背後牽涉的道德問題、隱私權問題，甚至是潛在的國安威脅皆不容忽視。為此相關研究人員希望找出Deepfake演算法的破綻，隨著Deepfake影片生成技術越趨成熟，相對應的偵測技術也不斷在改良。本研究以此為出發點，首先探討Deepfake影片的生成方法，還有相關的資料集、換臉軟體與檢測方法，接著考量到現行人臉辨識的資料集當中，無論是靜態或是動態影像幾乎都是正面的睜眼狀態，缺少閉眼狀態的特徵，而導致Deepfake換臉影片生成時，眼部狀態變化的過程將出現破綻，因此提出藉由偵測影片中眼部狀態的特徵，進行Deepfake影片的檢測，結合長短期記憶模型使用長期遞歸卷積神經網路的架構，來實現以時序為基準之眼部狀態變化的量化。最後再評估資料集所提供之特徵的影響並且與相關論文比較，驗證了本研究提出方法的可行性。

With the rapid development in deep learning in recent years, big data analytics and image recognition technologies are also widely used in different fields. Simultaneously, the fake video generated by deepfake has become more challenging to spot, resulting in incalculable harm. Moreover, the ability to verify the authenticity of the content is also essential. We cannot ignore the privacy issues and national security threats caused by deepfake, as well as we urgently need the ability to detect deepfake video. Researchers are looking for ways to spot flaws in current algorithms that are used to create fake videos.
On the other hand, deepfake detection methods are improving while deepfake videos are in flood. At this stage, the detection technologies of deepfake voice and images have become more and more mature. Our research takes this as a starting point and discusses the deepfake video generation method, the related datasets, face swap software, and detection methods.
Currently, most of the data in the face datasets are frontal eyes opened images. That means while generating deepfake video, lacking the eyes closed features is the main problem. Therefore, our research proposes a method to determine whether it is a deepfake video by detecting the features of the eye status changed in the video using deep learning to realize the quantification of eye status changes based on time sequences. Eventually, the impact of the dataset's features is evaluated and compared with related papers to verify the feasibility of the method proposed in our research.

[1] Dave Lee, “Deepfakes porn has serious consequences,” 2018. [Online]. Available: https://www.bbc.com/news/technology-42912529. [Accessed Nov. 21, 2019].
[2] Sarah Mervosh, “Distorted Videos of Nancy Pelosi Spread on Facebook and Twitter, Helped by Trump,” 2019. [Online]. Available: https://www.nytimes.com/2019/05/24/us/politics/pelosi-doctored-video.html. [Accessed Nov. 24, 2019].
[3] David Ruiz, “Deepfakes laws and proposals flood US,” MalwarebytesLABS, 2020. [Online]. Available: https://blog.malwarebytes.com/artificial-intelligence/2020/01/deepfakes-laws-and-proposals-flood-us/. [Accessed Apr. 23, 2020].
[4] Angela Chen, “Three threats posed by deepfakes that technology won’t solve,” 2019. [Online]. Available: https://www.technologyreview.com/2019/10/02/75400/deepfake-technology-detection-disinformation-harassment-revenge-porn-law/. [Accessed Dec. 02, 2019].
[5] Mousa Tayseer Jafar, Mohammad Ababneh, Mohammad Al-Zoube, Ammar Elhassan, “Forensics and Analysis of Deepfake Videos,” in 2020 11th International Conference on Information and Communication Systems (ICICS), Irbid, Jordan, Apr. 7-9, 2020.
[6] Jeremy Jordan, “Introduction to autoencoders,” 2018. [Online]. Available: https://www.jeremyjordan.me/autoencoders/. [Accessed Dec. 20, 2019].
[7] David Güera, Edward J. Delp, “Deepfake Video Detection Using Recurrent Neural Networks,” in The 15th IEEE International Conference on Advanced Video and Signal-based Surveillance (AVSS), Auckland, New Zealand, Nov. 27-30, 2018.
[8] shaoanlu, “faceswap-GAN,” 2020. [Online]. Available: https://github.com/shaoanlu/faceswap-GAN. [Accessed Mar. 15, 2020].
[9] Alec Radford, Luke Metz, Soumith Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks.,” in ICLR 2016, San Juan, Puerto Rico, May. 2-4, 2016.
[10] Nick Dufour, “Contributing Data to Deepfake Detection Research,” 2019. [Online]. Available: https://ai.googleblog.com/2019/09/contributing-data-to-deepfake-detection.html. [Accessed Jan. 03, 2020].
[11] Xin Geng, Zhi-Hua Zhou, Kate Smith-Miles, “Individual Stable Space: An Approach to Face Recognition Under Uncontrolled Conditions,” IEEE Transactions on Neural Networks, vol. 19, no. 8, pp. 1354 - 1368, 2008.
[12] Pranjal Ranjan, Sarvesh Patil, Faruk Kazi, “Improved Generalizability of Deep-Fakes Detection using Transfer Learning Based CNN Framework,” in 2020 3rd International Conference on Information and Computer Technologies (ICICT), San Jose, USA, May. 9-12, 2020.
[13] Shifeng Zhang, Xiangyu Zhu, Zhen Lei, Hailin Shi, Xiaobo Wang, Stan Z. Li, “S^3FD: Single Shot Scale-Invariant Face Detector,” in 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, Oct. 22-29, 2017.
[14] Jiankang Deng, Jia Guo, Yuxiang Zhou, Jinke Yu, Irene Kotsia, Stefanos Zafeiriou, “RetinaFace: Single-stage Dense Face Localisation in the Wild,” 2019. [Online]. Available: https://arxiv.org/abs/1905.00641. [Accessed Dec. 02, 2019].
[15] Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, Yu Qiao, “Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks,” IEEE Signal Processing Letters, vol. 23, no. 10, pp. 1499 - 1503, 2016.
[16] Yuezun Li, Siwei Lyu, “Exposing DeepFake Videos By Detecting Face Warping Artifacts,” 2018. [Online]. Available: https://arxiv.org/abs/1811.00656. [Accessed Dec. 14, 2019].
[17] Sepp Hochreiter, Jürgen Schmidhuber, “Long Short-Term Memory,” Neural Computation, vol. 9, no. 8, pp. 1735-1780, 1997.
[18] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, George Toderici, “Beyond short snippets: Deep networks for video classification,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, USA, Jun. 7-12, 2015.
[19] Yuezun Li, Ming-Ching Chang, Siwei Lyu, “In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking,” in 2018 IEEE International Workshop on Information Forensics and Security (WIFS), Hong Kong, Dec. 11-13, 2018.
[20] Avishek Joey Bose, Parham Aarabi, “Virtual Fakes: DeepFakes for Virtual Reality,” in 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP), Kuala Lumpur, Malaysia, Sep. 27-29, 2019.
[21] Jeff Donahue, Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, Trevor Darrell, “Long-Term Recurrent Convolutional Networks for Visual Recognition and Description,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 4, pp. 677 - 691, 2017.
[22] Karen Simonyan, Andrew Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” 2014. [Online]. Available: https://arxiv.org/abs/1409.1556. [Accessed Dec. 11, 2019].
[23] faiculty, “VGGNet：Very Deep Convolution Networks for Large-Scale Image Recognition,” 2018. [Online]. Available: https://zhuanlan.zhihu.com/p/32853559. [Accessed Nov. 28, 2019].
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, “Deep Residual Learning for Image Recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, USA, Jun. 27-30, 2016.
[25] Fengyi Song, Xiaoyang Tan, Xue Liu, Songcan Chen, “Eyes closeness detection from still images with multi-scale histograms of principal oriented gradients,” Pattern Recognition, vol. 47, no. 9, pp. 2825-2838, 2014.
[26] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting,” Journal of Machine Learning, vol. 15, no. 1, p. 1929–1958, 2014.
[27] svetozarstojkovic, “eeg-eye-state,” 2020. [Online]. Available: https://github.com/datasets/eeg-eye-state. [Accessed Jan. 07, 2020].
[28] Gang Pan, Lin Sun, Zhaohui Wu, Shihong Lao, “Eyeblink-based Anti-Spoofing in Face Recognition from a Generic Webcamera,” in 2007 IEEE 11th International Conference on Computer Vision, Rio de Janeiro, Brazil, Oct. 14-21, 2007.
[29] Jae-Yong Baek, Yong-Sang Yoo, Seung-Hwan Bae, “Generative Adversarial Ensemble Learning for Face Forensics,” IEEE Access, vol. 8, pp. 45421 - 45431, 2020.
[30] iperov, “DeepFaceLab,” 2020. [Online]. Available: https://github.com/iperov/DeepFaceLab. [Accessed Feb. 09, 2020].
[31] torzdf, “FACESWAP,” 2019. [Online]. Available: https://faceswap.dev//. [Accessed Dec. 20, 2019].
[32] DeepfakesWeb, “Deepfakes web β,” 2020. [Online]. Available: https://deepfakesweb.com/. [Accessed Feb. 08, 2020].
[33] Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, Siwei Lyu , “Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics,” 2020. [Online]. Available: https://arxiv.org/abs/1909.12962. [Accessed Apr. 16, 2020].
[34] Falko Matern, Christian Riess, Marc Stamminger, “Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations,” in 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW), Waikoloa Village, USA, Jan. 7-11, 2019.
[35] Peng Zhou, Xintong Han, Vlad I. Morariu, Larry S. Davis, “Two-Stream Neural Networks for Tampered Face Detection,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, USA, Jul. 21-26, 2017.
[36] Darius Afchar, Vincent Nozick, Junichi Yamagishi, Isao Echizen, “MesoNet: a Compact Facial Video Forgery Detection Network,” in 2018 IEEE International Workshop on Information Forensics and Security (WIFS), Hong Kong, Dec. 11-13, 2018.
[37] Xin Yang, Yuezun Li, Siwei Lyu, “Exposing Deep Fakes Using Inconsistent Head Poses,” in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, United Kingdom, May. 12-17, 2019.
[38] Catherine Stupp, “Fraudsters Used AI to Mimic CEO’s Voice in Unusual Cybercrime Case,” 2019. [Online]. Available: https://www.wsj.com/articles/fraudsters-use-ai-to-mimic-ceos-voice-in-unusual-cybercrime-case-11567157402. [Accessed Nov. 30, 2019].
[39] Sarah Cahlan, “How misinformation helped spark an attempted coup in Gabon,” 2020. [Online]. Available: https://www.washingtonpost.com/politics/2020/02/13/how-sick-president-suspect-video-helped-sparked-an-attempted-coup-gabon/. [Accessed Apr. 13, 2020].
[40] Kurt Wagner, “Twitter will label and remove deepfake videos under a new policy,” 2020. [Online]. Available: https://www.latimes.com/business/story/2020-02-04/twitter-deepfake-videos. [Accessed Mar. 04, 2020].
[41] Mark R. Warner, “Warner, Rubio Express Concern over Growing Threat Posed by Deepfakes,” 2019. [Online]. Available: https://www.warner.senate.gov/public/index.cfm/2019/10/warner-rubio-express-concern-over-growing-threat-posed-by-deepfakes. [Accessed Dec. 02, 2019].
[42] Mike Schroepfer, “Creating a data set and a challenge for deepfakes,” 2019. [Online]. Available: https://ai.facebook.com/blog/deepfake-detection-challenge/. [Accessed Dec. 30, 2019].
[43] Adi Robertson, “Virginia’s ‘revenge porn’ laws now officially cover deepfakes,” The Verge, 2019. [Online]. Available: https://www.theverge.com/2019/7/1/20677800/virginia-revenge-porn-deepfakes-nonconsensual-photos-videos-ban-goes-into-effect. [Accessed Dec. 01, 2019].
[44] Moderators, “Celebrity Facesets,” 2020. [Online]. Available: https://mrdeepfakes.com/forums/forum-celebrity-facesets. [Accessed Feb. 11, 2020].
[45] deepfakes, “faceswap,” 2020. [Online]. Available: https://github.com/deepfakes/faceswap. [Accessed Mar. 17, 2020].
[46] Changsha Shenduronghe Network Technology Company Limited, “ZAO,” 2019. [Online]. Available: https://apps.apple.com/cn/app/zao-%E8%83%BD%E6%8D%A2%E8%84%B8%E7%9A%84%E6%97%B6%E5%B0%9A%E6%9D%82%E5%BF%97/id1465199127. [Accessed Apr. 16, 2020].
[47] Donie O'Sullivan, “When seeing is no longer believing,” 2020. [Online]. Available: https://edition.cnn.com/interactive/2019/01/business/pentagons-race-against-deepfakes/. [Accessed Mar. 10, 2020].
[48] Giorgio Patrini PhD, Francesco Cavalli, Henry Ajder, Miguel Varela Ramos, Urja Khurana, Jan Pieter Bruins Slot, “DEEPTRACE,” Deeptrace, 2020. [Online]. Available: https://deeptracelabs.com/. [Accessed Feb. 05, 2020].
[49] Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, Matthias Nießner, “FaceForensics++: Learning to Detect Manipulated Facial Images,” 2019. [Online]. Available: https://arxiv.org/abs/1901.08971. [Accessed Nov. 25, 2019].
[50] P. Korshunov, S. Marcel, C. Sanderson, B.C. Lovell, “DeepfakeTIMIT,” 2020. [Online]. Available: https://www.idiap.ch/dataset/deepfaketimit. [Accessed Mar. 11, 2020].
[51] Justus Thies, Michael Zollhöfer, Matthias Nießner, “Deferred Neural Rendering: Image Synthesis using Neural Textures,” 2019. [Online]. Available: https://arxiv.org/abs/1904.12356. [Accessed Dec. 08, 2019].
[52] Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, Matthias Niessner, “Face2Face: Real-Time Face Capture and Reenactment of RGB Videos,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Jun. 27-30, 2016.
[53] Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, Matthias Nießner, “FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces,” 2018. [Online]. Available: https://arxiv.org/abs/1803.09179. [Accessed Dec. 27, 2019].
[54] Ivan Perov, Daiheng Gao, Nikolay Chervoniy, Kunlin Liu, Sugasa Marangonda, Chris Umé, Mr. Dpfks, Carl Shift Facenheim, Luis RP, Jian Jiang, Sheng Zhang, Pingyu Wu, Bo Zhou, Weiming Zhang, “DeepFaceLab: A simple, flexible and extensible face swapping framework,” 2020. [Online]. Available: https://arxiv.org/abs/2005.05535. [Accessed May. 19, 2020].
[55] Nikita S. Ivanov, Anton V. Arzhskov, Vitaliy G. Ivanenko, “Combining Deep Learning and Super-Resolution Algorithms for Deep Fake Detection,” in 2020 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus), St. Petersburg and Moscow, Russia, Jan. 27-30, 2020.
[56] Thanh Thi Nguyen, Cuong M. Nguyen, Dung Tien Nguyen, Duc Thanh Nguyen, Saeid Nahavandi, “Deep Learning for Deepfakes Creation and Detection,” 2019. [Online]. Available: https://arxiv.org/abs/1909.11573. [Accessed Dec. 11, 2019].
[57] Ying Zhang, Lilei Zheng, Vrizlynn L. L. Thing, “Automated face swapping and its detection,” in 2017 IEEE 2nd International Conference on Signal and Image Processing (ICSIP), Singapore, Singapore, Aug. 4-6, 2017.
[58] Ekraam Sabir, Jiaxin Cheng, Ayush Jaiswal, Wael AbdAlmageed, Iacopo Masi, Prem Natarajan, “Recurrent Convolutional Strategies for Face Manipulation Detection in Videos,” 2019. [Online]. Available: https://arxiv.org/abs/1905.00582. [Accessed Dec. 02, 2019].
[59] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, “Generative Adversarial Nets,” in Advances in Neural Information Processing Systems 27 (NIPS 2014), Montreal, Canada, Dec. 8-13, 2014.
[60] Faten F. Kharbat, Tarik Elamsy, Ahmed Mahmoud, Rami Abdullah, “Image Feature Detectors for Deepfake Video Detection,” in 2019 IEEE/ACS 16th International Conference on Computer Systems and Applications (AICCSA), Abu Dhabi, United Arab Emirates, Nov. 3-7, 2019.
[61] Badhrinarayan Malolan, Ankit Parekh, Faruk Kazi, “Explainable Deep-Fake Detection Using Visual Interpretability Methods,” in 2020 3rd International Conference on Information and Computer Technologies (ICICT), San Jose, USA, Mar. 9-12, 2020.
[62] Huaizu Jiang, Erik Learned-Miller, “Face Detection with the Faster R-CNN,” in 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA, May. 30 - Jun. 3, 2017.
[63] J.R.R. Uijlings, K.E.A. van de Sande, T. Gevers, A.W.M. Smeulders, “Selective Search for Object Recognition,” International Journal of Computer Vision, vol. 104, no. 2, p. 154–171, 2013.
[64] Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, Lior Wolf, “DeepFace: Closing the Gap to Human-Level Performance in Face Verification,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, USA, Jun. 23-28, 2014.