傳統的虛擬量測（Virtual Metrology, VM）系統在實務中仍有一些缺失，例如：它們需要在資料前處理與特徵提取上花費大量時間。此外，它們需要具有專業領域知識的專家來幫助選取資料的關鍵特徵，以建置足夠準確的VM系統。近年來，深度學習神經網路（Deep Learning Neural Networks, DLNNs）已成功應用於許多產業，例如：影像辨識和自然語言處理，它與淺層神經網絡不同，DLNNs可以透過輸入原始資料並自動提取不同層級的特徵，以實現自動化預測。在本研究中，我們應用DLNNs來解決傳統VM系統的缺失。實驗測試結果顯示，直接使用現有經典的DLNNs（例如：LeNet, AlexNet, VGG16）來建置VM模型仍無法準確預測生產品質，特別是當生產品質超出規格時，這些經典的DLNNs會產生較差的預測結果。
本論文提出了一種基於可關注量測邊界之分層卷積神經網路（Hierarchical Convolution Neural Network with Attention to Metrology Boundaries, HCNN-AMB）。 HCNN-AMB的架構由三個階層組成，一個CNN位於中間層（即CNN(M)）輸入製程資料以生成VM值。在上層，我們設計了一個CNN（即CNN(U)）並結合一個注意力網路（即AN(U)）來生成另一個VM值。其中，AN(U)的輸出可以用來調整製程資料，然後作為CNN(U)的輸入，如此設計可以更好地預測超出上邊界的生產品質。在下層，我們設計了一個CNN（即CNN(L)）結合一個注意力網絡（即AN(L)）來生成另一個VM值。其中AN(L) 的輸出可以用來調整製程資料，然後作為CNN(L)的輸入，如此設計可以更好地預測超出下邊界的生產品質。然後，我們設計了一個虛擬量測選擇器來選擇三個VM值之一作為輸出。此外，我們開發了一個帶有懲罰項和三階段訓練策略的損失函數，以提高預測精度並縮短HCNN-AMB的訓練時間。本論文以預測PET保特瓶生產品質為研究案例，在實驗測試結果中HCNN-AMB與LeNet, AlexNet, VGG16相比，整體平均的平均絕對百分比誤差（Mean Absolute Percentage Error, MAPE）方面可以達到最佳的預測精度，其結果分別為：0.664, 0.884, 0.705, 1.577。 特別是對於超出規格的生產品質預測性能，HCNN-AMB明顯優於其他三個DLNNs，其結果分別為：0.08, 1.258, 0.62, 1.647。

Traditional VM systems have some shortcomings in practice. For example, they need to consume a lot of time in data pre-processing and feature extraction. Also, they require experts with domain knowledge to facilitate selecting critical features for building a VM system with good enough accuracy. In recent years, deep learning neural networks (DLNNs) have successfully applied to many applications, such as image recognition and natural language processing. Unlike shadow neural networks, a DLNN can achieve automatic end-to-end prediction by inputting raw data, automatically extracting different-level features, and then generating the desired outputs. Thus, in this study, we apply DLNNs to address the abovementioned shortcomings of traditional VM systems. However, our experimental testing results show that directly using existing DLNNs (e.g., LeNet, AlexNet, and VGG16－three classic convolutional neural works) to create VM models could not predict production quality items accurately. In particular, when the production quality is out of specification, these classic DLNNs generate poor predictions.
This thesis proposes a hierarchical convolutional neural network with attention to metrology boundaries, called HCNN-AMB. The architecture of HCNN-AMB consists of three levels. A CNN is situated in the middle level (i.e., CNN(M)) to generate a VM value by inputting a sample of process data. In the upper level, we design a CNN (i.e., CNN(U)) combined with an attention network (i.e., AN(U)) to generate another VM value, where the output of AN(U) can adjust the values of the sample process data and then serve as the input of CNN(U). Such as design can better predict the production quality whose values are out of the upper bound. In the lower level, we design a CNN (i.e., CNN(L)) combined with an attention network (i.e., AN(L)) to generate another VM value, where the output of AN(L) can adjust the values of the sample process data and then serve as the input of CNN(L). Such as design can better predict the production quality whose values are out of the lower bound. Then, we design a virtual metrology selector to select one of the three VM values as the output. In addition, we develop a loss function with a penalty term and a three-phase training strategy to enhance the prediction accuracy and shorten the training time and of HCNN-AMB. Testing results of a case study in predicting the production quality of PET bottles show that the proposed HCNN-AMB can achieve the best overall average prediction accuracy compared with LeNet, AlexNet, and VGG16, in terms of mean absolute percentage error (MAPE): 0.664, 0.884, 0.705, and 1.577, respectively. In particular, regarding the prediction performance for the production quality item whose values are out of specification, HCNN-AMB is significantly better than the other three DLNNs: 0.08, 1.258, 0.62, and 1.647, respectively.

[1] J. Lee, M. Ghaffari, and S. Elmeligy, “Self-maintenance and engineering immune systems: Towards smarter machines and manufacturing systems,” Annual Reviews in Control, vol. 35, no. 1, pp. 111–122, 2011. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1367578811000083
[2] F.-T. Cheng, H.-C. Huang, and C.-A. Kao, “Developing an automatic virtual metrology system,” IEEE Transactions on Automation Science and Engineering, vol. 9, no. 1, pp. 181–188, 01 2012.
[3] F.-T. Cheng, J. Y.-C. Chang, H.-C. Huang, C.-A. Kao, Y.-L. Chen, and J.-L. Peng, “Benefit model of virtual metrology and integrating avm into mes,” IEEE Transactions on Semiconductor Manufacturing, vol. 24, no. 2, pp. 261–272, 05 2011.
[4] M. Koitzsch and A. Honold, “Evaluation of economic effects as the basis for assessing the investment into virtual metrology,” Proceedings of SEMATECH Advanced Equipment Control/Advanced Process Control Symposium XXII, pp. 172–194, 11 2010.
[5] B. Gill, T. Edgar, and J. Stuber, “A novel approach to virtual metrology using kalman filtering,” Proceedings of SEMATECH Advanced Equipment Control/Advanced Process Control Symposium XXII, pp. 267–291, 11 2010.
[6] J. Moyne, “Semi-e133: Process control system standard (pcs) standard: Standard update: Providing specifications for virtual metrology,” Proceedings of SEMATECH Advanced Equipment Control/Advanced Process Control Symposium XXII, pp. 422–449, 11 2010.
[7] O. Rothe, “ISMI next generation factory,” 2008. [Online]. Available: http://www.sematech.org/meetings/archives/emanufacturing/8546/01-NGF.pdf
[8] P. Gargini, “The international technology roadmap for semiconductors (itrs): ”past, present and future”,” in GaAs IC Symposium. IEEE Gallium Arsenide Integrated Circuits Symposium. 22nd Annual Technical Digest 2000. (Cat. No.00CH37084), 08 2000, pp. 3–5.
[9] S. Imai and M. Kitabata, “Prevention of copper interconnection failure in system on chip using virtual metrology,” IEEE Transactions on Semiconductor Manufacturing, vol. 22, no. 4, pp. 432–437, 11 2009.
[10] B. Kim and K. Park, “Modeling plasma etching process using a radial basis function network,” Microelectronic Engineering, vol. 77, no. 2, pp. 150–157, 02 2005. [Online].Available: https://www.sciencedirect.com/science/article/pii/S0167931704004939
[11] D. Han, S. B. Moon, K. Park, B. Kim, K. K. Lee, and N. J. Kim, “Modelling of plasma etching process using radial basis function network and genetic algorithm,” Vacuum, vol. 79, no. 3, pp. 140–147, 03 2005. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0042207X05001624
[12] T.-H. Lin, M.-H. Hung, R.-C. Lin, and F.-T. Cheng, “A virtual metrology scheme for predicting cvd thickness in semiconductor manufacturing,” in Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006., 2006, pp. 1054–1059.
[13] D. Zeng and C. J. Spanos, “Virtual metrology modeling for plasma etch operations,” IEEE Transactions on Semiconductor Manufacturing, vol. 22, no. 4, pp. 419–431, 11 2009.
[14] Y.-C. Su, M.-H. Hung, F.-T. Cheng, and Y.-T. Chen, “A processing quality prognostics scheme for plasma sputtering in tft-lcd manufacturing,” IEEE Transactions on Semiconductor Manufacturing, vol. 19, no. 2, pp. 183–194, 05 2006.
[15] T.-H. Pan, B.-Q. Sheng, D. S.-H. Wong, and S.-S. Jang, “A virtual metrology system for predicting end-of-line electrical properties using a mancova model with tools clustering,” IEEE Transactions on Industrial Informatics, vol. 7, no. 2, pp. 187–195, 05 2011.
[16] A. A. Khan, J. R. Moyne, and D. M. Tilbury, “An approach for factory-wide control utilizing virtual metrology,” IEEE Transactions on Semiconductor Manufacturing, vol. 20, no. 4, pp. 364–375, 11 2007.
[17] M.-H. Hung, C.-F. Chen, H.-C. Huang, H.-C. Yang, and F.-T. Cheng, “Development of an avm system implementation framework,” IEEE Transactions on Semiconductor Manufacturing, vol. 25, no. 4, pp. 598–613, 11 2012.
[18] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, ser. NIPS’12. Red Hook, NY, USA: Curran Associates Inc., 2012, p. 1097–1105.
[19] B. Esmael, A. Arnaout, R. Fruhwirth, and G. Thonhauser, “A statistical feature-based approach for operations recognition in drilling time series,” International Journal of Computer Information Systems and Industrial Management Applications., vol. 5, pp. 454–461, 12 2013.
[20] K. Delac, M. Grgic, and S. Grgic, “Independent comparative study of pca, ica, and lda on the feret data set,” International Journal of Imaging Systems and Technology, vol. 15, pp. 252 – 260, 01 2005.
[21] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” Neural Comput., vol. 18, no. 7, p. 1527–1554, 07 2006. [Online]. Available: https://doi.org/10.1162/neco.2006.18.7.1527
[22] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press, 2016, http://www.deeplearningbook.org.
[23] Q. Zhang, L. T. Yang, Z. Chen, P. Li, and M. J. Deen, “Privacy-preserving doubleprojection deep computation model with crowdsourcing on cloud for big data feature learning,” IEEE Internet of Things Journal, vol. 5, no. 4, pp. 2896–2903, 08 2018.
[24] X. Yuan, P. He, Q. Zhu, and X. Li, “Adversarial examples: Attacks and defenses for deep learning,” IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 9, pp. 2805–2824, 09 2019.
[25] M. Mahmud, M. S. Kaiser, A. Hussain, and S. Vassanelli, “Applications of deep learning and reinforcement learning to biological data,” IEEE Transactions on Neural Networks and Learning Systems, vol. PP, 11 2017.
[26] A. Panayides, M. Pattichis, S. Leandrou, C. Pitris, A. Constantinidou, and C. Pattichis, “Radiogenomics for precision medicine with a big data analytics perspective,” IEEE Journal of Biomedical and Health Informatics, vol. 23, pp. 2063–2079, 09 2019.
[27] S. Sakhavi, C. Guan, and S. Yan, “Learning temporal information for brain-computer interface using convolutional neural networks,” IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 11, pp. 5619–5629, 11 2018.
[28] X. Yang, P. Zhou, and M. Wang, “Person reidentification via structural deep metric learning,” IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 10, pp. 2987–2998, 10 2019.
[29] S. Zhang, S. Zhang, T. Huang, and W. Gao, “Speech emotion recognition using deep convolutional neural network and discriminant temporal pyramid matching,” IEEE Transactions on Multimedia, vol. 20, no. 6, pp. 1576–1590, 06 2018.
[30] Z. Yang, K. Merrick, L. Jin, and H. A. Abbass, “Hierarchical deep reinforcement learning for continuous action control,” IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 11, pp. 5174–5184, 11 2018.
[31] S. F. Dodge and L. J. Karam, “Visual saliency prediction using a mixture of deep neural networks,” IEEE Transactions on Image Processing, vol. 27, no. 8, p. 4080–4090, 08 2018. [Online]. Available: http://dx.doi.org/10.1109/TIP.2018.2834826
[32] W. Dong, P. Wang, W. Yin, G. Shi, F. Wu, and X. Lu, “Denoising prior driven deep neural network for image restoration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 10, p. 2305–2318, 10 2019. [Online]. Available: http://dx.doi.org/10.1109/TPAMI.2018.2873610
[33] J. Lu, J. Hu, and Y. Tan, “Discriminative deep metric learning for face and kinship verification,” IEEE Transactions on Image Processing, vol. 26, no. 9, pp. 4269–4282, 09 2017.
[34] V. Sze, Y. Chen, T. Yang, and J. S. Emer, “Efficient processing of deep neural networks: A tutorial and survey,” Proceedings of the IEEE, vol. 105, no. 12, pp. 2295–2329, 12 2017.
[35] K. Weiss, T. Khoshgoftaar, and D. Wang, “A survey of transfer learning,” Journal of Big Data, vol. 3, 05 2016.
[36] S. H. Lee, C. S. Chan, and P. Remagnino, “Multi-organ plant classification based on convolutional and recurrent neural networks,” IEEE Transactions on Image Processing, vol. 27, no. 9, pp. 4287–4301, 09 2018.
[37] K. Muhammad, J. Ahmad, Z. Lv, P. Bellavista, P. Yang, and S. W. Baik, “Efficient deep cnn-based fire detection and localization in video surveillance applications,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 49, no. 7, pp. 1419– 1434, 01 2019.
[38] J. Wu, S. Zhao, V. S. Sheng, J. Zhang, C. Ye, P. Zhao, and Z. Cui, “Weak-labeled active learning with conditional label dependence for multilabel image classification,” IEEE Transactions on Multimedia, vol. 19, no. 6, pp. 1156–1169, 06 2017.
[39] R. Ratajczak, C. F. Crispim-Junior, E. Faure, B. Fervers, and L. Tougne, “Automatic land cover reconstruction from historical aerial images: An evaluation of features extraction and classification algorithms,” IEEE Transactions on Image Processing, vol. 28, no. 7, pp. 3357–3371, 07 2019.
[40] L. Yu, H. Chen, Q. Dou, J. Qin, and P. Heng, “Automated melanoma recognition in dermoscopy images via very deep residual networks,” IEEE Transactions on Medical Imaging, vol. 36, no. 4, pp. 994–1004, 04 2017.
[41] W. Chu and Y. Wu, “Image style classification based on learnt deep correlation features,” IEEE Transactions on Multimedia, vol. 20, no. 9, pp. 2491–2502, 09 2018.
[42] R. Caldelli, R. Becarelli, and I. Amerini, “Image origin classification based on social network provenance,” IEEE Transactions on Information Forensics and Security, vol. 12, no. 6, pp. 1299–1308, 06 2017.
[43] P. Viswanath, S. Nagori, M. Mody, M. Mathew, and P. Swami, “End to end learning based self-driving using jacintonet,” in 2018 IEEE 8th International Conference on Consumer Electronics - Berlin (ICCE-Berlin), 2018, pp. 1–4.
[44] X. Zhao, Y. Lin, and J. Heikkilä, “Dynamic texture recognition using volume local binary count patterns with an application to 2d face spoofing detection,” IEEE Transactions on Multimedia, vol. 20, no. 3, pp. 552–566, 03 2018.
[45] C. Liu, T. Lin, Y. Wu, Y. Lin, H. Lee, Y. Tsao, and S. Chien, “Computation-performance optimization of convolutional neural networks with redundant filter removal,” IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 66, no. 5, pp. 1908–1921, 05 2019.
[46] F. Gama, A. G. Marques, G. Leus, and A. Ribeiro, “Convolutional neural network architectures for signals supported on graphs,” IEEE Transactions on Signal Processing, vol. 67, no. 4, p. 1034–1049, 02 2019. [Online]. Available: http://dx.doi.org/10.1109/TSP.2018.2887403
[47] S. Kiranyaz, T. Ince, O. Abdeljaber, O. Avci, and M. Gabbouj, “1-d convolutional neural networks for signal processing applications,” in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 8360–8364.
[48] S. Miao, Z. J. Wang, and R. Liao, “A cnn regression approach for real-time 2d/3d registration,” IEEE Transactions on Medical Imaging, vol. 35, no. 5, pp. 1352–1363, 05 2016.
[49] A. Ameri, M. Akhaee, E. Scheme, and K. Englehart, “Regression convolutional neural network for improved simultaneous emg control,” Journal of Neural Engineering, vol. 16, 03 2019.
[50] R. Zhao, R. Yan, Z. Chen, K. Mao, P. Wang, and R. X. Gao, “Deep learning and its applications to machine health monitoring,” Mechanical Systems and Signal Processing, vol. 115, pp. 213–237, 01 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0888327018303108
[51] L. Jing, M. Zhao, P. Li, and X. Xu, “A convolutional neural network based feature learning and fault diagnosis method for the condition monitoring of gearbox,” Measurement, vol. 111, pp. 1–10, 12 2017. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0263224117304517
[52] W. Zhang, G. Peng, C. Li, Y. Chen, and Z. Zhang, “A new deep learning model for fault diagnosis with good anti-noise and domain adaptation ability on raw vibration signals,” Sensors, vol. 17, p. 425, 02 2017.
[53] M. Terzi, C. Masiero, A. Beghi, M. Maggipinto, and G. A. Susto, “Deep learning for virtual metrology: Modeling with optical emission spectroscopy data,” in 2017 IEEE 3rd International Forum on Research and Technologies for Society and Industry (RTSI), 2017, pp. 1–6.
[54] M. Maggipinto, M. Terzi, C. Masiero, A. Beghi, and G. A. Susto, “A computer visioninspired deep learning architecture for virtual metrology modeling with 2-dimensional data,” IEEE Transactions on Semiconductor Manufacturing, vol. 31, no. 3, pp. 376– 384, 08 2018.
[55] T. Tsutsui and T. Matsuzawa, “Virtual metrology model robustness against chamber condition variation by using deep learning,” 12 2018, pp. 1–4.
[56] T. Tsutsui and T. Matsuzawa, “Virtual metrology model robustness against chamber condition variation using deep learning,” IEEE Transactions on Semiconductor Manufacturing, vol. 32, no. 4, pp. 428–433, 11 2019.
[57] C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 05 2015, pp. 1–9.
[58] M. Maggipinto, A. Beghi, S. McLoone, and G. A. Susto, “Deepvm: A deep learning-based approach with automatic feature extraction for 2d input data virtual metrology,” Journal of Process Control, vol. 84, pp. 24–34, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S095915241930191X
[59] K. B. Lee and C. O. Kim, “Recurrent feature-incorporated convolutional neural network for virtual metrology of the chemical mechanical planarization process,” Journal of Intelligent Manufacturing, vol. 31, no. 1, pp. 73–86, 01 2020. [Online]. Available: https://ideas.repec.org/a/spr/joinma/v31y2020i1d10.1007_s10845-018-1437-4.html
[60] L. C. Jain and L. R. Medsker, Recurrent Neural Networks: Design and Applications, 1st ed. USA: CRC Press, Inc., 1999.
[61] H. Tieng, Y.-Y. Li, K.-P. Tseng, H.-C. Yang, and F.-T. Cheng, “An automated dynamic-balancing-inspection scheme for wheel machining,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 2224–2231, 04 2020.
[62] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 11 1998.
[63] S. Liu and W. Deng, “Very deep convolutional neural network based image classification using small training sample size,” in 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), 2015, pp. 730–734.