簡易檢索 / 詳目顯示

回結果列表
研究生： 簡佑軒
Chien, Yu-Hsuan
論文名稱： 使用類Transformer Layer Module的FMM網絡進行IC載板的超解析處理
Super-Resolution of IC Carrier Boards Using Transformer Layer Module-Like FMM Network
指導教授： 連震杰
Lien, Jenn-Jier
學位類別： 碩士
Master
系所名稱： 工學院 - 智慧製造國際碩士學位學程
International Master Program on Intelligent Manufacturing
論文出版年： 2024
畢業學年度： 112
語文別： 英文
論文頁數： 114
中文關鍵詞： 超解析 、Transformer Layer Module-Like 、Spatially-Adaptive Feature Modulation 、Convolutional Channel Mixer 、Pixel Shuffle 、高效率 、CUDA
外文關鍵詞： Super-Resolution, Transformer Layer Module-Like, Spatially-Adaptive Feature Modulation, Convolutional Channel Mixer, Pixel Shuffle, High Efficiency, CUDA
相關次數： 點閱：87  下載：10 
分享至:
查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報

    • 隨著半導體產業的生態越來越蓬勃發展且在全球經濟價值地位越來越高，半導體封裝行業也變得至關重要，因此對於積體電路 (Integrated Circuit, IC) 載板上的自動化光學檢測 (Auto Optical Inspection, AOI) 設備的要求也越來越高。目前光學設備技術已達到1 um 的精度這意味著對於成像品質的要求極高，在設備上皆使用高倍率鏡頭 (2X 以上的顯微鏡頭)。但在高倍率鏡頭的使用上容易導致景深不足，進而影響成像品質。在解決這一問題的過程中也遇到了硬體穩定度和產品設備的挑戰。這些因素直接影響著光學設備的成像效果，因此需要綜合考慮並找到解決方案。本論文方法包括兩部分：1) 使用Transformer Layer Module Like Feature Mixing Module Network (TLM-Like FMM Network) 做兩倍的高效率的超解析深度學習演算來達到影像品質的提升，2) 運用 PyTorch CUDA 及不同並行處理的方式來達成在多張影像的情形下來加快執行的速度優化。透過論文的方法實現在低倍率下保持高解析度的應用，且在移動速度300 mm/s 的平台上提升AOI 系統的能力，從而克服目前高倍率鏡頭導致的成像品質問題。第一部分，TLM-Like FMM Network 運用到類似Vision Transformer (ViT) 的概念，是由Spatially-Adaptive Feature Modulation (SAFM) 保有全局的特徵提取以及Convolutional Channel Mixer (CCM) 局部特徵提取組合而來並結合Pixel shuffle 的上採樣方式來達到目標大小，這可以讓模型能夠更輕量化且更有效率去處理影像的超解析。第二部分，整體的模型開發使用PyTorch，因此影像在進行模型的推論演算時會加入PyTorch CUDA 應用程式介面來加以運用GPU 進行加速的動作，接著在多張影像的處理上面運用DataLoader 以及影像預處理的方式來加快影像讀取及模型推論演算的速度，最後再由CPU 的另一個Thread 進行當前圖像的存寫達到模型推論及圖像存寫的並行處理來做程式的加速優化。

    As the semiconductor industry continues to flourish and its economic value on a global scale rises, the semiconductor packaging sector has become increasingly crucial. Consequently, the demand for automated optical inspection (AOI) equipment on integrated circuit (IC) carrier boards has intensified. Current optical equipment technology has achieved a precision of 1 um, indicating extremely high requirements for imaging quality and typically utilizes high-magnification lenses (microscopes with 2X magnification or higher). However, the use of high-magnification lenses often results in insufficient depth of field, adversely affecting imaging quality. Addressing this issue involves overcoming challenges related to hardware stability and equipment performance. These factors directly impact the imaging results of optical devices, necessitating a comprehensive approach to find effective solutions. This thesis proposes a method comprising two parts: 1) Using a Transformer Layer Module Like Feature Mixing Module Network (TLM-Like FMM Network) for efficient 2X super-resolution deep learning calculations to enhance image quality. 2) Employing PyTorch CUDA and various parallel processing techniques to optimize execution speed when processing multiple images. The proposed approach aims to maintain high resolution at lower magnifications and enhance the AOI system's capabilities on platforms moving at speeds of 300 mm/s, thereby overcoming the current imaging quality issues caused by high-magnification lenses. The first part involves the application of TLM-Like FMM Network, which incorporates concepts like Vision Transformer (ViT). This network combines global feature extraction using Spatially-Adaptive Feature Modulation (SAFM) with local feature extraction using Convolutional Channel Mixer (CCM) and achieves target size through Pixel Shuffle upsampling. This combination allows the model to be more lightweight and efficient in handling super-resolution imaging tasks. The second part focuses on the overall model development using PyTorch. During model inference calculations, PyTorch CUDA API is utilized to accelerate processing with GPU. For processing multiple images, DataLoader and image preprocessing techniques are employed to expedite image reading and model inference calculations. Finally, another CPU thread is used for concurrent image storage and retrieval, achieving parallel processing of model inference and image storage for optimized program acceleration.

    摘要 I Abstract II 誌謝 III Table of Contents V List of Tables VII List of Figures IX Chapter 1 Introduction 1 1.1 Motivation and Objective 1 1.2 Global Framework 3 1.3 Related Works 6 1.4 Contributions 11 Chapter 2 System Setup and Functions Specification 13 2.1 System Setup 13 2.2 Hardware Specifications: PC 13 2.3 Hardware Specifications: Camera and Lens 14 2.4 User Interface Function 15 Chapter 3 Transformer Layer Module-Like FMM Network 18 3.1 Training and Inference of TLM-Like FMM Network 18 3.1.1 TLM-Like FMM Network Training Framework 18 3.1.2 TLM-Like FMM Network Inference Framework 22 3.1.3 Loss Function 23 3.2 Tokenize Training Data to Tokens 25 3.3 Training TLM-Like FMM Network 29 3.3.1 Spatially-adaptive Feature Modulation module 33 3.3.2 Convolutional Channel Mixer Module 39 3.3.3 Pixel-shuffle Upsampling 42 Chapter 4 Optimizing TLM-Like FMM Network Demo Speed 46 4.1 Introduction to CUDA 46 4.2 CUDA Operations in PyTorch 49 4.3 TLM-Like FMM Net Using PyTorch CUDA framework 53 4.3.1 TLM-Like FMM Net Process Flow 53 4.3.2 TLM-Like FMM Network Using PyTorch CUDA Code 57 4.4 Computing Time Improvement: Process One Image 59 4.5 Computing Time Improvement: Process Multi Images 61 4.5.1 Method 1: Continuous Processing of Single Muti Images 62 4.5.2 Method 2: Parallel Wright Data Method Ⅰ 64 4.5.3 Method 3: Parallel Wright Data Method Ⅱ 65 4.5.4 Method 4: DataLoader with Parallel Wright Data Method Ⅰ 67 4.5.5 Method 5: Data Preprocessing with Parallel Wright Data Ⅰ 69 Chapter 5 Experimental Results 73 5.1 Data Collection, Metric and Implementation details 73 5.1.1 Data Collection 73 5.1.2 Metrics 79 5.1.3 TLM-Like FMM Network Implementation details 81 5.2 Experiment Result 85 5.2.1 TLM-Like FMM Network Total Result 85 5.2.A Comparison of Computational Costs for 4 Models 86 5.2.B Evaluation of IQA Metrics Across 4 Models 87 5.2.C Demo Time Comparison for 4 Models Across 5 Methods 94 5.3 Result Analysis 95 5.3.1 6FMMs-gray Model Demo IC Carrier Boards Result 95 Chapter 6 Conclusion and Future Work 97 Reference 99

    [1] P. Arbelaez, M. Maire, C.C. Fowlkes and J. Malik, “Contour Detection and Hierarchical Image Segmentation,” In IEEE Transactions on Pattern Analysis and Machine Intelligence, pp 898-916, 2011.
    [2] M. Bevilacqua, A. Roumy, C. Guillemot and M.A. Morel, “Low-Complexity Single-Image Super-Resolution Based on Nonnegative Neighbor Embedding,” In British Machine Vision Conference, 135.1-135.10, 2012.
    [3] B.B. Bilecen and M. Ayazoglu, “Bicubic++: Slim, Slimmer, Slimmest Designing An Industry-Grade Super-Resolution Network,” ArXiv, 2305.02126, 2023.
    [4] K. Briechle and U.D. Hanebeck, “Template Matching Using Fast Normalized Cross Correlation,” In International Society for Optics and Photonics, 2001.
    [5] X. Chen, X. Wang, J. Zhou, Y. Qiao and C. Dong, “Activating More Pixels In Image Super-Resolution Transformer, ” In Conference on Computer Vision and Pattern Recognition, pp. 22367-22377, 2023.
    [6] M.V. Conde, E. Zamfir, R. Timofte, D. Motilla, C. Liu, Z. Zhang, Y. Peng, Y. Lin, J. Guo, X. Zou, Y. Chen, Y. Liu, J. Hao, Y. Yan, Y. Zhang, G. Li and L. Sun, “Efficient Deep Models For Real-Time 4K Image Super-Resolution NTIRE 2023 Benchmark And Report,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1495-1521, 2023.
    [7] C. Dong, C.C. Loy, K. He and X. Tang, “Image Super-Resolution Using Deep Convolutional Networks,” In IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 295-307, 2016.
    [8] J. Guo, X. Zou, Y. Chen, Y. Liu, J. Hao, J. Liu and Y. Yan, “AsConvSR: Fast And Lightweight Super-Resolution Network With Assembled Convolutions,” In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 1582-1592, 2023.
    [9] D. Hendrycks and K. Gimpel, “Gaussian error linear units,” In arXiv, 1606.08415, 2016.
    [10] Y. Hu, Y. Chen, X. Li and J. Feng, “Dynamic Feature Fusion For Semantic Edge Detection,” In International Joint Conference on Artificial Intelligence, Vol. 1, pp. 782–788, 2019.
    [11] J. Hu, L. Shen and G. Sun, “Squeeze-and-Excitation Networks,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7132-7141, 2018.
    [12] J.B. Huang, A. Singh and N. Ahuja, “Single Image Super-Resolution from Transformed Self-Exemplars,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 5197-5206, 2015.
    [13] Q. Jiang, Z. Liu, K. Gu, F. Shao, X. Zhang, H. Liu and W. Lin, “Single Image Super-Resolution Quality Assessment: A Real-World Dataset, Subjective Studies and an Objective Metric,” In IEEE Transactions on Image Processing, pp. 2279–2294, 2022.
    [14] A. Kirillov, Y. Wu, K.He and R. Girshick, “PointRend: Image Segmentation As Rendering,” In Conference on Computer Vision and Pattern Recognition, pp. 9799-9808, 2020.
    [15] J.P. Lewis and I.L. Magic, “Fast Normalized Cross-Correlation,” In Vision Interface, pp. 819-843, 1995.
    [16] J. Liang, J. Cao, G. Sun, K. Zhang, L.V. Gool and R. Timofte, “SwinIR: Image Restoration Using Swin Transformer,” In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 1833-1844, 2021.
    [17] B. Lim, S. Son, H. Kim, S. Nah and K.M. Lee, “Enhanced Deep Residual Networks for Single Image Super-Resolution,” In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136-144, 2017.
    [18] Z. Lin, Z. Zhang, L.Z. Chen, M.M. Cheng and S.P. Lu, “Interactive Image Segmentation With First Click Attention,” In Conference on Computer Vision and Pattern Recognition, pp. 13339-13348, 2020.
    [19] W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert and Z. Wang, “Real-Time Single Image And Video Super-Resolution Using An Efficient Sub-Pixel Convolutional Neural Network,” In Conference on Computer Vision and Pattern Recognition, pp. 1874-1883, 2016.
    [20] X. Soria, E. Riba and A. Sappa, “Dense Extreme Inception Network: Towards A Robust CNN Model For Edge Detection,” In Conference on Computer Vision and Pattern Recognition, pp. 1923-1932, 2020.
    [21] L. Sun, J. Pan and J. Tang, “ShuffleMixer: An Efficient Convnet for Image Super-Resolution,” In Neural Information Processing Systems, 35, 2022.
    [22] L. Sun, J. Dong, J. Tang and J. Pan, “Spatially-Adaptive Feature Modulation For Efficient Image Super-Resolution,” In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 13190-13199, 2023.
    [23] M.X. Tan and Q. Le, “EfficientNetV2: Smaller Models and Faster Training,” In International Conference on Machine Learning, PMLR 139:10096-10106, 2021.
    [24] R. Timofte, E. Agustsson, L.V. Gool, M.H. Yang and L. Zhang, “NTIRE 2017 challenge on single image super-resolution: Methods and results,” In IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 114-125, 2017.
    [25] Z. Wang, J. Chen and S.C.H. Hoi, “Deep Learning for Image Super-Resolution: A Survey,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, pp 3365 - 3387, 2020.
    [26] E. Zamfir, M.V. Conde and R. Timofte, “Towards Real-Time 4K Image Super-Resolution,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1522-1532, 2023.
    [27] R. Zeyde, M. Elad and M. Protter, “On Single Image Scale-Up Using Sparse-Representations,” In International Conference on Curves and Surfaces, pp 711–730, 2012.
    [28] X. Zhu, S. Lyu, X. Wang and Q. Zhao, “TPH-YOLOv5: Improved YOLOv5 Based On Transformer Prediction Head For Object Detection On Drone-captured Scenarios,” In IEEE International Conference on Computer Vision, pp. 2778-2788, 2021.
    [29] “Pinned Host Memory,” [Online] Available: https://developer.nvidia.com/blog/how-optimize-data-transfers-cuda-cc/.
    [30] “PyTorch FFT-Based Frequency Loss,” [Online] Available: https://pytorch.org/docs/stable/generated/torch.fft.fft2.html.
    [31] “Pytorch DataLoader Document,” [Online] Available: https://pytorch.org/docs/stable/data.html.
    [32] J. Hui, "“CUDA Tutorial”," Jhui.github.io, 2017. [Online]. Available: https://jhui.github.io/2017/03/06/CUDA/. [Accessed: 22- Jan- 2021].
    [33] “Source Code Analysis: How PyTorch Use GPU,” [Online] Available: https://www.cnblogs.com/rossiXYZ/p/15518457.html.
    [34] “Pytorch CUDA Document,” [Online] Available: https://pytorch.org/docs/stable/notes/cuda.html.

    下載圖示 校內：立即公開
    校外：立即公開
    QR CODE