在半導體與材料先進製程中，掃描電子顯微鏡（Scanning Electron Microscope, SEM）影像之失效分析高度仰賴工程師之領域知識與經驗，且目前不存在具備缺陷與嚴重程度標註之公開 SEM 資料集；同時，先進製程資料之高度機密性使雲端方案難以落地，模型必須於廠內本地部署。此研究針對上述問題，提出一套領域知識驅動之自動化偽標籤（pseudo-label）方法：以 Gemma 4 31B 為生成式標註引擎，將失效分析工程師之領域知識編碼為提示規則，為公開之 NFFA-EUROPE SEM 資料集（去重後 15,810 張、六類樣品型態）自動生成結構化之形貌描述與嚴重程度（None/Minor/Fatal）偽標籤，並以「標註層／轉換層」雙層解耦之迭代錯誤分析持續改進品質：標註層以分層人工盲審稽核驅動特化提示介入，將 MEMS 類別之 curtaining 幻覺誤判率自 8.4% 壓制至 1.9%，最終 180 筆全面盲審之整體核可率達 80.0%；轉換層以三個受控版本（v7–v9）之資料工程迭代，解決否定句特徵崩潰、聯合分佈失衡與思維鏈不對稱等系統性失效模式，產出 14,424 筆最終訓練資料集。
實驗結果顯示，僅以此偽標籤資料集直接進行監督式微調（Supervised Fine-Tuning, SFT），兩個輕量化模型之嚴重度判讀效能均顯著超越參數量更大之 LLaVA-1.5 7B：MobileVLM V2 3B 取得 0.674 Macro F1 與 0.719 Severity Accuracy，遠勝 LLaVA-1.5 7B 之 0.520 與 0.612，證實訓練標籤之一致性與穩定度，較模型規模與單筆樣本之原始準確率更具決定性。知識蒸餾（Knowledge Distillation, KD）實驗顯示 KD 與直接 SFT 效能相當，且不受教師強弱影響，與 capacity gap 理論一致；zero-shot 基線則顯示三個模型底座未經偽標籤訓練時 Macro F1 均為 0，證實偽標籤訓練為模型具備本任務判讀能力之前提條件。部署端，TinyLLaVA 1.5B 於單張 RTX 3080（12GB VRAM）消費級顯卡上以 2.78 GB 峰值顯存達成每張樣本 2.15 秒之即時推論（較 LLaVA-1.5 7B 快約 3.24 倍），滿足半導體廠房對高機密性與低延遲本地部署之硬性需求。

Scanning electron microscope (SEM) image interpretation for failure analysis in semiconductor manufacturing depends on scarce domain expertise, and no public SEM dataset provides defect or severity annotations. This study establishes a domain-knowledge-driven pseudo-labeling methodology that converts the practical knowledge of failure analysis engineers into prompting rules for a generative annotation engine (Gemma 4 31B), automatically producing structured analyses and three-level severity labels (None/Minor/Fatal) for 15,810 public NFFA-EUROPE SEM images across six sample categories. Label quality is managed by a two-layer decoupled design: an annotation layer refined through stratified blind expert audits and specialized prompt interventions, and a conversion layer refined through three controlled data-engineering iterations. The resulting 14,424-sample dataset is used to fine-tune three models independently: LLaVA-1.5 7B and two lightweight models (MobileVLM V2 3B, TinyLLaVA 1.5B). Both lightweight models surpass the larger LLaVA-1.5 7B in severity assessment—MobileVLM V2 3B reaches 0.674 Macro F1 versus 0.520—showing that label consistency outweighs model scale. Knowledge distillation yields no additional gain over direct fine-tuning regardless of teacher strength and is therefore excluded from the final deployment plan; zero-shot baselines of all three base models score 0 Macro F1, confirming pseudo-label training as a prerequisite for task competence. The final deployment adopts MobileVLM V2 3B for precision-priority scenarios and TinyLLaVA 1.5B for latency-priority scenarios, both fine-tuned directly on the v9 pseudo-label dataset; the 1.5B model runs on a single consumer GPU (RTX 3080, 12GB) with 2.78 GB peak memory at 2.15 seconds per image, meeting the confidentiality and latency requirements of on-premises industrial deployment.

[1] Rossella Aversa, Mohammad H. Modarres, Stefano Cozzini, and Regina Ciancio. NFFA-EUROPE - 100% SEM dataset. https://b2share.eudat.eu/records/80df8606fcdb4b2bae1656f0dc6db8ba, 2018. NFFA-EUROPE Horizon 2020 Project, Grant No. 654360, v2.0. Distributed under Creative Commons Attribution (CC-BY) license. doi:10.23728/b2share.80df8606fcdb4b2bae1656f0dc6db8ba.
[2] Yongzhu Cai and Hong Wang. A visual language model enabling intelligent nanomaterial scanning electron micrograph annotation. Nanoscale, 17(43):25136–25151, 2025.
[3] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4794–4802, 2019.
[4] Kamal Choudhary. MicroscopyGPT: Generating atomic-structure captions from microscopy images of 2D materials with vision-language transformers. The Journal of Physical Chemistry Letters, 16(27):7028–7035, 2025.
[5] Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, and Chunhua Shen. MobileVLM V2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766, 2024.
[6] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023.
[7] Gemma Team, Google DeepMind. Gemma 4 technical report. arXiv preprint arXiv:2607.02770, 2026.
[8] Remco Geurts. LLM-driven automation in FIB-SEM: Enhancing microscope control with AutoScript and vision-based AI. Microscopy and Microanalysis, 31(Supplement 1):ozaf048.1088, 2025.
[9] Remco Geurts, Pavel Potocek, Remco Schoenmakers, and Giovanni Mariotta. Revolutionizing electron microscopy through intuitive language-driven interfaces: The emergence of the EM CoPilot. In BIO Web of Conferences (EMC 2024), volume 129, page 10018. EDP Sciences, 2024.
[10] Joseph I. Goldstein, Dale E. Newbury, Joseph R. Michael, Nicholas W. M. Ritchie, John Henry J. Scott, and David C. Joy. Scanning Electron Microscopy and X-ray Microanalysis. Springer, 4th edition, 2018.
[11] Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary language models. In International Conference on Learning Representations (ICLR), 2024.
[12] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C'esar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, S'ebastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023.
[13] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll'ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15979–15988, 2022.
[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
[15] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
[16] Ting-Hung Lin, Hung-Jen Chen, Yuan-Chung Wei, Sung-Po Yang, Yung-Lun Lin, Parthasarathy Sriram, Guan-Hong Liou, Yu-Chieh Huang, Yi-Hsuan Chiu, Po-Chun Lai, Yiyi Wang, and Mark Peng. GenAI applications of vision-language models for semiconductor defect classification. In Metrology, Inspection, and Process Control XXXIX, volume 13426, page 134260S. SPIE, 2025.
[17] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[18] Nagarajan Natarajan, Inderjit S. Dhillon, Pradeep Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in Neural Information Processing Systems (NeurIPS), volume 26, 2013.
[19] Curtis G. Northcutt, Lu Jiang, and Isaac L. Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373–1411, 2021.
[20] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 8748–8763. PMLR, 2021.
[21] David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit. Deep learning is robust to massive label noise. arXiv preprint arXiv:1705.10694, 2017.
[22] Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in RLHF. In Conference on Language Modeling (COLM), 2024.
[23] Prateek Verma, Minh-Hao Van, and Xintao Wu. Beyond human vision: The role of large vision language models in microscope image analysis. arXiv preprint arXiv:2405.00876, 2024.
[24] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with Noisy Student improves ImageNet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10687–10698, 2020.
[25] Christoph Zauner. Implementation and benchmarking of perceptual image hash functions. Master's thesis, Upper Austria University of Applied Sciences, Hagenberg, 2010.
[26] Lanfeng Zhong, Zongyao Huang, Yang Liu, Wenjun Liao, Shichuan Zhang, Guotai Wang, and Shaoting Zhang. VLM-CPL: Consensus pseudo labels from vision-language models for annotation-free pathological image classification. IEEE Transactions on Medical Imaging, 2025.
[27] Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. TinyLLaVA: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402.14289, 2024.
[28] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023.