簡易檢索 / 詳目顯示

回結果列表
研究生： 廖柏棠
Liao, Bo-Tang
論文名稱： 結合多層次特徵提取與圖神經網絡之5G專利多標籤分類技術
Multi-Level Feature Extraction and Graph Neural Network Techniques for Multi-Label Classification of 5G-Related Patents
指導教授： 鄭憲宗
Cheng, Sheng-Tzong
學位類別： 碩士
Master
系所名稱： 電機資訊學院 - 資訊工程學系
Department of Computer Science and Information Engineering
論文出版年： 2024
畢業學年度： 112
語文別： 英文
論文頁數： 95
中文關鍵詞： 專利分類問題 、多標籤分類問題 、自然語言處理 、圖神經網路 、5G通訊
外文關鍵詞： patent classification, multi-label classification, natural language processing, graph neural network, 5G communication
相關次數： 點閱：42  下載：2 
分享至:
查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報

    • 隨著5G技術的迅速普及，有效的智慧財產管理變得尤為重要，特別是在專利分類上，以捕捉快速的進步和應用。機器學習已成為這一領域的關鍵工具。透過自動化將專利分類到相應類別的過程，這些技術提高專利審查的效率和精確性，減少對人工排序和分析的依賴。本研究開發一種創新的人工智慧模型，該模型基於機器學習技術，用於自動化5G相關專利的分類。我們的方法通過將BERT模型與圖神經網絡（GNN）結合，改進以往的框架，促進專利數據的精確特徵提取和語境理解。首先，從Google專利獲取全面的專利描述和元數據，並進行資料前處理，包括文本標準化和分詞。特徵提取採用由BERT衍生的模型來有效處理專利中的複雜專有名詞。隨後，將這些特徵整合進用於捕捉專利之間關係的GNN中，從而增強分類結果。結合模型的效能通過精度、準確性、召回率和F1分數等傳統指標進行驗證，證明其在5G進步領域中簡化專利管理過程的潛力。這種結構化的方法不僅簡化專利分類過程，還通過自動化分類提高比以前方法更高的準確性，有效地適應迅速變化的需求。

    The swift proliferation of 5G technology necessitates effective intellectual property management, particularly in patent classification to capture the rapid advancements and applications. Machine learning has emerged as an essential tool in this arena. By automating the process of categorizing patents into their respective classes, these technologies enhance the efficiency and precision of patent examination, reducing the reliance on manual sorting and analysis. This study develops an innovative AI model, grounded in machine learning techniques, to automate the classification of 5G-related patents. Our methodology enhances prior frameworks by integrating BERT models with a Graph Neural Network (GNN), facilitating precise feature extraction and contextual understanding of patent data. Initially, comprehensive patent descriptions and metadata are acquired from Google Patents and subjected to rigorous preprocessing, including text normalization and tokenization. Feature extraction employs BERT-derived models to handle complex technical terminologies within the patents effectively. Subsequently, these features are incorporated into a GNN utilized to capture the relationships among patents, thereby enhancing the classification results. The combined model's efficacy is validated against traditional metrics such as accuracy, precision, recall, and the F1 score, proving its potential to streamline patent management processes in the realm of 5G advancements. This structured approach not only streamlines the patent classification process but also advances the field by automating classification with greater accuracy than previous methodologies, effectively adapting to the demands of a rapidly evolving digital landscape.

    摘要 i Abstract ii ACKNOWLEDGMENT iv TABLE OF CONTENTS v LIST OF TABLES vii LIST OF FIGURES viii Chapter 1. Introduction and Motivation 1 Chapter 2. Background and Related Works 4 2.1 Overview of Patent Classification Systems 4 2.1.1 Introduction to Patent Classification 5 2.1.2 Demands and Challenges of 5G-related Patents Classification 6 2.2 Advancements in 5G Technology and its Patents 8 2.3 Application of Machine Learning in Patent Analysis 9 2.3.1 SVM Model 9 2.3.2 CNN / RNN Model 11 2.3.3 Attention Mechanism Model 15 2.3.4 Other Model 20 2.4 Multi-Class Multi-Label Classification 21 2.4.1 Definition and Characteristics 21 2.4.2 Challenges 22 2.4.3 Multi-Label Classification of 5G-Related Patents 23 2.5 NLP Techniques in Patent Classification 25 2.5.1 Detailed Review of NLP Methodologies Applied to Patent Texts 25 2.5.2 Specific Focus on the Use of BERT and Other Transformer Models 26 2.5.3 Advantages and Challenges 26 Chapter 3. Approach 28 3.1 Overview of Research Methodology 28 3.2 Data Collection 30 3.3 Data Preprocessing 30 3.3.1 TF-IDF 31 3.3.2 LDA 32 3.3.3 Noun Phrase Extraction 33 3.3.4 BERT-derived Tokenizer 35 3.4 Initial Period Classification Models : BERT-derived Models 35 3.4.1 BERT model 37 3.4.2 BERT derivative models 40 3.4.3 ALBERT 40 3.4.4 RoBERTa 41 3.4.5 ELECTRA 41 3.4.6 DistilBERT 42 3.4.7 Limitations of Using GPT Models for Multi-Label Classification of 5G-Related Patents 43 3.5 Framework for Graph Construction 46 3.6 Second Period Classification Models : GNN models 49 3.6.1 GNN mechanism 50 3.6.2 GCN Model 51 3.6.3 GAT Model 53 3.7 Loss Function 55 3.8 Evaluation Metrics 56 3.8.1 TP, TN, FP and FN 56 3.8.2 Accuracy 59 3.8.3 Precision 59 3.8.4 Recall 60 3.8.5 F1 Score 60 Chapter 4. Implementation and Experiments 61 4.1 Datasets and Development Environment 61 4.1.1 Dataset 61 4.1.2 Development Environment 62 4.2 Data Preprocessing 64 4.3 Selection of Initial Classification Models 68 4.4 Selection of Framework for Graph Structure 72 4.5 Selection of Second Classification Models 73 4.6 Evaluation of Differences Between Two Loss Functions 75 4.7 Hyperparameter 77 Chapter 5. Conclusions and Future Work 79 5.1 Conclusions 79 5.2 Future Work 80 Reference 82

    1. Woo, S., P. Jang, and Y. Kim, Effects of intellectual property rights and patented knowledge in innovation and industry value added: A multinational empirical analysis of different industries. Technovation, 2015. 43: p. 49-63.
    2. Zhang, H.E., K.H. Wong, and V. Chang, Patent analysis in the 5G network. Journal of Global Information Management (JGIM), 2021. 29(6): p. 1-28.
    3. Gao, F., et al., Revealing development trends in blockchain-based 5g network technologies through patent analysis. Sustainability, 2021. 13(5): p. 2548.
    4. Liu, D., W. Huang, and G. Xing. Research on 5G technology competition situation based on patent analysis. in 2021 2nd International Conference on Artificial Intelligence and Information Systems. 2021.
    5. 曾凰嘉, "融合 BERT 的多層次特徵提取和卷積神經網 路模型於多標籤 5G 專利分類與專利資訊提取的研究". 碩士, 資訊工程學系, 國立成功大學. 2023. [Online]. Available: https://hdl.handle.net/11296/p84f57.
    6. Devlin, J., et al., Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
    7. Kipf, T.N. and M. Welling, Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
    8. Veličković, P., et al., Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.
    9. Aronson, J.E., T.-P. Liang, and R.V. MacCarthy, Decision support systems and intelligent systems. Vol. 4. 2005: Pearson Prentice-Hall Upper Saddle River, NJ, USA:.
    10. Teece, D.J., Technological leadership and 5G patent portfolios: Guiding strategic policy and licensing decisions. California Management Review, 2021. 63(3): p. 5-34.
    11. Chang, S.-H., Revealing development trends and key 5G photonic technologies using patent analysis. Applied Sciences, 2019. 9(12): p. 2525.
    12. Attaran, M., The impact of 5G on the evolution of intelligent automation and industry digitization. Journal of ambient intelligence and humanized computing, 2023. 14(5): p. 5977-5993.
    13. Benzineb, K. and J. Guyot, Automated patent classification. Current challenges in patent information retrieval, 2011: p. 239-261.
    14. Zhang, X., Interactive patent classification based on multi-classifier fusion and active learning. Neurocomputing, 2014. 127: p. 200-205.
    15. Chen, Y.-L. and Y.-C. Chang, A three-phase method for patent classification. Information Processing & Management, 2012. 48(6): p. 1017-1030.
    16. Venugopalan, S. and V. Rai, Topic based classification and pattern identification in patents. Technological Forecasting and Social Change, 2015. 94: p. 236-250.
    17. Liu, D.R. and M.J. Shih, Hybrid‐patent classification based on patent‐network analysis. Journal of the American Society for Information Science and Technology, 2011. 62(2): p. 246-256.
    18. Montecchi, T., D. Russo, and Y. Liu, Searching in Cooperative Patent Classification: Comparison between keyword and concept-based search. Advanced Engineering Informatics, 2013. 27(3): p. 335-345.
    19. Jotisakulratana, M., et al. Applying International Patent Classification (IPC) to strategic planning processes of an R&D organization: The case of NECTEC, Thailand. in 2013 Proceedings of PICMET'13: Technology Management in the IT-Driven Services (PICMET). 2013. IEEE.
    20. Kamateri, E., et al. Automated single-label patent classification using ensemble classifiers. in Proceedings of the 2022 14th International Conference on Machine Learning and Computing. 2022.
    21. Goyal, M. and Y. Lather, Advancement of communication technology from 1G to 5G. International Journal of Advanced Research in IT and Engineering, 2015. 4(5): p. 1-17.
    22. Andrews, J.G., et al., What will 5G be? IEEE Journal on selected areas in communications, 2014. 32(6): p. 1065-1082.
    23. Hao, H., D. Hui, and D. Lau, Material advancement in technological development for the 5G wireless communications. Nanotechnology Reviews, 2020. 9(1): p. 683-699.
    24. Vishwanathan, S. and M.N. Murty. SSVM: a simple SVM algorithm. in Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No. 02CH37290). 2002. IEEE.
    25. Wu, C.-H., Y. Ken, and T. Huang, Patent classification system using a new hybrid genetic algorithm support vector machine. Applied Soft Computing, 2010. 10(4): p. 1164-1177.
    26. Wu, J.-L., et al., A patent quality analysis and classification system using self-organizing maps with support vector machine. Applied soft computing, 2016. 41: p. 305-316.
    27. Yun, J. and Y. Geum, Automated classification of patents: A topic modeling approach. Computers & Industrial Engineering, 2020. 147: p. 106636.
    28. Li, S., et al., DeepPatent: patent classification with convolutional neural networks and word embedding. Scientometrics, 2018. 117(2): p. 721-744.
    29. Xiao, L., G. Wang, and Y. Zuo. Research on patent text classification based on word2vec and LSTM. in 2018 11th International Symposium on Computational Intelligence and Design (ISCID). 2018. IEEE.
    30. Risch, J. and R. Krestel, Domain-specific word embeddings for patent classification. Data Technologies and Applications, 2019. 53(1): p. 108-122.
    31. Haghighian Roudsari, A., et al., PatentNet: multi-label classification of patent documents using deep learning based language understanding. Scientometrics, 2022. 127(1): p. 207-231.
    32. Hu, J., et al., A hierarchical feature extraction model for multi-label mechanical patent classification. Sustainability, 2018. 10(1): p. 219.
    33. Lee, J.-S. and J. Hsiang, Patent classification by fine-tuning BERT language model. World Patent Information, 2020. 61: p. 101965.
    34. Chikkamath, R., et al. Patent Classification Using BERT-for-Patents on USPTO. in Proceedings of the 2022 5th International Conference on Machine Learning and Natural Language Processing. 2022.
    35. Yu, L., et al., Semantic Similarity Matching for Patent Documents Using Ensemble BERT-related Model and Novel Text Processing Method. arXiv preprint arXiv:2401.06782, 2024.
    36. Bekamiri, H., D.S. Hain, and R. Jurowetzki, Patentsberta: A deep nlp based hybrid model for patent distance and classification using augmented sbert. arXiv preprint arXiv:2103.11933, 2021.
    37. Lu, X. and B. Ni, BERT-CNN: A hierarchical patent classifier based on a pre-trained language model. arXiv preprint arXiv:1911.06241, 2019.
    38. Salton, G., Introduction to modern information retrieval. McGraw-Hill, 1983.
    39. Blei, D.M., A.Y. Ng, and M.I. Jordan, Latent dirichlet allocation. Journal of machine Learning research, 2003. 3(Jan): p. 993-1022.
    40. Barker, K. and N. Cornacchia. Using noun phrase heads to extract document keyphrases. in Advances in Artificial Intelligence: 13th Biennial Conference of the Canadian Society for Computational Studies of Intelligence, AI 2000 Montéal, Quebec, Canada, May 14–17, 2000 Proceedings 13. 2000. Springer.
    41. Feng, F. and W.B. Croft, Probabilistic techniques for phrase extraction. Information processing & management, 2001. 37(2): p. 199-220.
    42. Subhashini, R. and V.J.S. Kumar. Shallow NLP techniques for noun phrase extraction. in Trendz in information sciences & computing (tisc2010). 2010. IEEE.
    43. Veritawati, I., I. Wasito, and T. Basaruddin, Text preprocessing using annotated suffix tree with matching keyphrase. International Journal of Electrical and Computer Engineering, 2015. 5(3): p. 409.
    44. Liu, Y., A Context Free Gramma for Key Noun-Phrase Extraction from Text. 2019.
    45. Wang, H. and J. Li. Unsupervised keyphrase extraction from single document based on bert. in 2022 International Seminar on Computer Science and Engineering Technology (SCSET). 2022. IEEE.
    46. Sun, S., et al., Joint keyphrase chunking and salience ranking with bert. arXiv preprint arXiv:2004.13639, 2020.
    47. Nikzad-Khasmakhi, N., et al., Phraseformer: Multimodal key-phrase extraction using transformer and graph embedding. arXiv preprint arXiv:2106.04939, 2021.
    48. Imani, A., et al., Graph-based multilingual label propagation for low-resource part-of-speech tagging. arXiv preprint arXiv:2210.09840, 2022.
    49. Lai, P.-T. and Z. Lu, BERT-GT: cross-sentence n-ary relation extraction with BERT and Graph Transformer. Bioinformatics, 2020. 36(24): p. 5678-5685.
    50. Lan, Z., et al., Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019.
    51. Liu, Y., et al., Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
    52. Clark, K., et al., Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020.
    53. Sanh, V., et al., DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
    54. Sanchez-Lengeling, B., et al., A gentle introduction to graph neural networks. Distill, 2021. 6(9): p. e33.

    下載圖示 校內：立即公開
    校外：立即公開
    QR CODE