在大型語言模型（Large Language Models, LLM）開始發展之後，取代了傳統的資料擴增方法，使用LLM做資料擴增成了資料擴增方式的新的手段，LLM可以將文字和語料做換句話說或者語意延伸，以此增加資料和語料的豐富度。但LLM背後的風險就會是LLM產生的幻覺現象（hallucination）或者生成內容偏離遠原本的語意，進一步影響到任務訓練的精確度。
為此，本研究提出一個系統性的方法架構，透過不同的資料擴增方法和設計四種語意對齊機制，來生成多樣化的資料並篩選掉與原始語料不一致的生成資料。我們將「語料對齊」視為一個從資料面影響模型泛化能力的問題，目標是從大量生成樣本中挑選出在語料上與原始語料具一致性的資料，藉此提升資料擴增的品質與模型學習的穩健度。實驗討論三種產生LLM的擴增資料的方法:「換句話說」、「照樣造句」、「展延句子」，以及四種對齊模型，分別是關鍵詞相關的「基於關鍵詞相似性」、零樣本學習相關的「基於Zero-Shot分類分數相似性」、分類置信度相關的「基於預訓練分類器置信度」、語句相似度相關的「基於後訓練相似度模型」，根據這些組合和基本的BERT模型的比較去探討對齊模型對於資料過濾的效益。本研究驗證了在低資源條件下，結合大型語言模型（LLM）進行資料擴增並配合適當的語意對齊策略，能夠顯著提升分類模型的泛化能力。研究設計的四種對齊機制皆能有效過濾語意偏離的生成資料，使模型在過濾後的驗證準確率均顯著高於未經擴增的原始模型表現。其中基於預訓練分類器置信度與基於後訓練相似度模型兩種對齊方法表現最為優異，驗證準確率最高可達 92.20%，不僅明顯優於相同資料量下的原始10%訓練資料模型，甚至超越使用完整原始資料所訓練模型之最高驗證準確率 92.08%。有效的增加模型的泛化程度，提升模型整體的準確度和穩健度。

In natural language processing (NLP) tasks, data augmentation is a common technique to enrich data features and complexity. Traditional augmentation methods, such as back translation and rule-based replacement strategies, can generate new sentences but are often constrained by the original dataset. As a result, the generated data tends to have limited diversity and complexity, which can be insufficient for tasks requiring nuanced semantic features, and may even negatively affect model training.
With the advancement of Large Language Models (LLMs), using LLMs for data augmentation has become a new approach. By paraphrasing or semantically extending existing text, LLMs can significantly enhance the richness of datasets. This method leverages the extensive pretraining corpora and strong semantic understanding capabilities of LLMs, enabling the generation of diverse and contextually rich text. Compared to traditional methods, LLM-based augmentation provides broader coverage and improves generalization in low-resource scenarios. However, this approach also introduces potential risks, such as hallucination effects or semantic drift, which may impair the training accuracy of downstream tasks.
To address this, the study investigates alignment models that ensure LLM-generated content remains semantically consistent with the original corpus. Three augmentation strategies are explored: paraphrasing, sentence reconstruction, and sentence extension. These are paired with four alignment methods: keyword similarity, Zero-Shot classification score similarity, pretrained classifier confidence, and fine-tuned sentence similarity modeling. The combinations are evaluated against a baseline BERT model to assess their effectiveness in filtering augmented data. Experimental results show that the baseline model trained on 100% of the original dataset achieves a validation accuracy of 92.08%. In comparison, the proposed method—using only 10% of the original dataset combined with LLM-based augmentation and appropriate alignment—can achieve a slightly higher validation accuracy of 92.20%. This demonstrates the potential of alignment-enhanced LLM augmentation in significantly improving model generalization, even under limited data conditions.

Anaby-Tavor, A., Carmeli, B., Goldbraich, E., Kantor, A., Kour, G., Shlomov, S., ... & Zwerdling, N. (2020, April). Do not have enough data? Deep learning to the rescue!. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 05, pp. 7383-7390).
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., ... & Wu, Y. (2023). Palm 2 technical report. arXiv preprint arXiv:2305.10403.
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017). Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 39(12), 2481-2495.
Barker, K., Awasthy, P., Ni, J., & Florian, R. (2021, August). IBM MNLP IE at CASE 2021 task 2: NLI reranking for zero-shot text classification. In Proceedings of the 4th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021) (pp. 193-202).
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 1877-1901.
Chowdhary, K., & Chowdhary, K. R. (2020). Natural language processing. Fundamentals of artificial intelligence, 603-649.
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.
Church, K. W. (2017). Word2Vec. Natural Language Engineering, 23(1), 155-162.
Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019, June). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) (pp. 4171-4186).
Dong, C., Li, Y., Gong, H., Chen, M., Li, J., Shen, Y., & Yang, M. (2022). A survey of natural language generation. ACM Computing Surveys, 55(8), 1-38.
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., ... & Sui, Z. (2022). A survey on in-context learning. arXiv preprint arXiv:2301.00234.
Feng, S. Y., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., & Hovy, E. (2021). A survey of data augmentation approaches for NLP. arXiv preprint arXiv:2105.03075.
Gong, Y., Luo, H., & Zhang, J. (2017). Natural language inference over interaction space. arXiv preprint arXiv:1709.04348.
Goodfellow, I. (2016). Deep learning.
Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020). Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964.
He, P., Liu, X., Gao, J., & Chen, W. (2020). Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654.
Hedderich, M. A., Lange, L., Adel, H., Strötgen, J., & Klakow, D. (2020). A survey on recent approaches for natural language processing in low-resource scenarios. arXiv preprint arXiv:2010.12309.
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., ... & Fung, P. (2023). Survey of hallucination in natural language generation. ACM computing surveys, 55(12), 1-38.
Jurafsky, D. (2000). Speech & language processing. Pearson Education India.
Kumar, V., Choudhary, A., & Cho, E. (2020). Data augmentation using pre-trained transformer models. arXiv preprint arXiv:2003.02245.
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., ... & Zettlemoyer, L. (2019). Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Li, Y., Chen, C. P., & Zhang, T. (2022). A survey on siamese network: Methodologies, applications, and opportunities. IEEE Transactions on artificial intelligence, 3(6), 994-1014.
Li, Z., Liu, F., Yang, W., Peng, S., & Zhou, J. (2021). A survey of convolutional neural networks: analysis, applications, and prospects. IEEE transactions on neural networks and learning systems, 33(12), 6999-7019.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... & Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
M. Grootendorst, “Keybert: Minimal keyword extraction with bert.”
Medsker, L. R., & Jain, L. (2001). Recurrent neural networks. Design and Applications, 5(64-67), 2.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., ... & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35, 27730-27744.
Pennington, J., Socher, R., & Manning, C. D. (2014, October). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543).
Phang, J., Févry, T., & Bowman, S. R. (2018). Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9.
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
Ramos, J. (2003, December). Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning (Vol. 242, No. 1, pp. 29-48).
Reimers, N., & Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
Schank, R. C. (1972). Conceptual dependency: A theory of natural language understanding. Cognitive psychology, 3(4), 552-631.
Sennrich, R., Haddow, B., & Birch, A. (2015). Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709.
Shu, L., Xu, H., Liu, B., & Chen, J. (2022). Zero-shot aspect-based sentiment analysis. arXiv preprint arXiv:2202.01924.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., & Potts, C. (2013, October). Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing (pp. 1631-1642).
Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., ... & Kenealy, K. (2024). Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018). GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Wang, Y., Yao, Q., Kwok, J. T., & Ni, L. M. (2020). Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3), 1-34.
Wei, J., & Zou, K. (2019). Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv:1901.11196.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., ... & Fedus, W. (2022). Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.
Xie, Q., Dai, Z., Hovy, E., Luong, T., & Le, Q. (2020). Unsupervised data augmentation for consistency training. Advances in neural information processing systems, 33, 6256-6268.
Yoo, K. M., Park, D., Kang, J., Lee, S. W., & Park, W. (2021). GPT3Mix: Leveraging large-scale language models for text augmentation. arXiv preprint arXiv:2104.08826.
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016). Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530.
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., ... & Wen, J. R. (2023). A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2).
de Carvalho Souza, M. E., & Weigang, L. (2025). Grok, Gemini, ChatGPT and DeepSeek: Comparison and Applications in Conversational Artificial Intelligence. INTELIGENCIA ARTIFICIAL, 2(1).
Data, F. Do we really have to filter out random noise in pre-training data for language models?.
xAI. Grok 3 beta — the age of reasoning agents. https://x.ai/blog/grok-3, 2025. Accessed: 2025-02-21.
Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training.
Hardin, P. J., & Shumway, J. M. (1997). Statistical significance and normalized confusion matrices. Photogrammetric engineering and remote sensing, 63(6), 735-739.