隨著深度學習以及多模態技術的快速發展，如何使模型同時理解影像內容並以自然語言進行描述，已成為視覺語言研究中的一個重要課題。視覺語言模型的發展，使影像描述生成模型逐漸能夠產生流暢且語意合理的句子。然而，現有模型在描述影像細節時仍存在限制，特別是在物件屬性、空間關係、互動行為與局部視覺資訊等方面，生成結果可能僅聚焦於主要物件，而忽略較細緻的內容。BLIP-2 這個方法透過凍結影像編碼器與大型語言模型，並使用 Q-Former 作為視覺和語言間的橋接模組，在降低訓練成本的同時展現良好的影像描述能力。然而，由於視覺資訊會被壓縮至有限數量的查詢表示中，部分細節資訊可能在生成過程中無法被充分保留。為改善此問題，本文提出一種細節感知查詢增強模組，針對 Q-Former 產生的視覺查詢表示進行強化，使其在投影至語言模型前能保留更多與細節描述相關的資訊。實驗結果顯示，所提出之方法能在維持模型架構效率的同時，提升生成描述中對影像細節的表達能力，進一步改善影像描述生成之品質。

With the evolution of deep learning and multimodal technologies, empowering models to understand visual content and describe it using natural language has become a key focus in vision-language research. The evolution of vision-language models has empowered image captioning frameworks to generate fluent and semantically coherent sentences. However, existing models still face limitations when describing fine-grained details. Specifically, they often overlook subtle aspects such as object attributes, spatial relationships, interactive behaviors, and localized visual information, leading to results that focus primarily on dominant objects. While BLIP-2 effectively reduces training costs and delivers robust image captioning performance by freezing the image encoder and large language model while utilizing a Q-Former as a bridging module, it compresses visual information into a fixed number of query representations. Consequently, fine-grained details may not be preserved during the generation process. This paper addresses this limitation by proposing a detailed-aware query enhancement module designed to reinforce the visual query representations generated by the Q-Former, thereby retaining richer detail-related information before projecting them into the language model. Results from our experiments show that the proposed method enhances the model's capability to articulate image details while maintaining structural efficiency, thereby boosting the quality of generated image captions.

[1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” in Advances in Neural Information Processing Systems, vol. 25, 2012.
[2] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” in Advances in Neural Information Processing Systems, vol. 28, 2015.
[3] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and Tell: A Neural Image Caption Generator,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3156–3164, 2015.
[4] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel, and Y. Bengio, “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention,” in Proceedings of the 32nd International Conference on Machine Learning, pp. 2048–2057, 2015.
[5] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention Is All You Need,” in Advances in Neural Information Processing Systems, vol. 30, 2017.
[6] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, et al., “An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale,” arXiv preprint arXiv:2010.11929, 2020.
[7] K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, et al., “A Survey on Vision Transformer,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 1, pp. 87–110, 2022.
[8] A. Radford et al., “Learning Transferable Visual Models from Natural Language Supervision,” in Proceedings of the 38th International Conference on Machine Learning, pp. 8748–8763, 2021.
[9] J. Li, D. Li, C. Xiong, and S. Hoi, “BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation,” in Proceedings of the 39th International Conference on Machine Learning, pp. 12888–12900, 2022.
[10] J.-B. Alayrac et al., “Flamingo: A Visual Language Model for Few-Shot Learning,” in Advances in Neural Information Processing Systems, vol. 35, pp. 23716–23736, 2022.
[11] J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,” in Proceedings of the 40th International Conference on Machine Learning, pp. 19730–19742, 2023.
[12] J. Mao, W. Xu, Y. Yang, J. Wang, Z. Huang, and A. Yuille, “Deep Captioning with Multimodal Recurrent Neural Networks,” in Proceedings of the International Conference on Learning Representations, 2015.
[13] J. Donahue, L. A. Hendricks, M. Rohrbach, S. Venugopalan, S. Guadarrama, K. Saenko, and T. Darrell, “Long-Term Recurrent Convolutional Networks for Visual Recognition and Description,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2625–2634, 2015.
[14] A. Karpathy and L. Fei-Fei, “Deep Visual-Semantic Alignments for Generating Image Descriptions,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3128–3137, 2015.
[15] L. You, C. Jin, Z. Wang, C. Fang, and J. Luo, “Image Captioning with Semantic Attention,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4651–4659, 2016.
[16] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang, “Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6077–6086, 2018.
[17] J. Lu, D. Batra, D. Parikh, and S. Lee, “ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks,” in Advances in Neural Information Processing Systems, vol. 32, 2019.
[18] H. Tan and M. Bansal, “LXMERT: Learning Cross-Modality Encoder Representations from Transformers,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing, pp. 5099–5110, 2019.
[19] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, “UNITER: UNiversal Image-TExt Representation Learning,” in Proceedings of the European Conference on Computer Vision, pp. 104–120, 2020.
[20] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171–4186, 2019.
[21] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-End Object Detection with Transformers,” in Proceedings of the European Conference on Computer Vision, pp. 213–229, 2020.
[22] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models,” in Proceedings of the International Conference on Learning Representations, 2024.
[23] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual Instruction Tuning,” in Advances in Neural Information Processing Systems, vol. 36, pp. 34892–34916, 2023.
[24] W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, “InstructBLIP: Towards General-Purpose Vision-Language Models with Instruction Tuning,” in Advances in Neural Information Processing Systems, vol. 36, pp. 49250–49267, 2023.
[25] Y. Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y. Cao, “EVA: Exploring the Limits of Masked Visual Representation Learning at Scale,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[26] Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao, “EVA-CLIP: Improved Training Techniques for CLIP at Scale,” arXiv preprint arXiv:2303.15389, 2023.
[27] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “OPT: Open Pre-trained Transformer Language Models,” arXiv preprint arXiv:2205.01068, 2022.
[28] T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár, “Microsoft COCO: Common Objects in Context,” in Proceedings of the European Conference on Computer Vision, pp. 740–755, 2014.
[29] H. Dong, J. Li, B. Wu, J. Wang, Y. Zhang, and H. Guo, “Benchmarking and Improving Detail Image Caption,” arXiv preprint arXiv:2405.19092, 2024.
[30] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: A Method for Automatic Evaluation of Machine Translation,” in Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 311–318, 2002.
[31] S. Banerjee and A. Lavie, “METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65–72, 2005.
[32] C.-Y. Lin, “ROUGE: A Package for Automatic Evaluation of Summaries,” in Proceedings of the ACL Workshop on Text Summarization Branches Out, pp. 74–81, 2004.
[33] R. Vedantam, C. L. Zitnick, and D. Parikh, “CIDEr: Consensus-Based Image Description Evaluation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4566–4575, 2015.
[34] P. Anderson, B. Fernando, M. Johnson, and S. Gould, “SPICE: Semantic Propositional Image Caption Evaluation,” in Proceedings of the European Conference on Computer Vision, pp. 382–398, 2016.