視覺語言模型（VLM）的視覺 token 數量直接主導推理延遲與記憶體開銷，例如 LLaVA-1.5 的 CLIP 編碼器輸出 576 個 token、PaliGemma 的 SigLIP 編碼器輸出 1024 個 token，皆佔輸入序列 80% 以上。現有訓練無關壓縮方法多採固定全域壓縮率，未區分不同問題類型對視覺細節的依賴程度，例如是非類問題僅需全域語意，文字辨識類問題卻高度依賴高頻細節，固定壓縮率因此在不同問題上造成不必要的精度損失或 token 浪費。

本研究提出 QRoute-DCT，一個訓練無關（training-free）、問題類型感知的離散餘弦轉換（DCT）視覺 token 壓縮框架。透過規則式問題路由器依問題類型查表決定保留係數 C，並以 2D DCT 對 CLIP/SigLIP 特徵空間的 patch grid 進行低頻保留壓縮，全程不修改任何模型權重，以 monkey-patch 方式整合進 PaliGemma-3B 與 LLaVA-1.5-7B 兩種架構。

本研究確立三項主要結果：(1) 效率上，QRoute-DCT 於本文所比較的方法中位於精度與效率權衡的效率端，PaliGemma prefill 加速 2.71×（-63%）、LLaVA 2.39×（-58%），影像 KV cache 縮減 85%/73%，batch size 8 吞吐量提升 81%/91%；在相近 token 預算的比較下，於 CV-Bench 與 GQA 兩個資料集上同時取得優於 HiPrune 的精度與效率；此優勢範圍有明確邊界，若擴大至完整 7-benchmark suite（該比較之 DCT 欄為 static-C12 復現，非 QRoute-DCT），則 HiPrune 實際勝 5/7，並揭露對 FastV/FEATHER/DivPrune 之精度—速度取捨；上述實測加速有理論 FLOPs 上界作為佐證（6.5×/3.6×，Gemma-2B/LLaMA-7B，線性項主導）；(2) 在 VQAv2 全量驗證集（214,354 樣本）上，1000-seed 隨機路由消融證實路由策略本身具統計顯著價值（OCR +17.95pp、Color +8.38pp），驗證問題感知配置、而非單純 token 數量帶來增益；(3) 以 LoRA 作為壓縮損失探針，Compression × LoRA-variant 消融網格顯示 DCT 造成的精度損失中 43–57% 可由輕量 PEFT（608K 參數、1.5 小時訓練）恢復，其中 QRoute-DCT 搭配 Multi-layer LoRA 達 81.93%（163 token），以少 36% 的 token 超越 token 預算較大之未訓練 Static C=16 配置（256 token, 81.12%）。

此外，本研究透過固定方法、只改變視覺編碼器的控制實驗，發現頻域與注意力兩派壓縮方法的相對優劣會隨編碼器（SigLIP vs. CLIP）反轉，並於五個推理型資料集上刻畫出頻域方法於粗略場景推理佔優、於精確空間幾何任務居劣的邊界，作為本研究的延伸發現。

整體而言，本研究提出一套訓練無關、即插即用、以推理效率為優先的問題類型感知視覺 token 壓縮框架，並透過 LoRA 探針與跨編碼器分析，為視覺壓縮方法的選擇與後續優化建立可驗證的依據。

Vision-language models (VLMs) encode an image into hundreds to over a thousand visual tokens, which occupy more than 80% of the input sequence and dominate inference latency and memory cost. Existing training-free compression methods apply one fixed compression ratio to every question, although question types differ widely in how much visual detail they require. This thesis proposes QRoute-DCT, a training-free and question-type-aware compression framework. A rule-based router maps each question to a retention coefficient C through a lookup table, and a two-dimensional discrete cosine transform (DCT) keeps only the low-frequency block of the encoder patch grid, reducing the token count to C^2 without modifying any model weight. The method is integrated into PaliGemma-3B and LLaVA-1.5-7B by monkey-patching and evaluated on seven benchmarks. On the full VQAv2 validation set it reaches 77.26% with an average of 163 tokens; against a static baseline of comparable budget it improves OCR by 19.14 percentage points, color by 9.32 and spatial by 5.15, at the cost of 2.15 points on yes/no questions. Measured prefill latency drops by 2.71× and 2.39× on the two models, and image KV cache by 85% and 73%. A lightweight LoRA probe recovers 43–57% of the compression loss, indicating that roughly half of it stems from projector distribution mismatch rather than irreversible information loss.

[1] Nasir Ahmed, T. Natarajan, and K. R. Rao. Discrete cosine transform. IEEE Transactions on Computers, C-23(1):90–93, 1974.

[2] Anthropic. Claude 3.5 sonnet model card addendum. Technical report, 2024.

[3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv:2308.12966, 2023.

[4] Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. PaliGemma: A versatile 3b VLM for transfer. arXiv:2407.07726, 2024.

[5] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations (ICLR), 2023.

[6] Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. In International Conference on Learning Representations (ICLR), 2025.

[7] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. FastV: An empirical study on eliminating redundant visual tokens in multimodal large language models. In European Conference on Computer Vision (ECCV), 2024.

[8] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to GPT-4V? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):220101, 2024.

[9] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.

[10] Mark Endo, Xiaohan Wang, and Serena Yeung-Levy. Feather the throttle: Revisiting visual token pruning for vision-language model acceleration. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22826–22835, 2025.

[11] Gemini Team, Google DeepMind. Gemini: A family of highly capable multimodal models. arXiv:2312.11805, 2023.

[12] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.

[13] Xiaochuang Han, Ananya Kumar, Yulia Tsvetkov, and Marjan Ghazvininejad. JPEG-LM: Llms as image generators with canonical codec representations. arXiv:2408.08459, 2024.

[14] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. Transactions on Machine Learning Research (TMLR), 2024.

[15] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.

[16] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.

[17] Wenbo Hu, Zi-Yi Chang, Yu-Chuan Liu, Yu-Sheng Hu, William Yang Wang, and Xin Eric Wang. Matryoshka query transformer for large vision-language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 50168–50188, 2024.

[18] Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

[19] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.

[20] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.

[21] James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. FNet: Mixing tokens with fourier transforms. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2022.

[22] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.

[23] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023.

[24] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the 39th International Conference on Machine Learning (ICML), 2022.

[25] Shuai Li, Jian Xu, Xiao-Hui Li, Chao Deng, and Lin-Lin Huang. QG-VTC: Question-guided visual token compression in MLLMs for efficient VQA. arXiv:2504.00654, 2025.

[26] Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jianke Zhu, and Lei Zhang. TokenPacker: Efficient visual projector for multimodal LLM. International Journal of Computer Vision, 133(10):6794–6812, 2025.

[27] Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 39, pages 5334–5342, 2025.

[28] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, 2024.

[29] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. Blog post, https://llava-vl.github.io/blog/2024-01-30-llava-next/, 2024.

[30] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.

[31] Jizhihui Liu, Feiyi Du, Guangdao Zhu, Niu Lian, Jun Li, and Bin Chen. HiPrune: Training-free visual token pruning via hierarchical attention in vision-language models. arXiv:2508.00553, 2025.

[32] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.

[33] OpenAI. GPT-4V(ision) system card. Technical report, 2023.

[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.

[35] Saeed Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. DivPrune: Diversity-based visual token pruning for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9392–9401, 2025.

[36] Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. GFNet: Global filter networks for visual recognition. In Advances in Neural Information Processing Systems (NeurIPS), 2021.

[37] Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In Proc. European Signal Processing Conference (EUSIPCO), 2007.

[38] Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. LLaVA-PruMerge: Adaptive token reduction for efficient large multimodal models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22857–22867, 2025.

[39] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.

[40] Huanyu Wang, Jushi Kai, Haoli Bai, Lu Hou, Bo Jiang, Ziwei He, and Zhouhan Lin. Fourier-VLM: Compressing vision tokens in the frequency domain for large vision-language models. arXiv:2508.06038v2, 2025.

[41] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-VL: Enhancing vision-language model's perception of the world at any resolution. arXiv:2409.12191, 2024.

[42] Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. VisionZip: Longer is better but not necessary in vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19792–19802, 2025.

[43] Qizhe Zhang, Aosong Cheng, Ming Lu, Renrui Zhang, Zhiyong Zhuo, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. Beyond text-visual attention: Exploiting visual cues for effective token pruning in VLMs. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20857–20867, 2025.

[44] Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. LLaVA-Mini: Efficient image and video large multimodal models with one vision token. In International Conference on Learning Representations (ICLR), 2025.

[45] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In International Conference on Learning Representations (ICLR), 2024.