視 覺 語 言 模 型 在 面 對 不 同 下 游 任 務 時， 通 常 仍 需 透 過 Low-Rank Adaptation（LoRA）進行任務專用化，且新增任務往往需要重新蒐集資料與訓練。為降低逐任務訓練與參數管理成本，本研究以 PaliGemma-3B 為基礎，建立可根據任務條件動態生成 LoRA 參數的 Hypernetwork，將文字條件式 LoRA 生成方法延伸至包含視覺編碼器與語言模型的視覺語言模型。
本研究以二十項視覺語言任務之任務專屬 LoRA 作為監督目標，透過參數重建方式訓練 Hypernetwork。為處理不同模型組件在深度、目標模組與參數維度上的差異，本研究引入組件－模組聯合嵌入、層級位置表示與跨組件輸出機制。此外，亦加入任務代表性影像，並比較向量串接與交叉注意力等視覺融合方式。
實驗結果顯示，Hypernetwork 能生成具實際下游效用的 LoRA：於二十項訓練任務上，生成 LoRA 之未加權巨集平均為 0.403，高於其直接重建目標逐任務 joint LoRA 的 0.371，並明顯高於未適配基礎模型的 0.093。生成模式消融進一步顯示，主要效益來自語言模型側——僅生成語言模型側參數即可達到 0.401，與同時生成兩側的 0.403 相差 0.002；僅生成視覺編碼器側參數則僅得 0.189，明顯低於對應的逐任務 LoRA 之 0.407，顯示視覺編碼器側參數較難重建。在未見任務上，生成 LoRA 於病理影像 PathVQA 之封閉式問題由 0.095 提升至 0.595、於放射影像 SLAKE 由 0.058提升至 0.674，顯示其具有一定程度的零樣本適配能力。視覺條件化結果則顯示，當文字描述已足以區分任務時，加入影像未能帶來穩定增益（三種條件設定平均皆約0.41）；但當所有任務共用同一句通用描述時，純文字條件降至 0.237，加入影像則維持 0.410，回復約 17.3 個百分點。綜合而言，本研究驗證了以 Hypernetwork 動態生成視覺語言模型 LoRA 的可行性，並釐清其跨組件生成、零樣本適配與視覺條件化的能力及限制。

Vision-language models typically require task specialization through Low-Rank Adaptation (LoRA), and supporting a new task usually means collecting annotated data and running another round of training. To reduce these costs, this study builds a Hypernetwork on PaliGemma-3B that generates LoRA parameters directly from task conditions, extending text-conditioned LoRA generation to a model containing both a vision encoder and a language model.
The Hypernetwork is trained by parameter reconstruction, using task-specific LoRA parameters from twenty vision-language tasks as supervision targets. To handle differences in model depth, target modules, and parameter dimensions across model components, it introduces component–module joint embeddings, shared layer-wise positional representations, and a variable-dimension output mechanism. Task-representative images are additionally examined as optional visual conditions.
Across the twenty tasks, the generated LoRA parameters attain an unweighted macro average of 0.403, above their reconstruction target (0.371) and far above the unadapted base model (0.093). The gains come almost entirely from the language-model side: generating only language-model parameters reaches 0.401, whereas generating only vision-encoder parameters reaches 0.189. On unseen tasks, the generated parameters raise closed-form accuracy from 0.095 to 0.595 on PathVQA and from 0.058 to 0.674 on SLAKE. Representative images bring no consistent gain while task descriptions remain distinctive; but when all tasks share one generic description, the text-only condition falls to 0.237 while image conditioning holds at 0.410, indicating that visual conditions mainly supply task identity rather than improved adapter content. Overall, the study demonstrates the feasibility of generating cross-component VLM LoRA parameters from task conditions, and delimits the range over which that ability holds.

[1] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in International Conference on Learning Representations (ICLR), 2022.
[2] R. Charakorn, E. Cetin, Y. Tang, and R. T. Lange, “Text-to-LoRA: Instant transformer adaption,” in International Conference on Machine Learning (ICML), 2025.
[3] J. Liang, W. Huang, X. Guo, G. Wan, B. Du, and M. Ye, “ThanoRA: Task heterogeneity-aware multi-task low-rank adaptation,” arXiv preprint arXiv:2505.18640, 2025.
[4] C. Huang, Q. Liu, B. Y. Lin, T. Pang, C. Du, and M. Lin, “Lorahub: Efficient cross-task generalization via dynamic LoRA composition,” in Conference on Language Modeling (COLM), 2024.
[5] Y. Gou, Z. Liu, K. Chen, L. Hong, H. Xu, A. Li, D.-Y. Yeung, J. T. Kwok, and Y. Zhang, “Mixture of cluster-conditional LoRA experts for vision-language instruction tuning,” IEEE Transactions on Image Processing, 2026.
[6] X. Chen, H. Zhang, Y. Qiu, X. Liang, Z. Li, G. Wang, W. Li, T. Mo, H. K.-H. So, and N. Wong, “GuiLoMo: Allocating expert number and rank for lora-moe via bilevel optimization with GuidedSelection vectors,” in Findings of the Association for Computational Linguistics: EMNLP 2025, 2025.
[7] H. Ivison, A. Bhagia, Y. Wang, H. Hajishirzi, and M. Peters, “HINT: Hypernetwork instruction tuning for efficient zero- and few-shot generalisation,” in Annual Meeting of the Association for Computational Linguistics (ACL), 2023.
[8] V. Akinwande, M. S. Norouzzadeh, D. Willmott, A. Bair, M. R. Ganesh, and J. Z. Kolter, “HyperCLIP: Adapting vision-language models with hypernetworks,” arXiv preprint arXiv:2412.16777, 2024.
[9] X. Song, J. Cui, H. Zhang, J. Shi, J. Chen, C. Zhang, and Y.-G. Jiang, “LoRA of change: Learning to generate LoRA for the editing instruction from a single before-after image pair,” arXiv preprint arXiv:2411.19156, 2024.
[10] H. Jo, H. Choi, M. Cho, and D. Min, “iConFormer: Dynamic parameter-efficient tuning with input-conditioned adaptation,” arXiv preprint arXiv:2409.02838, 2024.
[11] Y. Wang, C. Xiong, Z. Qin, M. Zhang, K. Xiao, and Z. Li, “Hylovqa: Dynamic hypernetwork-generated low-rank adaptation for continual visual question answering,” in International Joint Conference on Artificial Intelligence (IJCAI), 2026.
[12] L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, T. Unterthiner, D. Keysers, S. Koppula, F. Liu, A. Grycner, A. Gritsenko, N. Houlsby, M. Kumar, K. Rong, J. Eisenschlos, R. Kabra, M. Bauer, M. Bošnjak, X. Chen, M. Minderer, P. Voigtlaender, I. Bica, I. Balazevic, J. Puigcerver, P. Papalampidi, O. Henaff, X. Xiong, R. Soricut, J. Harmsen, and X. Zhai, “PaliGemma: A versatile 3B VLM for transfer,” arXiv preprint arXiv:2407.07726, 2024.
[13] X. Zhang, Y. Zhang, D. Long, W. Xie, Z. Dai, J. Tang, H. Lin, B. Yang, P. Xie, F. Huang, M. Zhang, W. Li, and M. Zhang, “mGTE: Generalized long-context text representation and reranking models for multilingual text retrieval,” in Conference on Empirical Methods in Natural Language Processing (EMNLP): Industry Track, 2024.
[14] A. Mishra, S. Shekhar, A. K. Singh, and A. Chakraborty, “OCR-VQA: Visual question answering by reading text in images,” in International Conference on Document Analysis and Recognition (ICDAR), 2019.
[15] A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach, “Towards VQA models that can read,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[16] A. F. Biten, R. Tito, A. Mafla, L. Gomez, M. Rusiñol, E. Valveny, C. V. Jawahar, and D. Karatzas, “Scene text visual question answering,” in IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[17] M. Mathew, D. Karatzas, and C. V. Jawahar, “DocVQA: A dataset for VQA on document images,” in IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021.
[18] R. Tanaka, K. Nishida, and S. Yoshida, “VisualMRC: Machine reading comprehension on document images,” in AAAI Conference on Artificial Intelligence (AAAI), 2021.
[19] A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque, “ChartQA: A benchmark for question answering about charts with visual and logical reasoning,” in Findings of the Association for Computational Linguistics: ACL, 2022.
[20] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the V in VQA matter: Elevating the role of image understanding in visual question answering,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[21] D. A. Hudson and C. D. Manning, “GQA: A new dataset for real-world visual reasoning and compositional question answering,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[22] D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham, “VizWiz grand challenge: Answering visual questions from blind people,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[23] Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen, “Evaluating object hallucination in large vision-language models,” in Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
[24] A. Suhr, S. Zhou, A. Zhang, I. Zhang, H. Bai, and Y. Artzi, “A corpus for reasoning about natural language grounded in photographs,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
[25] J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick, “CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[26] R. Zellers, Y. Bisk, A. Farhadi, and Y. Choi, “From recognition to cognition: Visual commonsense reasoning,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[27] P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao, “MathVista: Evaluating mathematical reasoning of foundation models in visual contexts,” in International Conference on Learning Representations (ICLR), 2024.
[28] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein, “Neural module networks,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[29] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi, “OK-VQA: A visual question answering benchmark requiring external knowledge,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[30] D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi, “A-OKVQA: A benchmark for visual question answering using world knowledge,” in European Conference on Computer Vision (ECCV), 2022.
[31] P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan, “Learn to explain: Multimodal reasoning via thought chains for science question answering,” in Advances in Neural Information Processing Systems (NeurIPS), 2022.
[32] A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi, “A diagram is worth a dozen images,” in European Conference on Computer Vision (ECCV), 2016.
[33] J. J. Lau, S. Gayen, A. B. Abacha, and D. Demner-Fushman, “A dataset of clinically generated visual questions and answers about radiology images,” Scientific Data, 2018.