人體網格重建可由影像估計人體姿態、體型與完整三維人體幾何資訊，並應用於醫療照護、監控系統、人機互動及運動分析等領域。然而，現有人體網格重建方法主要針對 RGB 影像設計，在低照度環境下容易受到可見光影像品質下降的影響。目前在具有不同人體姿勢、視角與背景的真實場景中，利用熱紅外影像進行人體網格重建的研究仍相對有限。本論文以預訓練的 RGB 人體網格重建模型為基礎，提出一套可同時處理 RGB、紅外以及成對 RGB-紅外影像的人體網格重建框架。模型以融合分支處理 RGB 或紅外單一輸入，並在成對 RGB-紅外輸入時額外使用與融合分支共用凍結權重的 RGB 分支，同時以輕量模組調整 ViT 深層特徵，並估計不同影像區域中 RGB 與紅外資訊的相對可靠度。對於成對輸入，模型首先根據可靠度權重融合 RGB 與紅外 token，並於 ViT 深層將 RGB 分支的特徵以可靠度引導的方式引入融合分支，以利用兩種模態之間的互補資訊。由於本研究使用的熱影像資料集缺乏 SMPL 標註，本研究利用 SMPLify 擬合產生 SMPL 人體網格標註，並作為姿態與體型的偽真值監督資訊。實驗結果顯示，所提出的方法能在 LLVIP-Pose 與 OpenThermalPose2 的紅外影像上重建合理的 SMPL 人體網格。相較於具代表性的 RGB 人體網格重建方法，所提出的方法在多項二維與三維評估指標上呈現較佳表現。此外，在 LLVIP-Pose 的成對 RGB-紅外影像輸入下，所提出的方法能利用兩種模態的互補資訊，進一步提升多項評估指標與人體網格重建表現。整體而言，透過紅外特徵適應與可靠度引導的 RGB 與紅外資訊融合，所提出的方法能將預訓練的 RGB 人體網格重建模型延伸至熱紅外影像，並建立一套支援 RGB、紅外及成對 RGB-紅外影像輸入的統一人體網格重建框架。

Human Mesh Recovery (HMR) estimates human pose, body shape, and complete three-dimensional body geometry from images, and supports applications such as healthcare monitoring, surveillance, human-computer interaction, and sports analysis. However, most existing HMR methods are designed for RGB images and can be affected by degraded visible-light observations under low-light conditions. Research on HMR from thermal infrared (IR) images in real-world scenes with diverse human poses, viewpoints, and backgrounds remains relatively limited. This thesis builds upon a pretrained RGB HMR model and proposes a framework that supports RGB, IR, and paired RGB-IR image inputs within a single model. The model processes RGB or IR single-modality input through the fusion branch, while paired RGB-IR input additionally uses an RGB branch that shares the frozen pretrained encoder parameters with the fusion branch. For paired input, RGB and IR tokens are first fused according to the estimated reliability weights, while deeper reliability-guided fusion brings RGB-branch features into the fusion branch to exploit complementary information from the two modalities. Since the thermal image datasets used in this study lack SMPL annotations, SMPLify is used to generate SMPL mesh annotations as pseudo ground truth for pose and shape supervision. Experimental results show that the proposed method can reconstruct plausible SMPL human meshes from IR images in LLVIP-Pose and OpenThermalPose2 and effectively exploit complementary information from the two modalities under paired RGB-IR input. Compared with representative RGB-based HMR methods, the proposed method also achieves improvements in multiple 2D and 3D evaluation metrics. The experimental results demonstrate that IR feature adaptation and reliability-guided RGB-IR fusion can extend a pretrained RGB HMR model to IR images while supporting RGB, IR, and paired RGB-IR inputs in real-world scenes.

[1] M. Zheng, B. Planche, X. Gong, F. Yang, T. Chen, and Z. Wu, “Self-supervised 3d patient modeling with multi-modal attentive fusion,” in Medical Image Computing and Computer Assisted Intervention (MICCAI), 2022.
[2] A. Tandon, A. Goyal, H. M. Clever, and Z. Erickson, “Bodymap: Jointly predicting body mesh and 3d applied pressure map for people in bed,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 2480–2490.
[3] Y. Zhang, W. Zeng, S. Jin, C. Qian, P. Luo, and W. Liu, “When pedestrian detection meets multi-modal learning: Generalist model and benchmark dataset,” in European Conference on Computer Vision (ECCV). Springer, 2024, pp. 430–448.
[4] S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, “Convolutional pose machines,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4724–4732.
[5] K. Sun, B. Xiao, D. Liu, and J. Wang, “Deep high-resolution representation learning for human pose estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 5693–5703.
[6] Y. Xu, J. Zhang, Q. Zhang, and D. Tao, “Vitpose: Simple vision transformer baselines for human pose estimation,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022, pp. 38 571–38 584.
[7] A. Kanazawa, M. J. Black, D. W. Jacobs, and J. Malik, “End-to-end recovery of human shape and pose,”in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[8] F. Bogo, A. Kanazawa, C. Lassner, P. Gehler, J. Romero, and M. J. Black, “Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image,” in Computer Vision – ECCV 2016. Springer International Publishing, 2016, pp. 561–578.
[9] S. Goel, G. Pavlakos, J. Rajasegaran, A. Kanazawa, and J. Malik, “Humans in 4D: Reconstructing and tracking humans with transformers,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[10] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black, “SMPL: A skinned multi-person linear model,” ACM Transactions on Graphics, vol. 34, no. 6, pp. 248:1–248:16, 2015.
[11] Z. Li, J. Liu, Z. Zhang, S. Xu, and Y. Yan, “Cliff: Carrying location information in full frames into human pose and shape estimation,” in European Conference on Computer Vision (ECCV), 2022.
[12] S. K. Dwivedi, Y. Sun, P. Patel, Y. Feng, and M. J. Black, “TokenHMR: Advancing human mesh recovery with a tokenized pose representation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024, pp. 1323–1333.
[13] P. Patel and M. J. Black, “CameraHMR: Aligning people with perspective,” arXiv preprint arXiv:2411.08128, 2024.
[14] M. Cormier, C. Ng Zhi Yi, A. Specker, B. Blaß, M. Heizmann, and J. Beyerer, “Leveraging thermal imaging for robust human pose estimation in low-light vision,” in Computer Vision – ACCV 2024 Workshops. Springer Nature Singapore, 2025, pp. 69–86.
[15] A. Kuzdeuov, M. Zakaryanov, A. Tleuliyev, and H. A. Varol, “Openthermalpose2: Extending the opensource annotated thermal human pose dataset with more data, subjects, and poses,” IEEE Transactions on Biometrics, Behavior, and Identity Science, pp. 1–1, 2025.
[16] F. Yang, R. Li, G. Georgakis, S. Karanam, T. Chen, H. Ling, and Z. Wu, “Robust multi-modal 3d patient body modeling,” in Medical Image Computing and Computer Assisted Intervention (MICCAI). Springer, 2020, pp. 86–95.
[17] Y. Yin, J. P. Robinson, and Y. Fu, “Multimodal in-bed pose and shape estimation under the blankets,” in Proceedings of the 30th ACM International Conference on Multimedia, 2022.
[18] A. Agarwal, “Multimodal human mesh recovery for stand-off triage in mass casualty scenarios,” Master’s thesis, Carnegie Mellon University, July 2025.
[19] J. Song, X. Chen, and O. Hilliges, “Human body model fitting by learned gradient descent,” arXiv preprint arXiv:2008.08474, 2020.
[20] N. Kolotouros, G. Pavlakos, M. J. Black, and K. Daniilidis, “Learning to reconstruct 3d human pose and shape via model-fitting in the loop,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[21] M. Kocabas, N. Athanasiou, and M. J. Black, “Vibe: Video inference for human body pose and shape estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[22] M. Kocabas, C.-H. P. Huang, O. Hilliges, and M. J. Black, “Pare: Part attention regressor for 3d human body estimation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[23] Y. Wang, Y. Sun, P. Patel, K. Daniilidis, M. J. Black, and M. Kocabas, “Prompthmr: Promptable human mesh recovery,” in Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), June 2025, pp. 1148–1159.
[24] Z. Wu, F. Xie, Y. Fang, Z. Liang, Q. Wan, Y. Xiong, and X. Cai, “Seeing through the tactile: 3d human shape estimation from temporal in-bed pressure images,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 8, no. 2, pp. 1–39, 2024.
[25] Z. Wu, Y. Xiong, M. Niu, F. Xie, Q. Wan, Q. Ying, B. Liu, and X. Cai, “Pi-hmr: Towards robust in-bed temporal human shape reconstruction with contact pressure sensing,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 27 739–27 749.
[26] Q. Wan, Z. Wu, F. Xie, M. Niu, and X. Cai, “In-bed pressure image-supported diffusion for 3d human mesh recovery,” in Proceedings of the IEEE International Conference on Pervasive Computing and Communications (PerCom), 2025, pp. 89–98.
[27] J. Gao, C. Zheng, L. A. Jeni, and Z. Erickson, “Disrt-in-bed: Diffusion-based sim-to-real transfer framework for in-bed human mesh recovery,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 1829–1838.
[28] Y. Wang, X. Chen, L. Cao, W. Huang, F. Sun, and Y. Wang, “Multimodal token fusion for vision transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[29] G. Li, Z. Liu, L. Ye, Y. Wang, and H. Ling, “Cross-modal weighting network for RGB-D salient object detection,” arXiv preprint arXiv:2007.04901, 2020.
[30] S. Chen, C. Ge, Z. Tong, J. Wang, Y. Song, J. Wang, and P. Luo, “AdaptFormer: Adapting vision transformers for scalable visual recognition,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 35, 2022.
[31] J. Zhu, S. Lai, X. Chen, D. Wang, and H. Lu, “Visual prompt multi-modal tracking,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 9516–9526.
[32] B. Cao, J. Guo, P. Zhu, and Q. Hu, “Bi-directional adapter for multimodal tracking,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 2, 2024, pp. 927–935.
[33] A. Chen, X. Wang, Z. Xu, K. Shi, Y. Qin, Y. Huo, J. Chen, and Q. Ye, “Adaptivefusion: Adaptive multi-modal multi-view fusion for 3d human body reconstruction,” arXiv preprint arXiv:2409.04851, 2024.
[34] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter-efficient transfer learning for NLP,” in Proceedings of the 36th International Conference on Machine Learning (ICML), vol. 97. PMLR, 2019, pp. 2790–2799.
[35] H. Joo, N. Neverova, and A. Vedaldi, “Exemplar fine-tuning for 3d human model fitting towards in-the-wild 3d human pose estimation,” in 2021 International Conference on 3D Vision (3DV), 2021, pp. 42–52.
[36] I. Kissos, L. Fritz, M. Goldman, O. Meir, E. Oks, and M. Kliger, “Beyond weak perspective for monocular 3d human pose estimation,” arXiv preprint arXiv:2009.06549, 2020.
[37] Y. Nie, M. Fan, C. Long, Q. Zhang, J. Zhu, and X. Xu, “Incorporating test-time optimization into training with dual networks for human mesh recovery,” arXiv preprint arXiv:2401.14121, 2024.
[38] S. Geman and D. E. McClure, “Statistical methods for tomographic image reconstruction,” Bulletin of the International Statistical Institute, vol. 52, pp. 5–21, 1987.