在運動影片中追蹤並重建球員的三維位置是一項極具挑戰的任務，主要的困難點來自球員的快速移動、不規則運動以及嚴重的遮擋問題。本研究提出一種跨視角關聯方法，透過多視角影片來進行多球員追蹤，並構建球員的 3D 軌跡。為了解決嚴重的遮擋問題，我們同步分析來自多個視角的影片，並利用圖（graph）來描述不同視角間的偵測關聯問題。再透過最大流最小割（maximum-flow-minimum-cut）演算法來解決關聯衝突。在確定視角間隨時間變化的關聯後，再重建球員的 3D 軌跡。透過在排球比賽資料集上的評估，我們證明所提出的方法在 2D 追蹤指標（HOTA、MOTA 和 IDF1）方面達到最領先的表現。本研究的主要優勢是基於多視角影片的 3D 追蹤，因此我們進一步擴展排球比賽資料集，利用資料集當中的 2D 追蹤標記資料與相機參數進行 3D 軌跡的重建，用以評估 3D 追蹤結果的性能。同時我們也將 2D 追蹤的指標擴展成 3D 的版本（3D-HOTA、3D-MOTA 和 3D-IDF1）以評估更全面的 3D 追蹤性能，並首次展示了多視角運動影片中卓越的 3D 追蹤結果。

Tracking and reconstructing 3-dimensional positions of players in sports videos is a very challenging task due to rapid motion, irregular movement, and severe occlusion. In this work, we present a cross-view association method to achieve multi-player tracking based on multi-view videos and then construct 3D trajectories of players. To tackle the severe occlusion problem, we jointly analyze videos synchronously captured from multiple views. The associations between detections in multiple views are described by a graph, and a maximum-flow-minimum-cut algorithm resolves the conflict association problem. After finding the associations across views over time, we construct 3D trajectories of players. By evaluating the proposed method on a volleyball video dataset, we show the proposed method achieves the SOTA tracking performance in terms of conventional 2D metrics, i.e., HOTA, MOTA, and IDF1. The main advantage of our work is 3D tracking based on multi-view videos. Therefore, we further extend the volleyball video dataset by reconstructing 3D trajectories from the annotated 2D tracking data and camera parameters, which allows us to evaluate the performance of 3D tracking. We also extend these metrics to 3D versions, i.e., 3D-HOTA, 3D-MOTA, and 3D-IDF1, and firstly demonstrate very promising 3D tracking performance for multi-view sports videos.

[1] Nir Aharon, Roy Orfaig, and Ben-Zion Bobrovsky. Bot-sort: Robust associations multi-pedestrian tracking. In arXiv:2206.14651, 2022.
[2] Hong-Yuan Mark Liao Alexey Bochkovskiy, Chien-Yao Wang. Yolov4: Optimal speed and accuracy of object detection. In arXiv:2004.10934, 2020.
[3] Anton Andriyenko and Konrad Schindler. Globally optimal multi-target tracking on a hexagonal lattice. In Proceedings of European Conference on Computer Vision, pages 466–479, 2010.
[4] Sermetcan Baysal and Pinar Duygulu. Sentioscope: A soccer player tracking system using model field particles. IEEE Transactions on Circuits and Systems for Video Technology, 26(7):1350–1362, 2016.
[5] Jerome Berclaz, Francois Fleuret, Engin Turetken, and Pascal Fua. Multiple object tracking using k-shortest paths optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(9):1806–1819, 2011.
[6] Keni Bernardin and Rainer Stiefelhagen. Evaluating multiple object tracking performance: The clear mot metrics. EURASIP Journal on Image and Video Processing, 2008.
[7] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In Proceedings of IEEE International Conference on Image Processing, pages 3464–3468, 2016.
[8] Guillem Braso and Laura Leal-Taixe. Learning a neural solver for multiple object tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[9] Jinkun Cao, Jiangmiao Pang, Xinshuo Weng, Rawal Khirodkar, and Kris Kitani. Observation-centric sort: Rethinking sort for robust multi-object tracking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 9686–9696, 2023.
[10] Anthony Cioppa, Adrien Deliege, Silvio Giancola, Bernard Ghanem, and Marc Van Droogenbroeck. Scaling up soccernet with multi-view spatial localization and reidentification. Scientific Data, 9:355, 2022.
[11] Anthony Cioppa, Silvio Giancola, Adrien Deliege, Le Kang, Xin Zhou, Zhiyu Cheng, Bernard Ghanem, and Marc Van Droogenbroeck. Soccernet-tracking: Multiple object tracking dataset and benchmark in soccer videos. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 3491–3502, 2022.
[12] Robert T. Collins and Peter Carr. Hybrid stochastic / deterministic optimization for tracking sports players and pedestrians. In Proceedings of European Conference on Computer Vision, pages 298–313, 2014.
[13] Yunhao Du, Zhicheng Zhao, Yang Song, Yanyun Zhao, Fei Su, and Tao Gong. Strongsort: Make deepsort great again. IEEE Transactions on Multimedia, 25:8725–8737, 2023.
[14] Martin Engilberge, Weizhe Liu, and Pascal Fua. Multi-view tracking using weakly supervised human motion prediction. In Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision, 2023.
[15] Pedro Felzenszwalb, David McAllester, and Deva Ramanan. A discriminatively trained, multiscale, deformable part model. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2008.
[16] Francois Fleuret, Jerome Berclaz, Richard Lengagne, and Pascal Fua. Multicamera people tracking with a probabilistic occupancy map. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2):267–282, 2008.
[17] Yiyang Gan, Ruize Han, Liqiang Yin, Wei Feng, and Song Wang. Self-supervised multi-view multi-human association and tracking. In Proceedings of ACM International Conference on Multimedia, pages 282–290, 2021.
[18] Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. In arXiv:2107.08430, 2021.
[19] Navneet S. Ghedia, C.H. Vithalani, and Ashish Kothari. A novel approach for monocular 3d object tracking in cluttered environment. International Journal of Computational Intelligence Research, 13(5):851–864, 2017.
[20] Andrew V. Goldberg and Robert E. Tarjan. A new approach to the maximum-flow problem. Journal of the ACM, 35(4):921–940, 1988.
[21] Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, 2003.
[22] Min-Chun Hu, Ming-Hsiu Chang, Ja-Ling Wu, and Lin Chi. Robust camera calibration and player tracking in broadcast basketball video. IEEE Transactions on Multimedia, 13(2):266–279, 2011.
[23] Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics yolo, January 2023.
[24] Stephanie A. Kovalchik. Player tracking data in sports. Annual Review of Statistics and Its Application, 10:677–697, 2023.
[25] Harold William Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83–97, 1955.
[26] Xiaobai Liu, Yuanlu Xu, Lei Zhu, and Yadong Mu. A stochastic attribute grammar for robust cross-view human tracking. IEEE Transactions on Circuits and Systems for Video Technology, 28(10):2884–2895, 2018.
[27] Wei-Lwun Lu, Jo-Anne Ting, James J. Little, and Kevin P. Murphy. Learning to track and identify players from broadcast sports videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(7):1704–1716, 2013.
[28] Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixe, and Bastian Leibe. Hota: A higher order metric for evaluating multi-object tracking. International Journal of Computer Vision, 129:548–578, 2021.
[29] Weiyi Lv, Yuhang Huang, Ning Zhang, Ruei-Sung Lin, Mei Han, and Dan Zeng. Diffmot: A real-time diffusion-based multiple object tracker with non-linear prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 19321–19330, 2024.
[30] Adrien Maglo, Astrid Orcesi, and Quoc-Cuong Pham. Efficient tracking of team sport players with few game-specific annotations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 3461–3471, 2022.
[31] Amir M. Mansourian, Vladimir Somers, Christophe De Vleeschouwer, and Shohreh Kasaei. Multi-task learning for joint re-identification, team affiliation, and role classification for sports visual tracking. In arXiv:2401.09942, 2024.
[32] Banoth Thulasya Naik, Mohammad Farukh Hashmi, Zong Woo Geem, and Neeraj Dhanraj Bokde. Deepplayer-track: Player and referee tracking with jersey color recognition in soccer. IEEE Access, 10:32494–32509, 2022.
[33] Yuri Nishikawa, Hitoshi Sato, and Jun Ozawa. Multiple sports player tracking system based on graph optimization using low-cost cameras. In Proceedings of IEEE International Conference on Consumer Electronics, 2018.
[34] Horst Possegger, Sabine Sternig, Thomas Mauthner, Peter M. Roth, and Horst Bischof. Robust real-time tracking of multiple objects by volumetric mass densities. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2013.
[35] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. Performance measures and a data set for multi-target, multi-camera tracking. In Proceedings of European Conference on Computer Vision Workshops, 2016.
[36] Taiki Sekii. Robust, real-time 3d tracking of multiple objects with similar appearances. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[37] Zheng Tang, Renshu Gu, and Jenq-Neng Hwang. Joint multi-view people tracking and pose estimation for 3d scene reconstruction. In Proceedings of IEEE International Conference on Multimedia and Expo, 2018.
[38] Ultralytics. Yolov8. In https://github.com/ultralytics/ultralytics, 2023.
[39] Kanav Vats, Pascale Walters, Mehrnaz Fani, David A. Clausi, and John S. Zelek. Player tracking and identification in ice hockey. Expert Systems With Applications, 213:119250, 2023.
[40] Christophe De Vleeschouwer, Fan Chen, Damien Delannay, Christophe Parisot, Christophe Chaudy, Eric Martrou, and Andrea Cavallaro. Distributed video acquisition and annotation for sport-event summarization. In Proceedings of NEM Summit, 2008.
[41] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[42] Yi-Peng Wang and Wei-Ta Chu. Multiple player tracking with 3d projection and spatio-temporal information in multi-view sports videos. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, 2024.
[43] Yu-Hsiang Wang, Jun-Wei Hsieh, Ping-Yang Chen, Ming-Ching Chang, Hung Hin So, and Xin Li. Smiletrack: Similarity learning for occlusion-aware multiple object tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 5740–5748, 2024.
[44] Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with a deep association metric. In Proceedings of IEEE International Conference on Image Processing, pages 3645–3649, 2017.
[45] Cheng-Wen Wu, Ming-Der Shieh, Jenn-Jier James Lien, Jar-Ferr Yang, Wei-Ta Chu, Tsang-Hai Huang, Han-Chuan Hsieh, Hung-Ta Chiu, Kuo-Cheng Tu, Yen-Ting Chen, Shian-Yu Lin, Jia-Jun Hu, Chen-Huan Lin, and Cheng-Siang Jheng. Enhancing fan engagement in a 5g stadium with ai-based technologies and live streaming. IEEE Systems Journal, 16(4):6590–6601, 2022.
[46] Junliang Xing, Haizhou Ai, Liwei Liu, and Shihong Lao. Multiple player tracking in sports video: A dual-mode two-way bayesian inference approach with progressive observation modeling. IEEE Transactions on Image Processing, 20(6):1652–1667, 2011.
[47] Yuanlu Xu, Xiaobai Liu, Yang Liu, and Song-Chun Zhu. Multi-view people tracking via hierarchical trajectory composition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[48] Yuanlu Xu, Xiaobai Liu, Lei Qin, and Song-Chun Zhu. Cross-view people tracking by scene-centered spatio-temporal parsing. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4299–4305, 2017.
[49] Bin Yan, Yi Jiang, Peize Sun, Dong Wang, Zehuan Yuan, Ping Luo, and Huchuan Lui. Towards grand unification of object tracking. In Proceedings of European Conference on Computer Vision, pages 9686–9696, 2022.
[50] Ruiheng Zhang, Lingxiang Wu, Yukun Yang, Wanneng Wu, Yueqiang Chen, and Min Xu. Multi-camera multi-player tracking with deep player identification in sports video. Pattern Recognition, 102, 2020.
[51] Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. Bytetrack: Multi-object tracking by associating every detection box. In Proceedings of European Conference on Computer Vision, 2022.
[52] Yifu Zhang, Chunyu Wang, Xinggang Wang, Wenjun Zeng, and Wenyu Liu. Fairmot: On the fairness of detection and re-identification in multiple object tracking. International Journal of Computer Vision, 129:3069–3087, 2021.