隨著科技的進步，實境系統也漸漸普及在我們生活中，目前主要的實境可以分為VR/MR/AR這三類，而本論文的遊戲系統則是以MR為基礎所設計。在MR環境下，玩家能透過頭盔進行場景建模，結合現實世界與虛擬世界，玩家不僅可以感受到虛擬物件的回饋，也可以與真實世界場景互動。目前MR的多人互動遊戲中，大多是讓玩家們共享同一個空間環境，因此當每位玩家對空間中的虛擬物件進行操作時，其他玩家都能共享操作的結果，以達到多人互動的效果。但在這些互動遊戲中仍然有不足的地方，例如玩家之間缺乏直接的體感互動。在我們提出的多人體感互動遊戲架構中，讓玩家能藉由自己身體的姿勢，與其他玩家進行直接的體感互動，大幅增加遊戲性以及趣味性。此外我們提出一個簡易近似估計3D 位置的方法，大幅降低系統運算效能，讓遊戲特效能即時的渲染在 MR 3D 空間中正確的位置上，大幅增加了玩家對此遊戲的體驗。我們提出的系統架構結合了骨架預測模型以及動作辨識模型來達到強體感的效果，其中我們設計了多個動作並進行資料的蒐集，最後成功訓練深度學習末型，以驗證整個系統的可靠性與效能。

Mixed reality (MR) is getting popular, but its application in entertainment is still limited due to the lack of intuitive and various interactions between the user and other players. In this demonstration, we propose an MR multiplayer game framework, which allows the player to interact directly with other players through intuitive body postures/actions. Moreover, a body depth approximation method is designed to decrease the complexity of virtual content rendering without affecting the immersive fidelity while playing the game. Our framework uses deep learning models to achieve motion sensing, and a multiplayer MR interaction game containing a variety of actions is designed to validate the feasibility of the proposed framework.

[1] Microsoft kinect. https://developer.microsoft.com/zh-tw/windows/kinect, 2010.
[2] Kinect video game - powerup heroes. https://www.youtube.com/watch?v=XrtvfUXVxKU, 2011.
[3] Microsoft hololens. https://www.microsoft.com/en-us/p/microsoft-hololens-development-edition/8xf18pqz17ts?activetab=pivot:overviewtab , 2015.
[4] 10 vr multiplayer games you must play. https://youtu.be/_heWEHzzN2E, 2016.
[5] Real world multiplayer ar demo. https://www.youtube.com/watch?v=dO1NpT2SSX4, 2018.
[6] World’s first mixed reality multiplayer game based on leap motion. https://www.youtube.com/watch?v=qpy31DVeERw, 2018.
[7] Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh. Openpose: real-time multi-person 2d pose estimation using part affinity fields. arXiv preprint arXiv:1812.08008, 2018.
[8] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venu-gopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2625–2634, 2015.
[9] H.-S. Fang, S. Xie, Y.-W. Tai, and C. Lu. Rmpe: Regional multi-person pose estimation. In Proceedings of the IEEE International Conference on Computer Vision, pages 2334–2343, 2017.
[10] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
[11] I. Kim. tf-pose-estimation. https://github.com/ildoonet/tf-pose-estimation, 2017.
[12] A. Newell, K. Yang, and J. Deng. Stacked hourglass networks for human pose estimation. In European conference on computer vision, pages 483–499. Springer, 2016.
[13] G. Papandreou, T. Zhu, N. Kanazawa, A. Toshev, J. Tompson, C. Bregler, and K. Murphy. Towards accurate multi-person pose estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4903–4911, 2017.
[14] L. Pishchulin, E. Insafutdinov, S. Tang, B. Andres, M. Andriluka, P. V. Gehler, and B. Schiele. Deepcut: Joint subset partition and labeling for multi person pose estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4929–4937, 2016.
[15] Z. Qiu, T. Yao, and T. Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In proceedings of the IEEE International Conference on Computer Vision, pages 5533–5541, 2017.
[16] A. Toshev and C. Szegedy. Deeppose: Human pose estimation via deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1653–1660, 2014.
[17] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatio-temporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 4489–4497, 2015.
[18] H. Wang, A. Kläser, C. Schmid, and L. Cheng-Lin. Action recognition by dense trajectories. 2011.
[19] H. Wang and C. Schmid. Action recognition with improved trajectories. In Proceedings of the IEEE international conference on computer vision, pages 3551–3558, 2013.
[20] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pages 20–36. Springer, 2016.
[21] S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh. Convolutional pose machines. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4724–4732, 2016.
[22] S. Yan, Y. Xiong, and D. Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[23] Y. Yang and D. Ramanan. Articulated human detection with flexible mixtures of parts. IEEE transactions on pattern analysis and machine intelligence, 35(12):2878–2890, 2012.
[24] J. Yue-Hei Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici. Beyond short snippets: Deep networks for video classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4694–4702, 2015.