無人機在現代戰場戰術應用相當廣泛，在偵蒐及打擊等層面都具有便利、性價比高等優點，搭載人工智慧的無人機系統更是未來戰爭中不可或缺的部份。然而無人機不僅可以作為進攻手段，亦可配合、搭載不同感測器作為防禦性武器使用，本文第一部份利用虛幻引擎以及Airsim的物理模型搭建任務場景，以深度強化學習網路SAC（soft-actor-critic）演算法各別訓練進攻用無人機以及防守用無人機，使其在相互競爭中學習動作策略。第二部分將深度強化學習結合視覺辨識方法，利用影像以及深度感測器數值資料訓練深度強化學習網路SAC，使其能夠自動追蹤敵方無人機，並且鎖定、撞擊以達成防禦任務。

This purpose of this paper focus on the integration of deep reinforcement learning (DRL) in one-on-one unmanned aerial vehicle (UAV) combat scenarios. We introduce a Soft Actor-Critic (SAC) framework for UAVs to learn tracking, evasion, and circumvention strategies. This equips UAVs to effectively engage in adversarial encounters. This paper also presents a novel approach merging image recognition and DRL for UAV pursuit and guidance. By utilizing image and depth sensors, the UAV calculates relative positions and employs DRL to pursue and intercept enemy drones, achieving a commendable interception rate. Addressing reward function design, this paper emphasizes calibrated incentives. By refining overly optimistic reward functions, we achieve a significant increase in draw ratios and a more balanced attack-defense win ratio.

Alsanad, H., Sadik, A., Ucan, O., Ilyas, M., & Bayat, O. (2022). YOLO-V3 based real-time drone detection algorithm. Multimedia Tools and Applications, 81, 1-14. https://doi.org/10.1007/s11042-022-12939-4
Guo, H. F., Hou, M. Y., Zhang, Q. J., & Tang, C. L. (2017). UCAV robust maneuver decision based on statistics principle. Binggong Xuebao/Acta Armamentarii, 38, 160-167. https://doi.org/10.3969/j.issn.1000-1093.2017.01.021
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., & Abbeel, P. (2018). Soft actor-critic algorithms and applications.
Kurniawan, B., Vamplew, P., Papasimeon, M., Dazeley, R., & Foale, C. (2019). An empirical study of reward structures for actor-critic reinforcement learning in air combat manoeuvring simulation. AI 2019: Advances in Artificial Intelligence: 32nd Australasian Joint Conference, Adelaide, SA, Australia, December 2–5, 2019, Proceedings 32,
Lei, X., Dali, D., Zhenglei, W., Zhifei, X., & Andi, T. (2021). Moving time UCAV maneuver decision based on the dynamic relational weight algorithm and trajectory prediction. Mathematical Problems in Engineering, 2021, 1-19.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015). Continuous control with deep reinforcement learning.
Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017). Feature pyramid networks for object detection. Proceedings of the IEEE conference on computer vision and pattern recognition,
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition,
Redmon, J., & Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv
Shah, S., Dey, D., Lovett, C., & Kapoor, A. (2018). Airsim: High-fidelity visual and physical simulation for autonomous vehicles. Field and Service Robotics: Results of the 11th International Conference,
Shah, S., Madaan, R., Vemprala, S., & Gyde, N. (2020). Airsim. Retrieved 5/13 from https://microsoft.github.io/AirSim/
Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press.
tusharsarkar3. (2021). Detect_Drone. Retrieved 7/15 from https://github.com/tusharsarkar3/Detect_Drone
Watkins, C. J., & Dayan, P. (1992). Q-learning. Machine Learning, 8(3), 279-292. https://doi.org/10.1007/BF00992698
Yang, Q., Zhu, Y., Zhang, J., Qiao, S., & Liu, J. (2019). UAV air combat autonomous maneuver decision based on DDPG algorithm. 2019 IEEE 15th international conference on control and automation (ICCA),
Zheng, J., Ma, Q., Yang, S., Wang, S., Liang, Y., & Ma, J. (2020). Research on cooperative operation of air combat based on multi-agent. Human Interaction, Emerging Technologies and Future Applications II: Proceedings of the 2nd International Conference on Human Interaction and Emerging Technologies: Future Applications (IHIET–AI 2020), April 23-25, 2020, Lausanne, Switzerland,
康曉嵐. (2021). 無人飛行載具在防衛作戰運用的探討. 戰略與評估, 第十一卷第一期.
陳津萍. (2021). 中國大陸無人機「集群作戰」發展之研究. 空軍學術雙月刊, 第680期.