無人飛行載具因其廣泛的應用領域且不受地面障礙限制，近年來在下世代行動網路環境中無人飛行載具輔助行動網路通訊已成為熱門的研究議題，基於無人飛行載具延伸地面基地台的訊號作為空中基地台，提供使用者更好的訊號品質以提高網路效能。本論文探究無人飛行載具軌跡優化的資料卸載問題，為了同時滿足處理時間和資料即時性的需求，提出並定義一種新的量測指標名為週轉度權重。週轉度權重共同考慮用戶的週轉時間、請求資料的即時性以及請求資料未完成服務之比例，並提出一種名為「查找最大週轉度權重用戶」的機制，用於改善具有最大週轉度權重的用戶。此外，本論文基於深度強化學習之演員與評論家演算法，提出了名為基於演員與評論家的週轉度權重軌跡優化演算法，用於解決無人飛行載具輔助行動通訊之軌跡優化問題。模擬結果顯示，相較於其他演算法，本論文提出的演員與評論家的週轉度權重軌跡優化演算法可以最小化請求高即時性資料用戶的資料剩餘比例和週轉時間。

Because of its various applications and unrestricted of terrestrial constructions, unmanned aerial vehicle (UAV) assisted mobile communication becomes a vital issue in next generation of mobile networks. On the basis of extending terrestrial base station’s signal by UAVs as aerial base stations, mobile users are provided with better signal quality and network performance can be improved. In this thesis, the data offloading problem in UAV trajectory optimization is investigated. To deal with the turnaround time and requesting data immediacy at the same time, a novel performance metric named weighted turnaround time (WTT) is proposed and defined. The WTT jointly considers the user’s turnaround time, the immediacy of requested data, and the residual portion of data requests. A mechanism called Find-Max-WTT-User is proposed to serve the user with the largest WTT and to eliminate its WTT. In addition, on the basis of Actor-Critic (AC) method in deep reinforcement learning algorithms, the Actor-Critic based WTT Trajectory Optimization (ACWTO) algorithm is proposed to solve the trajectory optimization problem of UAV-assisted mobile networks. Simulation results showed that the proposed ACWTO algorithm can minimize high immediacy users’ residual data portion and turnaround time with compared to other algorithms.

[1] E. Choi and S. Chang, “A consumer tracking estimator for vehicles in GPS-free environments,” IEEE Trans. Consum. Electron., vol. 63, no. 4, pp. 450-458, Nov. 2017.
[2] P.-V. Mekikis, A. Antonopoulos, E. Kartsakli, L. Alonso, and C. Verikoukis, “Communication recovery with emergency aerial networks,” IEEE Trans. Consum. Electron., vol. 63, no. 3, pp. 291-299, Aug. 2017.
[3] L.-H. Wang, Y.-M. Hsiao, X.-Q. Xie, and S.-Y. Lee, “An outdoor intelligent healthcare monitoring device for the elderly,” IEEE Trans. Consum. Electron., vol. 62, no. 2, pp. 128-135, May 2016.
[4] S. Wang, Y. Han, J. Chen, Z. Zhang, G. Wang, N. Du, “A Deep-learning-based sea search and rescue algorithm by UAV remote sensing,” in Proc. IEEE CSAA Guidance, Navigation and Control Conference (CGNCC), Aug 2018.
[5] R. Han, Y. Wen, L. Bai, J. Liu, and J. Choi, “Age of information aware UAV deployment for intelligent transportation systems,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 3, pp. 2705-2715, Mar. 2022.
[6] L. A. b. Burhanuddin, X. Liu, Y. Deng, U. Challita, A. Zahemszky, “QoE optimization for live video streaming in UAV-to-UAV communications via deep reinforcement learning,” IEEE Trans. Veh. Technol., vol. 71, no. 5, pp. 5358-5370, May 2022.
[7] J. Bai, S. Lian, Z. Liu, K. Wang, and D. Liu, “Deep learning based robot for automatically picking up garbage on the grass,” IEEE Trans. Consum. Electron., vol. 64, no. 3, pp. 382-389, Jul. 2018
[8] S. Vashist and Sushma Jain, “Location-aware network of drones for consumer applications: supporting efficient management between multiple drones,” IEEE Consum. Electron. Magazine, vol. 8, no. 3, pp. 68-73, May 2019.
[9] Y. Zeng, R. Zhang and T.J. Lim, “Wireless communications with unmanned aerial vehicles: opportunities and challenges,” IEEE Commun. Mag., vol. 54, no. 5, pp. 36-42, May 2016.
[10] S. Ahmed, M. Z. Chowdhury, and Y. M. Jang, “Energy-efficient UAV to-user scheduling to maximize throughput in wireless networks,” IEEE Access, vol. 8, pp. 21215-21225, 2020.
[11] Y. Xu, T. Zhang, J. Loo, D. Yang, and L. Xiao, “Completion time minimization for UAV-assisted mobile-edge computing systems,” IEEE Transactions on Vehicular Technology, vol. 70, no. 11, pp. 12 253– 12 259, 2021.
[12] J. Liu, L. Li, F. Yang, X. Liu, X. Li, X. Tang, and Z. Han, “Minimization of offloading delay for two-tier uav with mobile edge computing,” in 2019 15th International Wireless Communications Mobile Computing Conference (IWCMC), 2019, pp. 1534–1538.
[13] Y. Jia et al., “The UAV Path Coverage Algorithm Based on the Greedy Strategy and Ant Colony Optimization,” Electronics, vol. 11, no. 17, p. 2667, Aug. 2022, doi: 10.3390/electronics11172667.
[14] R. Shivgan and Z. Dong, “Energy-efficient drone coverage path planning using genetic algorithm,” in 2020 IEEE 21st International Conference on High Performance Switching and Routing (HPSR). IEEE, 2020, pp. 1–6.
[15] M. I. Jordan and T. M. Mitchell, “Machine learning: trends perspectives and prospects,” Science, vol. 349, no. 6245, pp. 255-260, 2015.
[16] V. Martin, J. Cabrera, and N. García, “Design, optimization and evaluation of a q-learning HTTP adaptive streaming client,” IEEE Trans. Consum. Electron., vol. 62, no. 4, pp. 380–388, Nov. 2016.
[17] S. K. Zaidi, S. F. Hasan, X. Gui, N. Siddique, and S. Ahmad, “Exploiting UAV as NOMA based relay for coverage extension,” in Proc. 2nd Int. Conf. Comput. Appl. Inf. Secur. (ICCAIS), Riyadh, Saudi Arabia, May. 2019, pp. 1-5.
[18] X. Zhong, Y. Guo, N. Li and Y. Chen, “Joint optimization of relay deployment channel allocation and relay assignment for UAVs-aided D2D networks,” IEEE/ACM Trans. Netw., vol. 28, no. 2, pp. 804-817, Apr. 2020.
[19] T. Zhang, Z. Wang, Y. Liu, W. Xu and A. Nallanathan, “Caching placement and resource allocation for cache-enabling UAV NOMA networks,” IEEE Trans. Veh. Technol., pp. 12897-12911, 2020.
[20] F. Zhou, N. Wang, G. Luo, L. Fan and W. Chen, “Edge caching in multi-UAV-enabled radio access networks: 3D modeling and spectral efficiency optimization,” IEEE Trans. Signal Inf. Process. Netw., vol. 6, pp. 329-341, Apr. 2020.
[21] G. Li, C. Zhuang, Q. Wang, Y. Li, X. Xu, and W. Zhou, “A UAV realtime trajectory optimized strategy for moving users,” in Proc. 11th Int. Conf. Wireless Commun. Signal Process. (WCSP), Oct. 2019, pp. 1–6.
[22] X. Jing and C. Masouros, “UAV trajectory design and bandwidth allocation for coverage maximization with energy and time constraints,” in Proc. IEEE Int. Symp. Pers. Indoor Mobile Radio Commun., Sep. 2020, pp. 1–6.
[23] F. Zhou, Y. Wu, H. Sun, and Z. Chu, “UAV-enabled mobile edge computing: Offloading optimization and trajectory design,” in Proc. IEEE Int. Conf. Commun. (ICC), May 2018, pp. 1–6
[24] Y. Wang, W. Fang, Y. Ding, and N. Xiong, “Computation offloading optimization for UAV-assisted mobile edge computing: a deep deterministic policy gradient approach,” Wireless Networks, vol. 27, no. 4, pp. 2991–3006, 2021.
[25] Z. Qin, A. Li, C. Dong, H. Dai, and Z. Xu, “Completion Time Minimization for Multi-UAV Information Collection via Trajectory Planning,” Sensors, vol. 19, no. 18, p. 4032, Sep. 2019, doi: 10.3390/s19184032.
[26] M. A. Abd-Elmagid, A. Ferdowsi, H. S. Dhillon, and W. Saad, “Deep reinforcement learning for minimizing age-of-information in UAV-assisted networks,” in 2019 IEEE Global Communications Conference (GLOBECOM), 2019, pp. 1–6.
[27] F. Cheng, S. Zhang, Z. Li, Y. Chen, N. Zhao, F. R. Yu, and V. C. M. Leung, “UAV trajectory optimization for data offloading at the edge of multiple cells,” IEEE Trans. Veh. Technol., vol. 67, no. 7, pp. 6732– 6736, Jul. 2018.
[28] Y. Zeng and R. Zhang, “Energy-efficient UAV communication with trajectory optimization,” IEEE Trans. Wireless Commun., vol. 16, no. 6, pp. 3747–3760, Jun. 2017.
[29] D. Han, H. Peng, H. Wu, W. Liao, and X. Shen, “Joint cache placement and content delivery in satellite-terrestrial integrated C-RANs,” in Proc. IEEE Int. Conf. Commun., 2021, pp. 1–6.
[30] Q. Wu, Y. Zeng, and R. Zhang, “Joint trajectory and communication design for multi-UAV enabled wireless networks,” IEEE Trans. Wirel. Commun., vol. 17, no. 3, pp. 2109-2121, Mar. 2018.
[31] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing Atari with deep reinforcement learning,” 2013, arXiv:1312.5602.
[32] H. van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” CoRR, vol. abs/1509.06461, 2015.
[33] Z. Wang, T. Schaul, M. Hessel, H. V. Hasselt, M. Lanctot, and N. De Freitas. 2015. Dueling network architectures for deep reinforcement learning. arXiv preprint arXiv:1511.06581 (2015).
[34] C. Zhou, H. He, P. Yang, F. Lyu, W. Wu, N. Cheng, and X. Shen, “Deep RL-based trajectory planning for AoI minimization in UAV-assisted IoT”, in Proc. 11th Int. Conf. Wireless Commun. Signal Process. (WCSP), Oct. 2019.
[35] D. Noh, W. Lee, H. Kim, I. Cho, I. Shim, S. Baek, “Adaptive Coverage Path Planning Policy for a Cleaning Robot with Deep Reinforcement Learning,” 2022 IEEE International Conference on Consumer Electronics (ICCE), Jan. 2022.
[36] Z. Chen, Y. Zhong, X. Ge, Y. Ma, “An actor-critic-based UAV-BSs deployment method for dynamic environments,” in Proc. 2020 IEEE International Conference on Communications (ICC), Jun. 2020.
[37] G. S. Rahman, T. Dang, and M. Ahmed, “Deep reinforcement learning based computation offloading and resource allocation for low-latency fog radio access networks,” Intell. Converged Netw., vol. 1, no. 3, pp. 243–257, 2020.
[38] L. Wang, K. Wang, C. Pan, W. Xu, N. Aslam and L. Hanzo, “Multiagent deep reinforcement learning based trajectory planning for multi-UAV assisted mobile edge computing,” IEEE Trans. Cogn. Commun. Netw., vol. 7, no. 1, pp. 73-84, Mar. 2021
[39] H. He, S. Zhang, Y. Zeng, and R. Zhang, “Joint altitude and beamwidth optimization for UAV-enabled multiuser communications,” IEEE Commun. Lett., vol. 22, no. 2, pp. 344–347, Feb. 2018.
[40] 3GPP TR38.900, “Study on Channel Model for Frequency Spectrum above 6 GHz,” v14.2.0, 2016.