雲端運算基礎設施面臨日益嚴峻的能源效率挑戰，其中資料中心已佔全球電力消耗的 1–2%。在這樣的多層級環境中進行作業排程本身就相當複雜，需跨資料中心、伺服器節點與應用容器進行協調。現有的深度強化學習方法通常針對每一層分別建立獨立模型，導致資訊無法共享、重複計算增加，並缺乏有效的跨層協調。
為了解決這些問題，我們提出MDPT4JS-HA，一種新型強化學習架構，具備 Transformer 編碼器與階層式輔助預測頭。利用編碼器整合所有層級的全局資訊，而輔助預測頭則預測不同層級間的動作，強化層級間的合作能力。
我們在 Alibaba Cluster Trace V2018 資料集上進行實驗，結果顯示：與傳統啟發式方法及現有 DRL 基準相比，MDPT4JS-HA 的能源消耗最多可降低 15%，同時無論是在 1000、3000 或 5000 個容器的場景中皆能維持良好效能。

Cloud computing infrastructure faces growing energy efficiency challenges, with data centers accounting for 1–2% of global electricity consumption. Multi-tier job scheduling in such environments is inherently complex, requiring coordination across data centers, server nodes, and application containers. Existing DRL-based methods often deploy independent models for each layer, which hinders information sharing and results in redundant computations and weak cross-layer coordination.

To address these limitations, we propose MDPT4JS-HA, a novel reinforcement learning framework featuring a unified transformer encoder and hierarchical auxiliary prediction heads. The unified encoder captures global representations across all layers, while auxiliary heads predict inter-level actions, enhancing learning stability and coordination.

Experiments on the Alibaba Cluster Trace V2018 dataset demonstrate that MDPT4JS-HA achieves up to 15% lower energy consumption and improved scheduling stability compared to traditional heuristics and existing DRL baselines, while maintaining performance scalability across 1000 to 5000 container scenarios.

[1] Yan Gu, Zhe Liu, Shengqiang Dai, Chuanping Liu, Yuxuan Wang, Shuai Wang, Long Cheng, “Deep reinforcement learning for job scheduling and resource management in cloud computing: An algorithm-level review”, arXiv preprint arXiv:2501.01007, pp. 1–30, 2025.
[2] Mingxi Cheng, Ji Li, Shahin Nazarian, “DRL-cloud: Deep reinforcement learning-based resource provisioning and task scheduling for cloud service providers”, Proceedings of the 23rd Asia and South Pacific Design Automation Conference (ASP-DAC), pp. 129–134, 2018.
[3] Jie Zhao, Maria A. Rodríguez, Rajkumar Buyya, “A deep reinforcement learning approach to resource management in hybrid clouds harnessing renewable energy and task scheduling”, Proceedings of the 2021 IEEE 14th International Conference on Cloud Computing (CLOUD), pp. 240–249, 2021.
[4] Yanbo Xu, Jiakun Zhao, “Actor-critic with transformer for cloud computing resource three stage job scheduling”, Proceedings of the 7th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA), pp. 33–37, 2022.
[5] Zhao Tong, Hao Chen, Xiaomin Deng, Keqin Li, Kuan-Ching Li, “A scheduling scheme in the cloud computing environment using deep Q-learning”, Information Sciences, vol. 512, pp. 1170–1191, 2020.
[6] Lei Yu, Philip S. Yu, Yuxin Duan, Hong Qiao, “A resource scheduling method for reliable and trusted distributed composite services in cloud environment based on deep reinforcement learning”, Frontiers in Genetics, vol. 13, art. no. 964784, pp. 1–15, 2022.
[7] Yongyi Ran, Haojie Hu, Xiaokang Zhou, Yonggang Wen, “Deepee: Joint optimization of job scheduling and cooling control for data center energy efficiency using deep reinforcement learning”, Proceedings of the 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), pp. 645–655, 2019.
[8] Jiaying Meng, Hongzhi Tan, Chuan Wu, Wenqi Cao, Lihua Liu, Baochun Li, “Dedas: Online task dispatching and scheduling with bandwidth constraint in edge computing”, Proceedings of IEEE INFOCOM 2019, pp. 2287–2295, 2019.
[9] Longyu Ran, Xiaoyu Shi, Mingsheng Shang, “SLAs-aware online task scheduling based on deep reinforcement learning method in cloud environment”, Proceedings of the 2019 IEEE 21st International Conference on High Performance Computing and Communications, pp. 1518–1525, 2019.
[10] Yaqiang Zhang, Zhi Zhou, Zhi Shi, Lihong Meng, Zili Zhang, “Online scheduling optimization for DAG-based requests through reinforcement learning in collaboration edge networks”, IEEE Access, vol. 8, pp. 72985–72996, 2020.
[11] Changyong Sun, Tan Yang, Youxun Lei, “DDDQN-TS: A task scheduling and load balancing method based on optimized deep reinforcement learning in heterogeneous computing environment”, International Journal of Intelligent Systems, vol. 37, no. 11, pp. 9138–9172, 2022.
[12] Lingxin Zhang, Jintao Ding, Ruixuan Li, Jiahui Li, Hai Jin, “Multi-task deep reinforcement learning for scalable parallel task scheduling”, Proceedings of the 2019 IEEE International Conference on Big Data, pp. 2992–3001, 2019.
[13] Yibo Zhou, Huabiao Qin, Guancheng Chen, “Intelligent task scheduling for multi-core processors based on graph neural networks and deep reinforcement learning”, Proceedings of the 4th International Conference on Artificial Intelligence and Computer Engineering, pp. 875–880, 2023.
[14] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin, “Attention is all you need”, Advances in Neural Information Processing Systems (NeurIPS), vol. 30, pp. 1–15, 2017.
[15] Yue Gao, Yanzhi Wang, Sandeep K. Gupta, Massoud Pedram, “An energy and deadline aware resource provisioning, scheduling and optimization framework for cloud systems”, Proceedings of the 2013 International Conference on Hardware/Software Codesign and System Synthesis (CODES+ISSS), pp. 1–10, 2013.
[16] Fengcun Li, Bo Hu, “DeepJS: Job scheduling based on deep reinforcement learning in cloud data center”, Proceedings of the 4th International Conference on Big Data and Computing, pp. 48–53, 2019.
[17] Jialu Zhao, Kai Xie, Hanhua Sun, Chengyuan Li, Yunchuan Zhang, “An MA-PPO based task scheduling method for computing first networks”, Proceedings of the 2022 5th International Conference on Telecommunications and Communication Engineering (ICTCE), pp. 260–265, 2022.
[18] Tiangang Li, Shaoyong Ying, Yaxing Zhao, Jie Shang, “Batch jobs load balancing scheduling in cloud computing using distributional reinforcement learning”, IEEE Transactions on Parallel and Distributed Systems, vol. 35, no. 1, pp. 169–185, 2023.
[19] Zhen Chen, Lei Zhang, Xiaofeng Wang, Kai Wang, “Cloud–edge collaboration task scheduling in cloud manufacturing: An attention-based deep reinforcement learning approach”, Computers & Industrial Engineering, vol. 177, art. no. 109053, pp. 1–17, 2023.
[20] Jingchun Li, Feng Zhou, Wei Li, Ming Zhao, Xiaoyang Yan, Yaqin Xi, Jianwu Wu, “Componentized task scheduling in cloud-edge cooperative scenarios based on GNN-enhanced DRL”, Proceedings of NOMS 2023 – IEEE/IFIP Network Operations and Management Symposium, pp. 1–4, 2023.
[21] Yi Liu, Jingwei Han, Kun Xue, Jie Li, Qibo Sun, Jianhua Lu, “DECC: Achieving low latency in data center networks with deep reinforcement learning”, IEEE Transactions on Network and Service Management, vol. 20, no. 4, pp. 4313–4324, 2023.
[22] Guangyao Zhou, Weijie Tian, Rajkumar Buyya, Rui Xue, Liyun Song, “Deep reinforcement learning-based methods for resource scheduling in cloud computing: A review and future directions”, Artificial Intelligence Review, vol. 57, no. 5, art. no. 124, pp. 1–42, 2024.
[23] Hussain Kahil, Saurabh Sharma, Petri Välisuo, Mohammed Elmusrati, “Reinforcement learning for data center energy efficiency optimization: A systematic literature review and research roadmap”, Applied Energy, vol. 389, art. no. 125734, pp. 1-27, 2025
[24] Tingting Dong, Feng Xue, Chen Xiao, Ji Li, “Task scheduling based on deep reinforcement learning in a cloud manufacturing environment”, Concurrency and Computation: Practice and Experience, vol. 32, no. 11, art. no. e5654, 2020.
[25] Haluk Topcuoglu, Salim Hariri, Min-You Wu, “Performance-effective and low-complexity task scheduling for heterogeneous computing”, IEEE Transactions on Parallel and Distributed Systems, vol. 13, no. 3, pp. 260–274, 2002.
[26] Muhammed Tawfiqul Islam, Shanika Karunasekera, Rajkumar Buyya, “Performance and cost-efficient Spark job scheduling based on deep reinforcement learning in cloud computing environments”, IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 7, pp. 1695–1710, 2021.
[27] Arfa Muteeh, Muhammad Sardaraz, Muhammad Tahir, “MrLBA: multi-resource load balancing algorithm for cloud computing using ant colony optimization”, Cluster Computing, vol. 24, no. 4, pp. 3135–3145, 2021.