本論文提出一套基於強化學習控制器的自動發電控制系統，應用於大型離岸風場整合儲能系統之再生能源併網多機電力系統，俾提升所研究的系統之主電網頻率調節與穩定度。首先，對所研究的系統架構進行穩態與小訊號穩定度分析，並用頻域分析方法，評估並設計比例-積分-微分型輔助阻尼控制器應用於儲能系統之效果。接著，於所研究的完整系統架構，訓練以深度確定性策略梯演算法為核心的強化學習控制器，實現智慧型的自動發電控制功能。最後，以動態與暫態時域模擬，比較下列各系統架構之穩定度：(1)無儲能系統、(2)有儲能系統但無輔助阻尼控制器、(3)有輔助阻尼控制器之儲能系統、(4)有輔助阻尼控制器之儲能系統再搭配強化學習控制器之自動發電控制。由模擬結果顯示，儲能系統能有效提升整體系統穩定度，輔助阻尼控制器進一步增強功率與電壓穩定性，而強化學習控制器則在頻率穩定度上表現最為顯著，顯示其在智慧型電力系統應用上具高度潛力。

This thesis proposes an automatic generation control (AGC) system based on a reinforcement learning (RL) controller, applied to a multi-machine power system integrating a large-scale offshore wind farm and an energy storage system (ESS) for renewable energy grid connection. The proposed control scheme aims to improve the frequency regulation capability and overall stability of the main power grid of the studied system. First, steady-state and small-signal stability analyses are conducted on the system studied. Frequency-domain analysis is employed to evaluate the effectiveness of a designed proportional-integral-derivative supplementary damping controller (SDC) applied to the ESS. Subsequently, within the complete system framework, an RL controller-based AGC is developed using the deep deterministic policy gradient algorithm to achieve intelligent generation control. Finally, dynamic and transient time-domain simulations are carried out to compare system stability under four system configurations: (1) without the ESS, (2) with the ESS but without an SDC, (3) with the ESS and the SDC, and (4) with the ESS and the SDC combined with the RL controller-based AGC system. Simulation results demonstrate that the ESS significantly improves overall system stability, the SDC further improves power and voltage stability, and the RL controller-based AGC system exhibits superior performance in frequency stability. These results highlight the high potential of the RL controller-based AGC system in the development of intelligent power systems.

[1] H. Saadat, Power System Analysis, 3rd ed., McGraw-Hill Education, Taiwan Branch, Taipei, Taiwan, 2015.
[2] J. J. Grainger, W. D. Stevenson, and G. W. Chang, Power System Analysis, McGraw-Hill Education, New York, NY, USA, 2016.
[3] H. Z. Kazme, K. Basumatary, and A. Shukla, “Analysis of power deployment characteristics and costs of energy storage systems for frequency regulation in renewable-rich power grid,” in Proc. 2024 IEEE 8th Conference on Energy Internet and Energy System Integration, Shenyang, China, Nov. 29-Dec. 02, 2024.
[4] R. Radhika and K. C. S. Thampatty, “Impacts of Renewable Energy Integration on Power System Stability,” in Proc. 2023 IEEE Technology & Engineering Management Conference, Bengaluru, India, Dec. 14-16, 2023, pp. 1-6.
[5] 經濟部能源署 能源統計, 再生能源發電裝置容量, [Online]. Available: https://www.esist.org.tw.
[6] L. Tian, L. Ba, and Weiguo He, “Analysis of the demand for energy storage due to the frequency stability in renewable energy source dominated power systems,” in Proc. 2025 10th Asia Conference on Power and Electrical Engineering (ACPEE), Beijing, China, Apr. 15-19, 2025, pp. 475-480.
[7] I. A. Khan, H. Mokhlis, N. N. Mansor, H. A. Illias, M. Usama, and A. Daraz, “Load frequency control using golden eagle optimization for multi-area power system connected through AC/HVDC transmission and supported with hybrid energy storage devices,” IEEE Access, vol. 11, pp. 44672-44695, May 2023.
[8] P. McNamara and F. Milano, “Model predictive control-based AGC for multi-terminal HVDC-connected AC grids,” IEEE Trans. Power Systems, vol. 33, no. 1, pp. 1036-1048, Jan. 2018.
[9] P. Kundur, Power System Stability and Control, McGraw-Hill, New York, NY, USA, 1994.
[10] C. Li, Y. Wu, Y. Sun, H. Zhang, Y. Liu, and V. Terzija, “Continuous under-frequency load shedding scheme for power system adaptive frequency control,” IEEE Trans. Power Systems, vol. 35, no. 2, pp. 950-961, Mar. 2020.
[11] A. Annamraju, V. B. Pamshetti, R. K. Avvari, N. R. Nareddy, and C. Srinivasarathnam, “Load frequency control in a renewable rich power systems: an artificial hummingbird optimization based multistage pid control approach,” in Proc. 2024 IEEE International Conference on Smart Power Control and Renewable Energy (ICSPCRE), Rourkela, India, Jul. 19-21, 2024, pp. 1-6.
[12] Mamta, V. P. Singh, A. V. Waghmare, V. P. Meena, F. Benedetto, and T. Varshney, “Rank exponent method based optimal control of AGC for two-area interconnected power systems,” IEEE Access, vol. 12, pp. 35571-35585, Mar. 2024.
[13] Y. Shen, W. Wu, and S. Sun, “Stochastic model predictive control based fast-slow coordination automatic generation control,” IEEE Trans. Power Systems, vol. 39, no. 3, pp. 5259-5271, May 2024.
[14] Xi. Su, X. Liu, and Y.-D. Song, “Event-triggered sliding-mode control for multi-area power systems,” IEEE Trans. Industrial Electronics, vol. 64, no. 8, pp. 6732-6741, Mar. 2017.
[15] C. Tan, Y. Li, Q. Lan, X. Xiao, Q. Ding, and X. Zhang, “Multi-area automatic generation control scheme considering frequency quality in southwest china grid: challenges and solutions,” IEEE Access, vol 8, pp. 199813-199828, Oct. 2020.
[16] S. A. Hosseini, M. Toulabi, A. S. Dobakhshari, A. Ashouri-Zadeh, and A. M. Ranjbar, “Delay compensation of demand response and adaptive disturbance rejection applied to power system frequency control,” IEEE Trans. Power Systems, vol. 35, no. 3, pp. 2037-2046, May 2020.
[17] K. Doenges, I. Egido, L. Sigrist, E. L. Miguélez, and L. Rouco, “Improving AGC performance in power systems with regulation response accuracy margins using battery energy storage system (BESS), IEEE Trans. Power Systems, vol 35, no. 4, pp. 2816-2825, Jul. 2020.
[18] S. Li, Y. Hu, X. He, Y. Zhu, T. He, and C. Luo, “Simulation research on primary frequency regulation coordination control strategy of hybrid wind-PV-ES power station based on AGC,” in Proc. Annual Meeting of CSEE Study Committee of HVDC and Power Electronics(HVDC 2023), Nanjing, China, Oct. 22-25, 2023, pp. 1-9.
[19] A. Alavi-Koosha, T. Amraee, and S. S. Oskouee, “A multi-area design of under frequency load shedding schemes considering energy storage system,” IET Generation, Transmission & Distribution, vol. 17, no. 19, pp. 4437-4452, Oct. 2023.
[20] L. Xi, L. Zhou, Y. Xu, and X. Chen, “A multi-step unified reinforcement learning method for automatic generation control in multi-area interconnected power grid,” IEEE Trans. Sustainable Energy, vol. 12, no. 2, pp. 1406-1415, Apr. 2021.
[21] Y. Shen, W. Wu, and S. Sun, “Stochastic model predictive control based fast-slow coordination automatic generation control,” IEEE Trans. Power Systems, vol. 39, no. 3, pp. 5259-5271, May 2024.
[22] P. M. Anderson and A. A. Fouad, Power System Control and Stability, The Iowa State University Press, Ames, IA, USA, 1988.
[23] P. C. Krause, O. Wasynczuk, S. Sudhoff, and S. Pekarek, Analysis of Electric Machinery and Drive Systems, 3rd ed., Hoboken, Wiley, NJ, USA, 2013.
[24] Vestas, Vestas Company News, [Online]. Available: https://www.vestas.com/en/media/company-news.
[25] 中能離岸風場, [Online]. Available: https://znwp.com.tw/
[26] 許瑛娟, [圖解電未來]海風吹來的能源寶藏 離岸風電的併網整合, 台電月刊, vol. 736, pp. 44-45, Mar. 2024.
[27] S. Heier, Grid Integration of Wind Energy Conversion Systems, Chichester, John Wiley & Son, Chichester, UK, 2014.
[28] D. Saidani, O. Hasnaoui, and R. Dhifaoui, “Control of double fed induction generator in WECS,” in Proc. The 2nd International Conference on Power Engineering and Renewable Energy (ICPERE), Bali, Indonesia, Dec. 9-11, 2014, pp. 25-30.
[29] 武光山(2017)。採用以超級電容器為基礎之儲能設備於含有市電併聯型混合再生能源系統之性能改善。國立成功大學電機工程學系博士論文，台南市。取自https://hdl.handle.net/11296/k4tptj
[30] H. Liang, Z. He, J. Guo, X. Wang, Z. Lin, and L. Zhang, “Mass-spring-damping-based tissue deformation modeling for robotic acupuncture,” in Proc. 2023 International Annual Conference on Complex Systems and Intelligent Science (CSIS-IAC), Shenzhen, China, Oct. 20-22, 2023, pp. 647-652.
[31] G. Huang, “Novel black start control strategy with power boundary analysis for PMSG-based wind turbines,” IEEE Trans. Sustainable Energy, vol. 15, no. 2, pp. 1224-1238, Apr. 2024.
[32] K. A. Singh, A. Chaudhary, and K. Chaudhary, “Three-phase AC-DC converter for direct-drive PMSG-based wind energy conversion system,” Journal of Modern Power Systems and Clean Energy, vol. 11, no. 2, pp. 589-598, Mar. 2023.
[33] S. Heier, Grid Integration of Wind Energy Conversion Systems, Chichester, UK: John Wiley & Son, 2014.
[34] I. K. Amin, M. N. Islam, A. Jaman, M. K. H, M. A. Bithi, M. N. Uddin, “Modeling and performance analysis of redox-flow battery unit for large-scale hybrid renewable energy systems,” in Proc. 2022 IEEE Industry Applications Society Annual Meeting, Detroit, MI, USA, Oct. 09-14, 2022, pp. 1-8.
[35] 柯王君奕(2020)。採用全釩氧化還原液流電池及超級電容器於市電併聯型混和再生能源系統之穩定度改善分析。國立成功大學電機工程學系碩士論文，台南市。取自https://hdl.handle.net/11296/he6pcn
[36] 高浩瑜(2021)。使用自適應類神經網路控制器於混合交流/直流為電網系統之性能改善。國立成功大學電機工程學系碩士論文，台南市。取自https://hdl.handle.net/11296/yx9v3z
[37] 李旻舫(2022)。雙向互連轉換器於孤島混合交流/直流微電網之頻率控制策略。國立成功大學電機工程學系碩士論文，台南市。取自https://hdl.handle.net/11296/4694j6
[38] 黃志宏(2022)。設計混合式儲能系統之自適應模糊邏輯控制器以達成混合太陽/風能微電網系統的功率平滑。國立成功大學電機工程學系碩士論文，台南市。取自https://hdl.handle.net/11296/6kb4c4
[39] C. Kim and Wonhee Kim, “Coordinated Fuzzy-Based Low-Voltage Ride-Through Control for PMSG Wind Turbines and Energy Storage Systems,” IEEE Access, vol. 8, pp. 105874-105885, Jun. 2020.
[40] M. Ahmed, L. Meegahapola, M. Datta, and A. Vahidnia, “A novel hybrid ac/dc microgrid architecture with a central energy storage system,” IEEE Trans. Power Delivery, vol. 37, no. 3, pp. 2060-2070, Agu. 2022.
[41] 秋庭伸也、杉山阿聖、寺田學，零基礎入門的機器學習圖鑑，ACME Publishing Company，台北市，2021。
[42] 郭卡、戴亮，PyTorch 深度學習，深智數位股份有限公司，台北市，2022。
[43] M. Lapan, Deep Reinforcement Learning Hands-On, DrMaster Press Company, New Taipei City, Taiwan, 2019.
[44] 小川雄太郎，實戰人工智慧之深度強化學習使用PyTorch x Python, Gotop Information Inc，台北市，2019。
[45] A. Zai, B. Brown, Deep Reinforcement Learning IN ACTION, Flag Technology Company, Taipei, Taiwan, 2021.
[46] 郭憲、方勇純，最新人工智慧應用 用強化學習快速上手AI，佳魁資訊，台北市，2018。
[47] 董豪、丁子涵、仉尚航，新一代AI霸主：深度強化學習從基礎開始到專案開發，深智數位股份有限公司，台北市，2022。
[48] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Deterministic policy gradient algorithms,” in Proc. 31st International Conference on Machine Learning, Beijing, China, Jun. 21-26, 2014, pp. 387-395.
[49] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” in Proc. International Conference on Learning Representations 2016, Santo Domingo, Dominican Republic, May 2-4, 2016, pp. 1-14.