近年來，由於溫室氣體大量排放，導致極端氣候不斷發生，嚴重威脅人類生存環境，世界各國因此紛紛實施相關減碳政策，然而傳統燃油汽車碳排占總體人類活動所產生之碳排放約13%，以電力汽車取代燃油汽車減少溫室氣體排放，已是許多國家首要之減碳目標。我國政府亦擬定於2030年，達成客運公車全面電動化之政策目標，使用低碳排放的電動巴士取代傳統燃油交通工具，雖能減緩溫室效應，但當大量電動巴士充電負載加入電網，若未經妥善控管，勢必對電力系統之供電品質及安全性造成影響。
傳統上為解決電動巴士實際行駛與充電站微電網內再生能源發電之不確定性，須利用最佳化演算法依據實際環境變化狀況，進行極短期最佳化規劃，然而當需要考量之環境參數數量上升，整體系統越加複雜，其運算時間將大於規劃時間，無法進行即時最佳化電動巴士充電控制，為解決上述情況，本研究提出一針對電動巴士車隊之增強式學習充電排程方法，考量電動巴士車隊及多個充電站，包含儲能系統及再生能源發電裝置，目標為最小化充電站營運商之成本，建立電力消耗模型結合公車班次表，模擬公車行駛環境，利用人工智慧之增強式學習模型與環境互動，使模型能夠產生最佳電動巴士充電決策。模擬案例使用本文所提之方法與混合整數線性規劃演算法進行比較，並分析參與需量反應之效益，同時，加入太陽能發電與電動巴士電力消耗變動之不確定性，證明本文所提之方法能即時響應環境中隨機變化，同時達成整體充電站成本最小化之目標。

In recent years, countries around the world have adopted carbon reduction policies because of the massive emission of greenhouse gases and extreme climate change. The carbon emissions of fuel vehicles account for about 13% of the total carbon emissions. Replacing fuel vehicles with electric vehicles to reduce carbon emissions has become the primary goal for many countries. Taiwan plans to replace all public buses on the road with electric buses (EBs) by 2030 to achieve the target of decreasing carbon emissions, but our power system may face a challenge of stability and security due to the massive uncontrollable charging load that integrates into the grid.
To address the uncertainty of the actual driving of EBs and the generation of renewable energy in the charging stations (CSs), the optimization algorithm needs to manage the uncertainty of environmental changes with short-term optimization.
However, as the system becomes more complex, the parameters of the environment increase, which leads to a longer computing time. When the computing time is longer than the control timeslot of the system, it could not be implemented in a real-time EB charging control.
This thesis proposes a method based on reinforcement learning (RL) to optimize the charging schedule for EBs. It takes EBs and multiple CSs composition, which include energy storage systems and renewable energy devices, into consideration. The objective is to minimize the cost of the charging point operator (CPO). This thesis establishes an energy consumption model combined with the bus timetable and simulates the actual bus driving scenario. It uses the RL model to interact with the environment and enables the model to learn and make an optimal charging decision for EBs. The case study simulates the intermittent photovoltaics in the CS and the uncertainty of the energy consumption of EBs driving. We compare the performances of the proposed method with a common optimization algorithm, which includes the profits analysis of participating in a demand response (DR). The result reveals that the proposed method can be applied in real-time control and cope with environmental uncertainties while minimizing the CPO's cost.

[1] 行政院環保署氣候公民對話平台，[Online]. Available: https://www.climatetalks.tw/blank-14 [Accessed 2 Aug. 2022].
[2] 交通部公路公共運輸補助電動大客車作業要點，2019年8月29日，[Online]. Available:https://www.mvdis.gov.tw/webMvdisLaw/LawContent.aspx?LawID=E0092003 [Accessed 2 Aug. 2022].
[3] 台灣電力公司，電動車充換電電價設施，2022年5月，[Online]. Available: https://www.taipower.com.tw/upload/6594/2022052710021581051.pdf [Accessed 2 Aug. 2022].
[4] 交通部運輸研究所，電動大客車推動策略初探，2019年9月，[Online]. Available: https://reurl.cc/kEMp49 [Accessed 2 Aug. 2022].
[5] Y. He, Z. Song, and Z. Liu, “Fast-Charging Station Deployment for Battery Electric Bus Systems Considering Electricity Demand Charges,” Sustainable Cities and Society, vol. 48, 101530, 2019.
[6] N. A. El-Taweel, H. E. Z. Farag, and M. Mohamed, “Integrated Utility-Transit Model for Optimal Configuration of Battery Electric Bus Systems,” IEEE Systems Journal, vol. 14, no. 1, pp. 738-748, Mar. 2020.
[7] H. Ding, Z. Hu, and Y. Song, “Value of The Energy Storage System in an Electric Bus Fast Charging Station,” Appl. Energy, vol. 157, no. 1, pp. 630-639, Nov. 2015.
[8] R. C. Leou and J. J. Hung, “Optimal Charging Schedule Planning and Economic Analysis for Electric Bus Charging Stations,” Energies, vol. 10, no. 4, 483, 2017.
[9] Y. Wang, Y. Huang, J. Xu, and N. Barclay, “Optimal Recharging Scheduling for Urban Electric Buses: A Case Study in Davis,” Transportation Research Part E: Logistics and Transportation Review, vol. 100, pp. 115-132, 2017.
[10] A. Jahic, M. Plenz, M. Eskander, and D. Schulz, “Route Scheduling for Centralized Electric Bus Depots,” IEEE Open Journal of Intelligent Transportation Systems, vol. 2, pp. 149-159, 2021.
[11] J. Teng, T. Chen, and W. Fan, “Integrated Approach to Vehicle Scheduling and Bus Timetabling for an Electric Bus Line,” Journal of Transportation Engineering, Part A: Systems, vol. 146, no. 2, Feb. 2020.
[12] Y. He, Z. Liu, and Z. Song, “Optimal Charging Scheduling and Management for a Fast-Charging Battery Electric Bus System,” Transportation Research Part E: Logistics and Transportation Review, vol. 142, 102056, Oct. 2020.
[13] K. Liu, H. Gao, Z. Liang, M. Zhao, and C. Li, “Optimal Charging Strategy for Large-Scale Electric Buses Considering Resource Constraints,” Transportation Research Part D: Transport and Environment, vol. 99, 103009, Oct. 2021.
[14] Y. Liu and H. Liang, “A Three-Layer Stochastic Energy Management Approach for Electric Bus Transit Centers With PV and Energy Storage Systems,” IEEE Transactions on Smart Grid, vol. 12, no. 2, pp. 1346-1357, Mar. 2021.
[15] P. Zhuang and H. Liang, “Stochastic Energy Management of Electric Bus Charging Stations with Renewable Energy Integration and B2G Capabilities,” IEEE Transactions on Sustainable Energy, vol. 12, no. 2, pp. 1206-1216, Apr. 2021.
[16] S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Computation, vol. 9, pp. 1735-1780, Nov. 1997.
[17] 中央氣象局氣象資料開放平台，[Online]. Available: https://opendata.cwb.gov.tw/index [Accessed 2 Aug. 2022].
[18] Y. Gao, J. Yang, M. Yang, and Z. Li, “Deep Reinforcement Learning Based Optimal Schedule for a Battery Swapping Station Considering Uncertainties,” IEEE Transactions on Industry Applications, vol. 56, no. 5, pp. 5775-5784, Oct. 2020.
[19] 台灣電力公司負載管理專區，需量競價措施，2021年12月，[Online]. Available: https://www.taipower.com.tw/upload/135/2022032510595355610.pdf [Accessed 2 Aug. 2022].
[20] C. J. C. H. Watkins and P. Dayan, “Q-learning,” Machine Learning, vol. 8, pp. 279-292, 1992.
[21] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, “Deep Reinforcement Learning: A Brief Survey,” IEEE Signal Processing Magazine, vol. 34, no. 6, pp. 26-38, Nov. 2017.
[22] T. Ding, Z. Zeng, J. Bai, B. Qin, Y. Yang, and M. Shahidehpour, “Optimal Electric Vehicle Charging Strategy with Markov Decision Process and Reinforcement Learning Technique,” IEEE Transactions on Industry Applications, vol. 56, no. 5, pp. 5811-5823, Oct. 2020.
[23] G. B. Dantzig, “Maximization of a Linear Function of Variables Subject to Linear Inequalities,” Activity Analysis of Production and Allocation, edited by T.C.Koopmans, John Wiley and Sons, New York, pp. 339-347, 1951.
[24] X. Li, G. Chen, and Z. Y. Dong, “Co-Optimisation Model for the Long-Term Design and Decision Making in Community Level Cloud Energy Storage System,” IET Renewable Power Generation, vol. 14, no. 17, pp. 3518-3525, 2020.
[25] Y. Liu and H. Liang, “A Data-Driven Approach for Electric Bus Energy Consumption Estimation,” IEEE Transactions on Intelligent Transportation Systems, pp. 1-12, Mar. 2022.
[26] W. D. Connor, Y. Wang, A. A. Malikopoulos, S. G. Advani and A. K. Prasad, “Impact of Connectivity on Energy Consumption and Battery Life for Electric Vehicles,” IEEE Transactions on Intelligent Vehicles, vol. 6, no. 1, pp. 14-23, Mar. 2021.
[27] N. A. El-Taweel, A. Zidan and H. E. Z. Farag, “Novel Electric Bus Energy Consumption Model Based on Probabilistic Synthetic Speed Profile Integrated With HVAC,” IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 3, pp. 1517-1531, Mar. 2021.
[28] 大台南公車時刻表，[Online]. Available: https://tourguide.tainan.gov.tw/newtnbusweb/IframeUrl.aspx?1 [Accessed 2 Aug. 2022].
[29] 台南市交通局公車載運量統計資料，2017年2月，[Online]. Available: https://data.gov.tw/dataset/53402 [Accessed 2 Aug. 2022].
[30] 比亞迪官網，BYD K9電動巴士規格，[Online]. Available: https://en.byd.com/wp-content/uploads/2019/07/4504-byd-transit-cut-sheets_k9-40_lr.pdf [Accessed 2 Aug. 2022].
[31] 華德動能官網，RAC-700電動巴士規格，[Online]. Available: https://www.racev.com/es/product/rE [Accessed 2 Aug. 2022].