需量反應作為未來電網中之能源選項，具有效抑低尖峰負載、反應時間短、可提高再生能源使用效率及低成本之優點。其可細分為不同類型如自動控制型需量反應、誘因型需量反應、緊急型需量反應及需量競價。然而對於需量反應用戶群代表參與需量競價之相關研究尚在起步階段。
對於整合不同用戶的用戶群代表而言，需在考慮用戶群卸載潛力的同時做出對系統運營商的競標價和投標量之決策。因此本研究之重點為如何整合不同用戶參與之意願及可能的卸載量並且藉由機器學習之方法決定投標策略，以確保用戶群之穩定卸載量及最佳化用戶群代表之獲利。本研究提出以深度確定性策略梯度演算法(DDPG)，透過學習過去的包含系統備轉容量率及用戶群之投標更新模型參數之投標經驗以最佳化投標價及投標量之決策。同時本研究使用每日最新獲得的投標經驗進行線上學習，確保模型有追隨市場變化之適應力。
本研究使用兩個模擬器驗證模型之強健性。模擬結果顯示本研究提出之模型可於各種情境中，藉由線下/線上學習提出精確之投標建議，以最佳化用戶群代表之獲利。

Demand response (DR), as one of the energy resources in future’s grid, provides the services of peak shaving, enhancing the efficiency of renewable energy utilization with short response period, and low cost. Various categories of DR are established, e.g. automated DR, incentive DR, emergency DR, and demand bidding. However, the researches about demand bidding aggregator are just on the beginning stage.
For this issue, the bidding price and bidding curtailment quantity are two bidding decisions required to be determined while considering the potential provided by participants. Therefore, this thesis emphases how to aggregate the bids from participated customers and then determine the bidding decisions by machine learning method while ensuring customers’ stable curtailment quantity to maximize the profit. Deep deterministic policy gradient (DDPG) method is employed to optimize the two bidding decisions through learning historical bidding experiences with the corresponding reserve rate and customers’ plans. The online learning further utilizes the daily newest bidding experience attained to ensure the trend tracing and self- adaptation.
Two environment simulators are adopted for testifying the robustness of the model. The results prove that when facing diverse situations the proposed model is able to earn the optimal profit via off/on-line learning the bidding laws and making the proper bid.

[1] L. Wang, C.-H. Chang, B.-L. Kuan, and A. V. Prokhorov, “Stability Improvement of a Two-Area Power System Connected with an Integrated Onshore and Offshore Wind Farm Using a STATCOM," IEEE Trans. on Ind. Apps., vol. 53, no. 2, pp.867-877, Mar./Apr. 2017.
[2] S.-D. Lu, M.-T. Kuo, C.-C. Wu, M.-C. Tsou, and H. Zeng, “Evaluation of The Maximum Allowable Generation Capacity after Offshore Wind Farms Are Integrated into The Existing Taipower System-A Case Study in Taiwan,” in Proc. 2018 IEEE/IAS 54th Ind. and Comm. Pwr. Sys. Tech. Conf. (I&CPS), Canada, May. 2018.
[3] T.Strasser, F. Andrén, J. Kathan, C. Cecati, C. Buccella, P. Siano, P. Leitão, G. Zhabelova, V. Vyatkin, P. Vrba, and V. Maˇrík, “A Review of Architectures and Concepts for Intelligence in Future Electric Energy Systems,” IEEE Trans. on Ind. Elec., vol.62, no.4, pp.2424-2438, Apr. 2015.
[4] P. Palensky and D. Dietrich, “Demand Side Management: Demand Response, Intelligent Energy Systems, and Smart Loads,” IEEE Trans. on Ind. Inf., vol. 7, no. 3, pp. 381-387, Aug. 2011.
[5] Q. Zhang and J. Li, “Demand Response in Electricity Markets: A Review,” in Proc. 2012 9th Intl. Conf. on the Eur. Energy Market, Florence, Italy, May. 2012
[6] M. Liu, F. L. Quilumba, and W.-J. Lee, “A Collaborative Design of Aggregated Residential Appliances and Renewable Energy for Demand Response Participation,” IEEE Trans. on Ind. Apps., vol. 51, no. 5, pp. 3561-3569, Sep/Oct. 2015.
[7] M. Parvania, M. Fotuhi-Firuzabad, and M. Shahidehpour, “Optimal Demand Response Aggregation in Wholesale Electricity Markets,” IEEE Tran. on Smart Grid, vol. 4, no. 4, pp. 1957-1965, Dec. 2013.
[8] L. Yao and W.-H. Lim, “Optimal Purchase Strategy for Demand Bidding,” IEEE Trans. on Pwr. Sys., vol. 33, no. 3, pp. 2754-2762, May. 2018.
[9] P. Li, H. Wang, and B. Zhang, “A Distributed Online Pricing Strategy for Demand Response Programs,” IEEE Trans. on Smart Grid, vol. 10, no.1, pp. 350-360, Jan. 2019.
[10] A. Rajabi, L. Li, J. Zhang, and J. Zhu, “Aggregation of Small Loads for Demand Response Programs-Implementation and Challenges: A review,” in Proc. 2017 IEEE Intl. Conf. on Env. and Elec. Eng. and 2017 IEEE Ind. and Comm. Pwr. Sys. Eur. (EEEIC / I&CPS Europe), Milan, Italy, Jun. 2017.
[11] M. Wei and J. Zhong, “Optimal Bidding Strategy for Demand Response Aggregator in Day-ahead Markets via Stochastic Programming and Robust Optimization,” in Proc. 2015 12th Intl. Conf. on the Eur. Energy Market (EEM), Lisbon, Portugal, Aug. 2015.
[12] H. Khajeh, A. A. Foroud, and H. Firoozi, “Robust Bidding Strategies and Scheduling of A Price-maker Microgrid Aggregator Participating in A Pool-based Electricity Market,” IET Gener. Transm. Distrib., vol. 13, iss. 4, pp. 468-477, Jan. 2019.
[13] G. Liu, Y. Xu, and K. Tomsovic, “Bidding Strategy for Microgrid in Day-Ahead Market Based on Hybrid Stochastic/Robust Optimization,” IEEE Tran. on Smart Grid, vol. 7, no. 1, pp.227-237, Jan. 2016.
[14] M. Rahimiyan and L. Baringo, “Strategic Bidding for a Virtual Power Plant in The Day-Ahead and Real-Time Markets: A Price-Taker Robust Optimization Approach,” IEEE Trans. on Pwr. Sys., vol. 31, no. 4, pp. 2676-2687, Jul. 2016.
[15] M. Asensio and J. Contreras, “Risk-Constrained Optimal Bidding Strategy for Pairing of Wind and Demand Response Resources,” IEEE Trans. on Smart Grid, vol. 8, no. 1, pp. 200-208, Jan. 2017.
[16] Z. Xu, Z. Hu, Y. Song, and J. Wang, “Risk-Averse Optimal Bidding Strategy for Demand-Side Resource Aggregators in Day-Ahead Electricity Markets Under Uncertainty,” IEEE Trans. on Smart Grid, vol. 8, no. 1, pp. 96-105, Jan. 2017.
[17] D. T. Nguyen and L. B. Le, “Risk-Constrained Profit Maximization for Microgrid Aggregators with Demand Response,” IEEE Trans. on Smart Grid, vol. 6, no. 1, pp. 135-146, Jan. 2015.
[18] H. T. Nguyen, L. B. Le, and Z. Wang, “A Bidding Strategy for Virtual Power Plants with the Intraday Demand Response Exchange Market Using the Stochastic Programming,” IEEE Trans. on Ind. App., vol. 54, no. 4, pp. 3044-3055, Jul./Aug. 2018.
[19] C. Dapeng, J. Zhaoxia, and T. Huijuan, “Optimal Bidding Strategy for Microgrids in the Day-Ahead Energy Market Considering the Price Elasticity of Load Demand,” in Proc. 2018 Intl. Conf. on Pwr. Sys. Tech., Guangzhou, China, Nov. 2018.
[20] Z. Baharlouei and M. Hashemi, “Efficiency-Fairness Trade-off in Privacy-Preserving Autonomous Demand Side Management,” IEEE Trans. on Smart Grid, vol. 5, no. 2, pp. 799-808, Mar. 2014.
[21] M. Rahimiyan and H. R. Mashhadi, “Supplier's Optimal Bidding Strategy in Electricity Pay-as-bid Auction: Comparison of the Q-learning and A Model-based Approach,” Sci. Dir. Elec. Pwr. Sys. Rsrch., vol. 78, iss. 1, pp. 165-175, Jan. 2008.
[22] G.R. Gajjar, S.A. Khaparde, P. Nagaraju, and S.A. Soman, “Application of Actor-critic Learning Algorithm for Optimal Bidding Problem of a Genco,” IEEE Trans. on Pwr. Sys., vol. 18, no. 1, pp. 11-18, Feb. 2003.
[23] Taiwan Power Company, (Jan. 2018). Demand Bidding Rule. [online]. Available: http://taipowerdsm-clinic.taipower.com.tw/files/TP_TreasureBox/TB_File/0216DM1.pdf.
[24] PJM, (Jan. 2019). PJM Capacity Market. [online]. Available: https://www.pjm.com/-/media/documents/manuals/m18.ashx?la=en
[25] IESO, (May. 2017). Introduction to the Demand Response Auction. [online]. Available:http://www.ieso.ca/en/Sector-Participants/Market-Operations/Markets-and-Related-Programs/Demand-Response-Auction
[26] California ISO, (May. 2014). Overview of Reliability Demand Response Resource. [online].Available:http://www.caiso.com/Documents/ReliabilityDemandResponseResourceOverview.pdf
[27] G. R. Gajjar, S. A. Khaparde, P. Nagaraju, and S. A. Soman, “Application of Actor-Critic Learning Algorithm for Optimal Bidding Problem of a Genco,” IEEE Trans. on Pwr. Sys., vol. 18, no. 1, pp. 11-18, Feb. 2003.
[28] H. Zhao, Y. Wang, S. Guo, M. Zhao, and C. Zhang, “Application of a Gradient Descent Continuous Actor-Critic Algorithm for Double-Side Day-Ahead Electricity Market Modeling,” Energies, 9, no. 9: 725, 2016.
[29] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing Atari with Deep Reinforcement Learning,” arXiv preprint, arXiv:1312.5602, 2013.
[30] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy Gradient Methods for Reinforcement Learning with Function Approximation,” Adv. in Neural Process. Sys. 12, MIT Press, pp. 1057-1063, 2000.
[31] V. R. Konda and J. N. Tsitsiklis, “Actor-critic Algorithms,” Massachusetts Institute of Technology, 2000.
[32] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous Control with Deep Reinforcement Learning,” in Proc. ICLR 2016, San Juan, Puerto Rico, May. 2016.
[33] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Deterministic Policy Gradient Algorithms,” in Proc. ICML'14 Pro. of the 31st Intl. Conf. on Intl. Conf. on Machine Learning, Beijing, China, Jun. 2014.
[34] Institute for Matematiske Fag, Example: the Ornstein-Uhlenbeck process. [online]. Available: http://web.math.ku.dk/~susanne/StatDiff/Overheads1b
[35] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg and D. Hassabis, “Human-level Control through Deep Reinforcement Learning,” Nature 518(7540), pp. 529-533, Feb. 2015.
[36] Taiwan Power Company, Demand Bidding Online System. [online]. Available: https://dbp.taipower-ami.com.tw/
[37] M.Song and M, Amelin, “Price-Maker Bidding in Day-Ahead Electricity Market for a Retailer with Flexible Demands,” IEEE Trans. on Pwr. Sys., vol. 33, no. 2, pp. 1948-1958, Mar. 2018.
[38] M.Song and M, Amelin, “Estimating the Price Elasticity of Residential Power Demand Using a Bottom-up Approach,” in Proc. 2016 IEEE PES Asia-Pacific Pwr. and Ener. Conf., Xi’an, China, Oct. 2016.
[39] F. Ramos, C. Cañizares, and K. Bhattacharya, “Effect of Price Responsive Demand on the Operation of Microgrids,” in Proc. 2014 Pwr. Sys. Comput. Conf., Wroclaw, Poland, Aug. 2014.