隨著歐盟即將實施碳邊境調整機制(Carbon Border Adjustment Mechanism, CBAM)，「淨零碳排(Net Zero)」已從環保訴求轉化為國際競爭力的關鍵經濟議題。各國紛紛訂定減碳政策，臺灣亦於2024年底逐步完善碳費制度。能源消耗為ISO 14064-1所定義的第二大類溫室氣體排放來源，對於長時間高負載運轉的製造業而言，如何兼顧產能維持與節能減碳，已成為關鍵挑戰。當前市面上能源管理系統(Energy Management System, EMS)雖具備一定程度的自動化能力，但在實務應用上仍面臨三項主要瓶頸：(1) 實際環境充滿不確定性導致自動控制效益不顯著、(2) 參數調控未考量實際能源基線回饋，致使控制效率低落。
本研究提出一套「基於強化學習的前饋設備控制與能耗預測最佳化(Reinforcement Learning-based Feedforward Equipment Control)」，以回應CBAM與2050淨零碳排所帶來的挑戰。系統構成如下：參數微調模組(Parameter Fine-tuning, PF)與第三代全自動虛擬量測(The third generation Automatic Virtual Metrology, AVMIII)系統，使其於虛實環境中互動學習，實現設備運行參數與能源消耗預測(Energy Consumption Prediction)的動態最佳化。PF模組中之PF Agent採用policy-based強化學習方法，調整如: 空壓機與冰水機…等設備參數，並結合考量能耗與運行穩定性的Energy Reward以達成多目標控制；AVMIII模組則運用雙階段(Dual-Phase)機制，針對長時間序列(Long-Horizon Time Series)問題進行了特別設計，使其能更有效地處理智慧製造廠域中的複雜時序資料，滿足多樣化的序列預測需求。
本研究藉由強化學習進行虛實整合聯動與分散式決策學習，實現在動態製造環境下之長期節能與碳排最小化，進而協助製造業邁向智慧化與淨零轉型。

With the European Union (EU) set to implement the Carbon Border Adjustment Mechanism (CBAM), “Net Zero” has evolved from an environmental advocacy into a critical issue of international economic competitiveness. Countries around the world are introducing carbon reduction policies, and Taiwan has also progressively refined its carbon fee system by the end of 2024. Energy consumption constitutes the second largest category of greenhouse gas emissions as defined by ISO 14064-1. For manufacturing industries operating under long hours and high-load conditions, achieving a balance between maintaining production capacity and reducing energy consumption and carbon emissions has become a critical challenge. Although current Energy Management Systems (EMS) provide a certain level of automation, practical applications still face two major bottlenecks: (1) The uncertainty of real-world environments limits the effectiveness of automated control, and (2) Parameter adjustments often fail to incorporate real-time energy baseline feedback, resulting in suboptimal control efficiency.
This study proposes a “Reinforcement Learning-based Feedforward Equipment Control” framework to address the challenges posed by CBAM and the 2050 Net Zero target. The system architecture consists of a Parameter Fine-tuning (PF) module and the third generation Automatic Virtual Metrology (AVMIII) system, enabling interactive learning between virtual and physical environments to achieve dynamic optimization of equipment operating parameters and energy consumption prediction. The PF module employs a policy-based reinforcement learning approach, where the PF Agent adjusts operational parameters of equipment such as air compressors and chillers. An Energy Reward mechanism, incorporating both energy consumption and operational stability considerations, is designed to achieve multi-objective control. Meanwhile, the AVMIII system adopts a Dual-Phase mechanism specifically designed for long-horizon time series problems, enabling more effective handling of complex temporal data in smart manufacturing environments and satisfying diverse sequence prediction requirements.
Through reinforcement learning-based virtual-physical integration and distributed decision-making, this research achieves long-term energy savings and carbon emission minimization in dynamic manufacturing environments, thereby supporting the manufacturing sector’s transition toward intelligent systems and net-zero transformation.

[1] 國家發展委員會等. (2022年3月30日) 臺灣2050淨零排放路徑及策略總說明.
[2] 全國法規資料庫. (2023年2月15日). 氣候變遷因應法.
[3] H. Tieng, T.-C. Ou, T.-H. Tsai, Y.-Y. Li, M.-H. Hung and F. -T. Cheng, “I4.2-GiM: A novel green intelligent manufacturing framework for net zero,” in IEEE Transactions on Automation Science and Engineering, vol. 22, pp. 18030-18050, 2025, doi: 10.1109/TASE.2023.3340149.
[4] E. Foruzan, L.-K. Soh, and S. Asgarpoor, “Reinforcement learning approach for optimal distributed energy management in a microgrid,” IEEE Transactions on Power Systems, vol. 33, no. 5, pp. 5749–5758, Sept. 2018, doi: 10.1109/TPWRS.2018.2823641.
[5] M. Panzer and B. Bender, “Deep reinforcement learning in production systems: A systematic literature review,” International Journal of Production Research, vol. 60, no. 13, pp. 4316-4341, Sep. 2021, doi: 10.1080/00207543.2021.1973138.
[6] Q. Fu, Z. Han, J. Chen, Y. Lu, H. Wu, and Y. Wang, “Applications of reinforcement learning for building energy efficiency control: A review,” Journal of Building Engineering, vol. 50, June. 2022, doi: 10.1016/j.jobe.2022.104165.
[7] S. Sierla, H. Ihasalo, and V. Vyatkin. “A review of reinforcement learning applications to control of heating, ventilation and air conditioning systems.” Energies, vol. 15, no. 10, May. 2022, doi: 10.3390/en15103526.
[8] C. Zhao and N. Deng, “An actor-critic framework based on deep reinforcement learning for addressing flexible job shop scheduling problems,” Math. Biosci. Eng, vol. 21, no. 1, pp. 1445-1471, Dec. 2023, doi: 10.3934/mbe.2024062.
[9] X.-Y. Zheng, C.-J. Liang, Y. Wang, J. Shi, and G. Lim, “Multi-AGV dynamic Scheduling in an automated container terminal: A deep reinforcement learning approach,” Mathematics, vol. 10, no. 23, Dec. 2022, doi: 10.3390/math10234575.
[10] C.-C. Chang, W.-Z. Zhao, C.-Y. Wang, and Z.-K. Luan, “An energy management strategy of deep reinforcement learning based on multi-agent architecture under self-generating conditions,” Energy, vol. 283, Nov. 2023, doi: 10.1016/j.energy.2023.128536.
[11] Y. Du, H. Zandi, O. Kotevska, K. Kurte, J. Munk, K. Amasyali, E. Mckee, and F. Li, “Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning,” Deep Learning for Power System Applications: Case Studies Linking Artificial Intelligence and Power Systems, vol. 281, pp. 71-96, Nov. 2023, doi: 10.1016/j.apenergy.2020.116117.
[12] X. Liu, M. Ren, Z. Yang, G. Yan, Y. Guo, L. Cheng, and C. Wu, “A multi-step predictive deep reinforcement learning algorithm for HVAC control systems in smart buildings,” Energy, vol. 259, Nov. 2022, doi: 10.1016/j.energy.2022.124857.
[13] M. Wang and B. Lin. “MF2: Model-free reinforcement learning for modeling-free building HVAC control with data-driven environment construction in a residential building,” Building and Environment, vol. 244, Oct. 2023, doi: 10.1016/j.buildenv.2023.110816.
[14] F.-T. Cheng, C.-A. Kao, C.-F. Chen, and W.-H. Tsai, “Tutorial on applying the VM technology for TFT-LCD manufacturing,” IEEE Transactions on Semiconductor Manufacturing, vol. 28, no. 1, pp. 55-69, Feb. 2015, doi: 10.1109/TSM.2014.2380433.
[15] Y.-M. Hsieh, T.-J. Wang, C.-Y. Lin, L.-H. Peng, F.-T. Cheng, and S.-Y. Shang, “Convolutional neural networks for automatic virtual metrology,” in IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 5720-5727, Jul. 2021, doi: 10.1109/LRA.2021.3084882.
[16] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017, doi: 10.48550/arXiv.1706.03762.
[17] Q. Wen, T. Zhou, C. Zhang, W. Chen, Z. Ma, J. Yan, and L. Sun, “Transformers in time series: A survey,” arXiv, May. 2023, doi: 10.48550/arXiv.2202.07125.
[18] R. M. Farsani, and E Pazouki. “A transformer self-attention model for time series forecasting,” Journal of Electrical and Computer Engineering Innovations, vol. 9, no. 1, pp. 1-10, Jan. 2021, doi: 10.22061/JECEI.2020.7426.391.
[19] G. Woo, C. Liu, D. Sahoo, A. Kumar, and S. Hoi, “Etsformer: Exponential smoothing transformers for time-series forecasting,” arXiv, Jun. 2022, doi: 10.48550/arXiv.2202.01381.
[20] S. Hochreiter and J Schmidhuber. “Long short-term memory,” Neural computation. Vol. 9, no. 8, pp. 1735-1780, Nov. 1997, doi: 10.1162/neco.1997.9.8.1735.
[21] D. L. Marino, K. Amarasinghe, and M Manic. “Building energy load forecasting using deep neural networks,” IECON 2016-42nd annual conference of the IEEE industrial electronics society. IEEE, Oct. 2016, doi: 10.1109/IECON.2016.7793413.
[22] B. Lim, S. Ö. Arik, N. Loeff, and T. Pfister, “Temporal fusion transformers for interpretable multi-horizon time series forecasting,” International journal of forecasting, vol. 37, no. 4, pp. 1748-1764, Dec. 2021, doi: 10.1016/j.ijforecast.2021.03.012.
[23] Z. Ding, W. Chen, and X. Xu, “Evolutionary double attention-based long short-term memory model for building energy prediction: Case study of a green building,” Applied Energy, vol. 288, Apr. 2021, doi: 10.1016/j.apenergy.2021.116660.