隨著microservice的崛起，有越來越多企業將應用程式部署到microservice上面，透過microservice，應用程式被分割為數個服務分散在fog 環境中，彼此互相合作提供服務，面對變動的workload時，microservice-based應用程式能以服務為單位調整資源的使用，提升應用程式的elasticity，滿足應用程式的QoS (Quality of Service)需求，Service elasticity在過去cloud computing的研究當中已經被廣泛的討論，但目前對於microservice-based應用程式的elasticity的研究才剛開始。
在過去elasticity的研究中，可以大致將elasticity policy 分為二種，non RL-based跟RL-based方法，non RL-based方法顧名思義就是以非RL的方法來達到elasticity，以threshold-based方法為大宗，threshold-based方法是一個熱門且設計容易的方法，但對於系統來說，threshold-based policy雖能增加elasticity，但他的效能取決於threshold設計的好壞，設定不好時可能造成資源浪費或服務表現不佳，在RL-based policy方面，過去已經有許多的RL-based的elasticity policy問世，透過強化學習找出最佳的elasticity policy，但大部分的方法都假設系統存在於Cloud當中，且不是用microservice的形式實現，多數方法都沒有同時考慮系統的實際需求，例如系統啟動時間、維持系統運作的最小資源量等，為此，本文提出一個二層式elasticity policy用於microservice-based的M2M環境，透過Docker實作，將M2M系統的需求納入考慮，第一層使用Q learning學習系統中單一服務的最佳Elasticity policy，再結合第二層擁有系統中所有服務資訊的Threshold-based policy，找出並改善系統的bottleneck，來改善這個系統的表現，最後我們再與static threshold policy跟dynamic threshold policy做比較，比較資源使用量、cpu utilization、response time、t_max violation，結果顯示我們的方法能夠使用較少的資源得到和其他二個方法差不多甚至更好的表現。

With the rise of microservice, more and more enterprises deploying their applications to microservice. Through microservice, applications can be split into several microservices and cooperate with each other to provide services. Facing the changing workload, microservice-based application can adjust the use of resources on microservice-based basis to enhance the elasticity of applications and meet the QoS(Quality of Service). Service elasticity has been widely discussed in cloud computing research in the past, but research on elasticity for microservice-based applications is just beginning.
In the past studies, elasticity policy can be broadly classified into two types, non RL-based and RL-based policy. The non RL-based policy, as the name implies, does not use RL policy to achieve elasticity, and the threshold-based policy is the most popular one. The threshold-based policy is a popular and easy to design policy, but for the system, threshold-based policy can increase elasticity, but its performance depends on the threshold design, which may cause waste of resources or poor service performance if not set properly. In terms of RL-based policy, many RL-based elasticity policies have been introduced in the past. They use reinforcement learning to find the best elasticity policy. However, most of them assume that the system exists in the Cloud and are not implemented in the form of a microservice. Most of them do not consider the actual requirements of the system at the same time, such as the system start-up time, the minimum amount of resources to keep the system running, etc. For this reason, this thesis proposes a two level elasticity policy for microservice-based M2M environment through Docker implementation. Taking the requirements of oneM2M system into consideration, the first level uses Q learning to learn the best elasticity policy for a single service in the system, and then combines the second level with a threshold-based policy that has information about all the services in the system to identify and improve the bottleneck of the system to improve the performance of this system. At the end of the thesis, we compare our policy with static threshold policy and dynamic threshold policy in resource use, cpu utilization, response time and t_max violation. The result shows that our policy can achieve similar or better performance than the other two policies with less resources.

[1] Prakash, Vijay, Seema Bawa, and Lalit Garg. "Multi-Dependency and Time Based Resource Scheduling Algorithm for Scientific Applications in Cloud Computing." Electronics vol. 10, no. 11 pp. 1320 2021 .
[2] Y. Al-Dhuraibi, F. Paraiso, N. Djarallah and P. Merle, "Elasticity in Cloud Computing: State of the Art and Research Challenges," in IEEE Transactions on Services Computing, vol. 11, no. 2, pp. 430-447, 1 March-April 2018.
[3] Tseng, Chih-Lung, and Fuchun Joseph Lin. "Extending scalability of IOT/M2M platforms with fog computing." 2018 IEEE 4th World Forum on Internet of Things (WF-IoT). pp. 825-830, 2018.
[4] oneM2M. Available on July 7,2021: https://www.onem2m.org/ .
[5] Docker. Available on July 7,2021: https://www.docker.com/ .
[6] F. Rossi, M. Nardelli and V. Cardellini, "Horizontal and Vertical Scaling of Container-Based Applications Using Reinforcement Learning," 2019 IEEE 12th International Conference on Cloud Computing (CLOUD) , pp. 329-338, 2019.
[7] F. S. Hillier and G. J. Lieberman, “Introduction to Operations Research”, 8th edition, 2005, McGraw-Hill Inc. https://reurl.cc/mvdbo7.
[8] F. Rossi, V. Cardellini and F. L. Presti, "Self-adaptive Threshold-based Policy for Microservices Elasticity," 2020 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) , pp. 1-8, 2020.
[9] F. Rossi, V. Cardellini and F. L. Presti, "Hierarchical Scaling of Microservices in Kubernetes," 2020 IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS) , pp. 28-37, 2020.
[10] Kubernetes. Available on July 26,2021: https://kubernetes.io/.
[11] OpenFog Consortium Architecture Working Group. "OpenFog reference architecture for fog computing." OPFRA001 20817 (2017): 162. https://www.iiconsortium.org/pdf/OpenFog_Reference_Architecture_2_09_17.pdf .
[12] oneM2M.org, “TS-0001- Functional_Architecture-V2_10_0”, 2016-Aug-30 . https://www.onem2m.org/images/files/deliverables/Release2/TS-0001-%20Functional_Architecture-V2_10_0.pdf .
[13] Ghirotti, Silvia Esparrachiari, Tanya Reilly, and Ashleigh Rentz. "Tracking and controlling microservice dependencies." Communications of the ACM: vol 61, no. 11 pp 98-104, 2018.
[14] Docker Engine API. Available on July 7,2021: https://docs.docker.com/engine/api/v1.41/ .
[15] Docker Swarm. Available on July 7,2021:https://docs.docker.com/engine/swarm/ .
[16] Docker overlay network. Available on July 7,2021 https://docs.docker.com/network/overlay/
[17] Ongaro, Diego, and John Ousterhout. "In search of an understandable consensus algorithm." 2014 {USENIX} Annual Technical Conference. 2014. pp. 305-319, 2014.
[18] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018. https://reurl.cc/yeyb6M.
[19] Y. Al-Dhuraibi, F. Paraiso, N. Djarallah and P. Merle, "Autonomic Vertical Elasticity of Docker Containers with ELASTICDOCKER," 2017 IEEE 10th International Conference on Cloud Computing (CLOUD) , pp. 472-479, 2017.
[20] M. Nardelli, V. Cardellini and E. Casalicchio, "Multi-Level Elastic Deployment of Containerized Applications in Geo-Distributed Environments," 2018 IEEE 6th International Conference on Future Internet of Things and Cloud (FiCloud) , pp. 1-8, 2018.
[21] Khazaei, Hamzeh, et al. "Elascale: Autoscaling and monitoring as a service." arXiv preprint arXiv:1711.03204 (2017).
[22] F. Tseng, M. Tsai, C. Tseng, Y. Yang, C. Liu and L. Chou, "A Lightweight Autoscaling Mechanism for Fog Computing in Industrial Applications," IEEE Transactions on Industrial Informatics, vol. 14, no. 10, pp. 4529-4537, Oct. 2018.
[23] L. A. Zadeh, "Fuzzy logic,"IEEE Computer, vol. 21, no. 4, pp. 83-93, April 1988.
[24] S. Horovitz and Y. Arian, "Efficient Cloud Auto-Scaling with SLA Objective Using Q-Learning," 2018 IEEE 6th International Conference on Future Internet of Things and Cloud (FiCloud) , pp. 85-92, 2018.
[25] H. Arabnejad, C. Pahl, P. Jamshidi and G. Estrada, "A Comparison of Reinforcement Learning Techniques for Fuzzy Cloud Auto-Scaling," 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID) , pp. 64-73, 2017.
[26] D. Tsoumakos, I. Konstantinou, C. Boumpouka, S. Sioutas and N. Koziris, "Automated, Elastic Resource Provisioning for NoSQL Clusters Using TIRAMOLA," 2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing , pp. 34-41, 2013.
[27] ML Puterman, “Markov Decision Processes: Discrete Stochastic Dynamic Programming”,2014. https://reurl.cc/em5bkL .
[28] S. Zhang, T. Wu, M. Pan, C. Zhang and Y. Yu, "A-SARSA: A Predictive Container Auto-Scaling Algorithm Based on Reinforcement Learning," 2020 IEEE International Conference on Web Services (ICWS) , pp. 489-497, 2020.
[29] R. N. Calheiros, E. Masoumi, R. Ranjan, and R. Buyya, “Workload prediction using arima model and its impact on cloud applications qos,” IEEE Transactions on Cloud Computing, vol. 3, no. 4, pp. 449–458, 2014.
[30] M. Borkowski, S. Schulte, and C. Hochreiner, “Predicting cloud resource utilization,” Proceedings of the 9th International Conference on Utility and Cloud Computing, pp. 37-42, 2016.
[31] X. Luo, Y. Lv, R. Li, and Y. Chen, “Web service qos prediction based on adaptive dynamic programming using fuzzy neural networks for cloud services,” IEEE Access, vol. 3, pp. 2260–2269, 2015.