Apache Kafka 是一個分散式訊息串流平台，其大數據的即時串流與儲存能力使 Apache Kafka 有廣大的使用者。監控 Kafka 效能指標將有助了解叢集狀況，使用者可以利用取得的效能指標分析系統狀況，或者即時調整叢集負載狀況。然而，Kafka 的效能指標索取介面並非作為大量供給使用，叢集難以應付多個需求端索取叢集效能指標。

本研究提出效能指標收集與查詢系統，根據效能指標儲存的特性設計系統；利用效能指標讀取的特性，集中收集節點效能指標，以及利用 Kafka 對大數據串流的優化，降低每筆效能指標獲取的成本，使得本系統得以供給大量效能指標給予大量需求端，讓所有需求端即時且有效率的取得詳細 Kafka 叢集效能指標。

本文對於所提出之系統效能做估算，評估本系統最佳優化效益在於讀取效能指標的重複性和 Kafka 對大數據串流的優化。我們實驗結果顯示，在大量需求端請求及大量效能指標下使用我們的系統，效能指標需求端連線數明顯降低。

Apache Kafka is a widely used distributed event streaming platform due to its real-time streaming and storage capabilities for big data. Monitoring Kafka metrics will assist in comprehending the status of the cluster. Users can employ metrics to analyze the system or balance the cluster load on-the-fly. Nevertheless, the interface for Kafka's metric querying is not suited for mass provisioning, making it difficult for clusters to cope with numerous demands for cluster metrics.

This study presents a metrics collection and query system designed based on the storage and reading characteristics of metrics. The system concentrates on collecting metrics from nodes and benefits from Kafka's optimization of big data streaming to reduce costs associated with obtaining each metric. This enables the system to provide numerous metrics to demand-side parties and allows them to acquire detailed Kafka cluster metrics in an efficient, real-time manner.

This paper estimates the proposed system's performance and evaluates the best optimization benefits of the system in terms of the repetitiveness of reading performance metrics and the optimization of Kafka for big data streaming. Experimental results show that our system decreases the number of demand-side TCP socket connections significantly when used to handle a large number of demand-side requests and metrics.

[1] node_exporter. https://github.com/prometheus/node_exporter/.
[2] jmxtrans. https://github.com/jmxtrans/jmxtrans.
[3] Grafana. https://grafana.com/.
[4] Graphite. https://graphiteapp.org/.
[5] Prometheus. https://prometheus.io/.
[6] Apache Kafka. https://kafka.apache.org/.
[7] Protocol Buffers. https://protobuf.dev/.
[8] Monitoring and Management Using JMX Technology. https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html.
[9] OpenTSDB. https://github.com/OpenTSDB/opentsdb.
[10] InfluxDB. https://www.influxdata.com/.
[11] collectd. https://collectd.org/.
[12] Guenter Hesse, Christoph Matthies, and Matthias Uflacker. How fast can we insert? an empirical performance evaluation of apache kafka. 2020 IEEE 26th International Conference on Parallel and Distributed Systems (ICPADS), pages 641–648, 2020.
[13] R. Hofmann, R. Klar, B. Mohr, A. Quick, and M. Siegle. Distributed performance monitoring: methods, tools, and applications. IEEE Transactions on Parallel and Distributed Systems, 5(6):585–598, 1994.
[14] Jeffrey Joyce, Greg Lomow, Konrad Slind, and Brian Unger. Monitoring distributed systems. ACM Trans. Comput. Syst., 5(2):121–150, mar 1987.
[15] Jay Kreps, Neha Narkhede, Jun Rao, et al. Kafka: A distributed messaging system for log processing. In Proceedings of the NetDB, volume 11, pages 1–7. Athens, Greece, 2011.
[16] Yuansheng Lou, Lin Chen, Feng Ye, Yong Chen, and Zihao Liu. Research and implementation of an aquaculture monitoring system based on flink, mongodb and kafka. In João M. F. Rodrigues, Pedro J. S. Cardoso, Jânio Monteiro, Roberto Lam, Valeria V. Krzhizhanovskaya, Michael H. Lees, Jack J. Dongarra, and Peter M.A. Sloot, editors, Computational Science – ICCS 2019, pages 648–657, Cham, 2019. Springer International Publishing.
[17] Kim-Thomas Rehmann, Changyun Seo, Dongwon Hwang, Binh Than Truong, Alexander Boehm, and Dong Hun Lee. Performance monitoring in sap hana’s continuous integration process. SIGMETRICS Perform. Eval. Rev., 43(4):43–52, feb 2016.
[18] Gwen Shapira, Neha Narkhede, Todd Palino, 許致軒, 蔡政廷, and 李尚譯. Kafka 技術手冊｜即時資料與串流處理. 碁峰資訊股份有限公司, 台北市南港區三重路 66 號 7 樓之 6, 2019.
[19] Dimitris Stripelis, José Luis Ambite, Yao-Yi Chiang, Sandrah P. Eckel, and Rima Habre. A scalable data integration and analysis architecture for sensor data of pediatric asthma. In 2017 IEEE 33rd International Conference on Data Engineering (ICDE), pages 1407–1408, 2017.
[20] N Sudhakar Yadav, B Eswara Reddy, and KG Srinivasa. Cloud-based healthcare monitoring system using storm and kafka. Towards extensible and adaptable methods in computing, pages 99–106, 2018.
[21] Eno Thereska, Brandon Salmon, John Strunk, Matthew Wachs, Michael Abd-El-Malek, Julio Lopez, and Gregory R. Ganger. Stardust: Tracking activity in a distributed storage system. SIGMETRICS Perform. Eval. Rev., 34(1):3–14, jun 2006.
[22] Robbert Van Renesse, Kenneth P. Birman, and Werner Vogels. Astrolabe: A robust and scalable technology for distributed system monitoring, management, and data mining. ACM Trans. Comput. Syst., 21(2):164–206, may 2003.
[23] D. Wybranietz and D. Haban. Monitoring and performance measuring distributed systems during operation. SIGMETRICS Perform. Eval. Rev., 16(1):197–206, may 1988.