現今，以安全為導向的即時作業系統需求正在日益增加。雖然有許多不同基準的測試技術，但其評估標準並不相同。為了更明確的比較即時作業系統以安全為導向的性能，本研究將評估標準分成兩個部分：重複使用性（Reusability）、及穩健性（Robustness）。重複使用性會針對即時作業系統是否便於移植，撰寫、編譯、執行時遇到的問題以及即時作業系統的特點為基準；而穩健性則是作業系統能夠在程式在編譯時能夠偵測出錯誤輸入並給予回應。本研究將穩健性評估標準分成了五個層級。這篇論文比較了FreeRTOS 、ChibiOS與NuttX，透過系統支援的應用程式介面，給予不同的輸入，觀察作業系統的回饋。最後將這三個作業系統根據評估標準給予1到5的評分。評估的結果顯示，NuttX表現最好。

Nowadays, the need for safety-critical real-time operating systems is growing. Although there are many different benchmarks for real-time operating systems, their evaluation criteria are not the same. This research is to have an evaluation on the safety-critical property of real-time operating systems. These evaluation criteria are divided into two features: reusability, and robustness. Reusability is based on portability, the problems encountered in coding, compiling and executing the operating systems and their characteristics. Robustness is the ability of the operating system to detect and respond to invalid input at compile time. The evaluation level of robustness is divided into five levels. Open-source FreeRTOS, ChibiOS, and NuttX are compared. Different inputs are given through the system application program interface to observe the feedback of the operating system. Finally, these three operating systems are given a score of 1 to 5 according to the evaluation criteria. The evaluation results show that NuttX is the best.

[1] R. P.Kar, “Implementing the {Rhealstone} Real-Time Benchmark,” Dr. Dobb’s J. Softw. Tools, vol. 15, no. 4, pp. 46–55, 1990.
[2] M. R.Guthaus, J. S.Ringenberg, D.Ernst, T. M.Austin, T.Mudge, andR. B.Brown, “MiBench: A Free, Commercially Representative Embedded Benchmark Suite,” 2001 IEEE Int. Work. Workload Charact. WWC 2001, pp. 3–14, 2001.
[3] D.Quarta, M.Pogliani, M.Polino, F.Maggi, A. M.Zanchettin, andS.Zanero, “An Experimental Security Analysis of an Industrial Robot Controller,” 2017 IEEE Symp. Secur. Priv., pp. 268–286, 2017.
[4] R. P.Weicker, “Dhrystone: a Synthetic Systems Programming Benchmark,” Commun. ACM, vol. 27, no. 10, pp. 1013–1030, 1984.
[5] H. J.Curnow andB. A.Wichmann, “Synthetic Benchmark.,” Comput. J., vol. 19, no. 1, pp. 43–49, 1976.
[6] J. A.Poovey, T. M.Conte, M.Levy, andS.Gal-On, “A Benchmark Characterization of the EEMBC Benchmark Suite,” IEEE Micro, vol. 29, no. 5, pp. 18–29, 2009.
[7] B. P.Miller et al., “Fuzz Revisited - A Re-examination of the Reliability of UNIX Utilities and Services,” Report, University of Wisconsin, 1995.
[8] D.Tang andR. K.Iyer, “FINE: A Fault Injection and Monitoring Environment for Tracing the UNIX System Behavior under Faults,” IEEE Trans. Softw. Eng., vol. 19, no. 11, pp. 1105–1118, 1993.
[9] Z.-M.Zhou, Z.-R.Zhu, andM.Cai, “Designing an Efficient and Extensible Robustness Benchmark of a Real-Time Operating System,” Cybern. Inf. Technol., vol. 15, no. 1, pp. 84–103, 2015.
[10] F.Maggi et al., “Rogue Robots: Testing the Limits of an Industrial Robot’s Security,” Technical report, Trend Micro, Politecnico di Milano, 2017.
[11] ISO, Safety of machinery - Safety-related parts of control systems. 2015.
[12] ISO/TC 199 Safety of machinery, ISO 12100:2010 Safety of Machinery - General Principles for Design - Risk Assessment and Risk Reduction. 2010.
[13] R. A.Stephans, System Safety for the 21st Century. Wiley-Interscience, 2004.
[14] National Aeronautics and Space Administration, NASA Software Safety Guidebook. 2004.
[15] R. H.Pierce, “Preliminary Assessment of Linux for Safety Related Systems.,” In: HSE Contract research report RR011/2002, 2002. [Online]. Available: http://www.hse.gov.uk/research/rrhtm/rr011.htm.
[16] G.LeLann, “An Analysis of the Ariane 5 Flight 501 Failure - A System Engineering Perspective,” IEEE Eng. Comput. Syst., pp. 339–346, 1997.
[17] A.Geraci, F.Katki, L.McMonegal, B.Meyer, andH.Porteous, “IEEE Standard Computer Dictionary. A Compilation of IEEE Standard Computer Glossaries,” in IEEE Std 610, 1991, p. 64.
[18] N. H.Gehani, “Exceptional C or C with exceptions,” Softw. Pract. Exp., vol. 22, no. 10, pp. 827–848, 1992.
[19] Z.Zhengmao, Z.Yun, andC.Ming, “An Approach to Evaluate the Robustness of Real Time Operating System,” in 2012 2nd International Conference on Computer Science and Network Technology, Zhangjiajie, China, 2012, pp. 306–310.
[20] Z.Zhengmao, Z.Yun, C.Ming, andS.Lei, “A Workload Model Based Approach to Evaluate the Robustness of Real-time Operating System,” in 2013 IEEE International Conference on High Performance Computing and Communications & 2013 IEEE International Conference on Embedded and Ubiquitous Computing, Changchun, China, 2013, pp. 2027–2033.
[21] R.Barry, “Mastering the FreeRTOSTM Real Time Kernel A Hands-On Tutorial Guide,” 2016. [Online]. Available: https://www.freertos.org/Documentation/RTOS_book.html.
[22] G.DiSirio, “ChibiOS/RT 5.0.0 Reference Manual,” 2018. [Online]. Available: http://chibios.sourceforge.net/docs3/rt/index.html.
[23] G.Nutt, “NuttX Documentation,” 2018. [Online]. Available: http://nuttx.org/doku.php?id=documentation.