隨著人工智慧技術的快速發展，邊緣運算裝置在模型推論中的應用日益重要。本研究聚焦於專為深度學習推論設計的 NPU （神經處理單元），基於本實驗室的Novella-NPU 硬體架構，結合編譯器優化與指令排程技術，以提升運算效率與資源利用率。本文透過編譯器實現張量運算的指令生成，支援從模型到 Novella-NPU 的端到端執行，並基於雙緩衝（Double Buffer）技術優化資料讀寫與計算平衡，顯著提升運算性能。此外，針對 N:M 準結構化稀疏矩陣運算，設計新的計算模式與指令排程，進一步提高執行效率。實驗結果顯示，透過指令相依性分析與雙緩衝優化，MobileNetv2 的 FPS 從 84.99 提升至 105.05，DeiT-tiny 模型的 FPS 從 6.82 提升至7.30。N:M 稀疏矩陣運算支援則在 DeiT 系列模型中實現最高 1.8 倍的加速比，並降低記憶體使用量。本研究不僅驗證了 Novella-NPU 與編譯器設計的有效性，也為邊緣運算中的高效深度學習推論提供了實用解決方案，未來可進一步探索硬體平行度與模型優化的結合。

With the rapid advancement of artificial intelligence, edge computing devices play an increasingly critical role in model inference. This study focuses on the design of a Neural Processing Unit (NPU) tailored for deep learning inference, based on the Novella-NPU hardware architecture. By integrating compiler optimizations and instruction scheduling techniques, we enhance computational efficiency and resource utilization. The proposed compiler enables end-to-end execution from model to Novella-NPU through optimized tensor operation instruction generation. Leveraging double buffering, we balance data access and computation, significantly improving performance. Additionally, we introduce a novel computation mode and instruction scheduling for N:M semi-structured sparse matrix operations, further boosting efficiency. Experimental results demonstrate that, through instruction dependency analysis and double buffering, the FPS of MobileNetv2 increases from 84.99 to 105.05, and DeiT-tiny improves from 6.82 to 7.30. The N:M sparse matrix support achieves up to a 1.8× speedup on DeiT models while reducing memory usage. This work validates the effectiveness of the Novella-NPU and its compiler, offering a practical solution for efficient deep learning inference on edge devices. Future work will explore further integration of hardware parallelism and model optimization.

[1] Y. Tai. Design of a kernel-agnostic compute core for convolution and gemm. Master’s thesis, NCKU, 2024.
[2] P.-H. Lee. Memory layout unit design of cnn/transformer unified accelerator and memory subsystem analysis. Master’s thesis, NCKU, 2024.
[3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
[4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoeit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), pages 5998–6008, 2017.
[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[6] W.-L. Lo. Compiler-hardware co-optimization on novella-npu, and the design of msfp-supported compute core. Master’s thesis, NCKU, 2025.
[7] T.-J. Wu. A one-dimensional convolution accelerator supporting data reuse and multiple dimensional filters. Master’s thesis, NCKU, 2020.
[8] C.-C. Hsiao. Quantization implementation for neural network accelerator based on cnn inference engine. Master’s thesis, NCKU, 2021.
[9] H.-Q. Huang. Integration of machine learning compiler framework with custom instruction set architecture for caslab-dla. Master’s thesis, NCKU, 2022.
[10] C.-Y. Xie. Optimization of cnn micro-architecture and memory sub-system for caslab-dla. Master’s thesis, NCKU, 2022.
[11] T.-Y. Lo. Compressed sparse convolution hardware and software co-design on caslab-dla. Master’s thesis, NCKU, 2023.
[12] H.-Y. Wang. Instruction scheduling optimization for convolution neural network on scalable caslab-dla –tvm system. Master’s thesis, NCKU, 2023.
[13] Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Meghan Cowan, Haichen Shen, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. Tvm: An automated end-to-end optimizing compiler for deep learning. In Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’18), pages 578–594, Carlsbad, CA, USA, 2018. USENIX Association.
[14] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Advances in Neural Information Processing Systems, 34:21620–21632, 2021.
[15] Jeff Pool, Abhishek Sawarkar, and Jay Rodge. Accelerating sparse deep neural networks.arXiv preprint, 2021.
[16] E.-Y. Pong. Non-linear function approximations and their npu operator legalization in tvm compiler. Master’s thesis, NCKU, 2025.