針對 Vision Transformer（ViT）高參數量與運算複雜度，設計專用神經網路處理器（Neural Processing Unit, NPU）為提升推論效率之重要方向。本實驗室開發之 Novella Brown NPU 已於核心運算單元導入微縮放整數格式（Microscaling Integer, MXInt），以提升線性運算效率。然而，現有架構在處理 LayerNorm、Softmax 與 GeLU 等 ViT 主要非線性函數時，後處理單元（PPU）仍依賴 BFloat16 浮點資料路徑，需配置成本較高的浮點運算單元，增加硬體面積負擔。此外，MXInt 及其他共享指數格式多應用於矩陣乘法與卷積等線性算子，對 LayerNorm、Softmax 與 GeLU 等 ViT 主要非線性函數的系統化支援仍相對有限。
本論文提出一套基於 MXInt 格式之全整數非線性函數近似架構，並實現於 Novella NPU 之硬體設計中。本研究針對 LayerNorm、Softmax 與 GeLU 設計 MXInt 近似演算法，將原先依賴浮點資料路徑的非線性計算轉換為純尾數域之整數運算，使所提出之非線性運算路徑可由整數運算單元支援。針對反平方根、倒數、指數函數與 GeLU 等特殊函數，本研究設計 MXInt-ISQRT、MXInt-RECIP、MXInt-EXP 與 MXInt-GeLU 等可重用運算子，並以整數資料路徑與小容量查找表（Look-Up Table, LUT）共同支援。透過演算法與硬體共同設計，本架構以整數運算單元與小容量 LUT 取代原有浮點運算單元，降低 PPU 支援 ViT 非線性函數所需的硬體成本。
本研究透過 AlgoSim 模擬 DeiT-Tiny 於 ImageNet-1K 進行模型層級驗證，且未重新訓練或微調模型。實驗結果顯示，所提出之 MXInt 非線性函數近似演算法相較於 FP32 baseline 僅造成 0.13% Top-1 準確度損失與 0.06% Top-5 準確度損失，顯示本方法在模型層級具有可行性。硬體層級方面，相較於 BFloat16 設計，單一 MXInt kernel 面積下降 24.91%，並使整體 PPU 面積由 173,181.74 μm² 降至 156,163.5441 μm²，下降 9.83%。實驗結果顯示，MXInt 資料格式可有效延伸至 ViT 主要非線性函數，在維持模型準確度的同時，降低 Novella NPU 中 PPU 支援非線性函數所需之硬體面積成本。

Vision Transformers (ViTs) have high parameter counts and computational complexity, motivating dedicated neural processing units (NPUs) for efficient inference. The Novella Brown NPU developed by our laboratory already uses the Microscaling Integer (MXInt) format in its core compute unit to improve the efficiency of linear operators. However, its Post-Processing Unit (PPU) still relies on a BFloat16 (BF16) datapath for major ViT nonlinear functions, including LayerNorm, Softmax, and GeLU. The required floating-point arithmetic units increase the PPU area. Moreover, MXInt and other shared-exponent formats have mainly been applied to linear operators, while systematic support for these nonlinear functions remains limited.
This thesis proposes an all-integer MXInt nonlinear function approximation architecture and implements it in the Novella NPU. The proposed LayerNorm, Softmax, and GeLU algorithms convert calculations that originally depend on a floating-point datapath into integer operations in the mantissa domain. MXInt-ISQRT, MXInt-RECIP, MXInt-EXP, and MXInt-GeLU are designed as reusable operators supported by integer arithmetic and small look-up tables (LUTs). Through algorithm-hardware co-design, the proposed architecture replaces the original floating-point arithmetic units with integer arithmetic units and small LUTs.
DeiT-Tiny inference on ImageNet-1K is evaluated in AlgoSim without retraining or fine-tuning. Relative to the FP32 baseline, the proposed MXInt nonlinear approximation causes only 0.13% Top-1 accuracy drop and 0.06% Top-5 accuracy loss. Compared with the BF16 design, the area of a single nonlinear kernel is reduced by 24.91%, resulting in a 9.83% reduction in the overall PPU wrapper area. These results show that MXInt can be extended to the major ViT nonlinear functions while maintaining model accuracy and reducing the PPU hardware area.

[1] Bita Darvish Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, Alessandro Forin, Haishan Zhu, Taesik Na, Prerak Patel, Shuai Che, Lok Chand Koppaka, Xia Song, Subhojit Som, Kaustav Das, Saurabh Tiwary, Steve Reinhardt, Sitaram Lanka, Eric Chung, and Doug Burger. Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point. In Advances in Neural Information Processing Systems, volume 33, pages 10271–10281, 2020.
[2] Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, Stosic Dusan, Venmugil Elango, Maximilian Golub, Alexander Heinecke, Phil James-Roxby, Dharmesh Jani, Gaurav Kolhe, Martin Langhammer, Ada Li, Levi Melnick, Maral Mesmakhosroshahi, Andres Rodriguez, Michael Schulte, Rasoul Shafipour, Lei Shao, Michael Siu, Pradeep Dubey, Paulius Micikevicius, Maxim Naumov, Colin Verrilli, Ralph Wittig, Doug Burger, and Eric Chung. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023.
[3] Peiyan Dong, Jinming Zhuang, Zhuoping Yang, Shixin Ji, Yanyu Li, Dongkuan Xu, Heng Huang, Jingtong Hu, Alex K. Jones, Yiyu Shi, Yanzhi Wang, and Peipei Zhou. Eq-vit: Algorithm-hardware co-design for end-to-end acceleration of real-time vision transformer inference on versal acap architecture. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 43(11):3949–3960, 2024.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[5] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
[6] Erik Hertz, Niclas Thuning, Leo Bärring, Bertil Svensson, and Peter Nilsson. Algorithms for implementing roots, inverse and inverse roots in hardware. Unpublished manuscript in dissertation, 2016.
[7] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2704–2713, 2018.
[8] Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. A study of bfloat16 for deep learning training. arXiv preprint arXiv:1905.12322, 2019.
[9] Piotr Kluska, Adrián Castelló, Florian Scheidegger, A. Cristiano I. Malossi, and Enrique S. Quintana-Ortí. QAttn: Efficient gpu kernels for mixed-precision vision transformers. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3648–3657, 2024.
[10] Zhengang Li, Mengshu Sun, Alec Lu, Haoyu Ma, Geng Yuan, Yanyue Xie, Hao Tang, Yanyu Li, Miriam Leeser, Zhangyang Wang, Xue Lin, and Zhenman Fang. Auto-vit-acc: An fpga-aware automatic acceleration framework for vision transformer with mixed-scheme quantization. In 2022 32nd International Conference on Field-Programmable Logic and Applications (FPL), pages 109–116, 2022.
[11] Zhikai Li and Qingyi Gu. I-ViT: Integer-only quantization for efficient vision transformer inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. arXiv:2207.01405.
[12] Yang Lin, Tianyu Zhang, Peiqin Sun, Zheng Li, and Shuchang Zhou. FQ-ViT: Post-training quantization for fully quantized vision transformer. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, pages 1173–1179, 2022.
[13] Siyuan Lu, Meiqi Wang, Shuang Liang, Jun Lin, and Zhongfeng Wang. Hardware accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the transformer. In 2020 IEEE 33rd International System-on-Chip Conference (SOCC), pages 84–89, 2020.
[14] Jacob R. Stevens, Rangharajan Venkatesan, Steve Dai, Brucek Khailany, and Anand Raghunathan. Softermax: Hardware/software co-design of an efficient softmax for transformers. In 2021 58th ACM/IEEE Design Automation Conference (DAC), pages 469–474, 2021.
[15] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10347–10357. PMLR, 2021.
[16] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
[17] Chin-Cheng Wei, Chun-Han Wang, En-Yu Pong, and Chung-Ho Chen. Low-cost non-linear function approximation for transformer deployment on edge NPU. In 2025 1st International Conference on Consumer Technology (ICCT-Pacific), pages 1–4, 2025.
[18] Can Xiao, Jianyi Cheng, and Yiren Zhao. Refining datapath for microscaling ViTs. In 2025 35th International Conference on Field-Programmable Logic and Applications (FPL), pages 263–272, 2025.
[19] 彭恩宇. 非線性函數近似及其於TVM編譯器的NPU運算子合法化. Master's thesis, 國立成功大學電機工程學系, 2025.