Transformer 模型中的 Softmax、GELU、SiLU 與層正規化（Layer Normalization）中的倒數平方根（reciprocal square root, RSQRT）皆涉及指數、對數、除法或開根號等高成本運算。雖然這些非線性函數的運算量低於矩陣乘法，但在量化 Transformer 加速器中仍可能造成顯著的硬體面積與延遲開銷。既有非線性函數硬體設計多透過查表法、分段線性近似、多項式近似或可重構架構降低硬體成本；然而，這些方法仍常受限於函數支援範圍、長序列 Softmax 的輸入擴展性，以及不同模型動態範圍下的精度彈性。
本論文提出一種精度可配置之可重構非線性函數硬體架構，延伸先前具線上正規化器（Online Normalizer）之 Softmax/GELU 架構與統一非線性函數可重構架構。本研究採用逐次逼近（successive approximation）技術降低基於 S 型函數（sigmoid）之 GELU 與 SiLU 的近似誤差，並整合線上正規化（Online Normalization）以改善長序列 Softmax 在有限硬體平行度下的資料相依性。此外，本論文提出具縮放因子（scale factor）之動態定點數資料路徑，將縮放因子融入對數與指數近似運算，使輸入與輸出可依不同函數需求調整可表示範圍與小數解析度。所提出架構亦支援 16/32-bit 多精度運算，並將 GELU/SiLU 的輸出乘法轉換至對數域，以降低多精度架構中的乘法器成本。
實驗結果顯示，本論文之非線性函數近似在 BERT-base 上能維持接近基準模型的推論準確率；在 LLaMA3.2-1B 上，Softmax 與 SiLU 於 16-bit 模式下可維持可接受的困惑度（perplexity, PPL），而倒數平方根需使用 32-bit 高精度運算模式，以避免高動態範圍輸入造成嚴重失真。硬體方面，本設計以 TSMC 40-nm 製程實現，於 500 MHz 下合成面積為 38,217 μm²。相較於先前發表之架構，本論文以額外硬體成本整合線上正規化、可配置縮放機制與多精度支援，同時透過無專用輸出乘法器之 GELU/SiLU 運算路徑降低乘法器成本，建立更具彈性的非線性函數運算核心。

Softmax, GELU, SiLU, and the reciprocal square root (RSQRT) operation in Layer Normalization are essential nonlinear functions in Transformer models. These functions involve costly operations such as exponentiation, logarithms, division, and square root computation. Although their operation count is lower than that of matrix multiplications, they can still introduce significant area and latency overhead in quantized Transformer accelerators. Existing hardware designs reduce nonlinear-function cost through look-up tables, piecewise linear approximation, polynomial approximation, or reconfigurable architectures. However, they are often limited by function coverage, input scalability for long-sequence Softmax, and precision flexibility under model-dependent dynamic range.
This thesis proposes a Precision-Aware Reconfigurable Architecture for nonlinear functions, extending a previous Online-Normalizer-based Softmax/GELU architecture and a unified reconfigurable nonlinear-function architecture. Successive Approximation is adopted to reduce the approximation errors of sigmoid-based GELU and SiLU, while Online Normalization is integrated to reduce the data dependency of long-sequence Softmax under limited hardware parallelism. In addition, a scaling-factor-aware dynamic fixed-point datapath is proposed to incorporate scaling factors into logarithmic and exponential approximations, allowing the representable range and fractional resolution of inputs and outputs to be configured according to different function requirements. The architecture also supports 16/32-bit multi-precision computation and transforms GELU/SiLU output multiplication into the log domain to reduce multiplier cost in the multi-precision architecture.
Experimental results show that the proposed nonlinear-function approximations maintain near-baseline inference accuracy on BERT-base. On LLaMA3.2-1B, Softmax and SiLU achieve acceptable perplexity in 16-bit mode, while RSQRT requires the 32-bit high-precision computation mode to avoid severe distortion caused by high-dynamic-range inputs. The proposed architecture is synthesized using a TSMC 40-nm process, achieving an area of 38,217 μm² at 500 MHz. Compared with previously published architectures, this thesis integrates Online Normalization, configurable scaling, and multi-precision support at the cost of additional hardware resources, while reducing multiplier cost through the dedicated-multiplier-free GELU/SiLU computation path.

[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30:5998–6008, 2017.
[2] Alberto Marchisio, Davide Dura, Maurizio Capra, Maurizio Martina, Guido Masera, and Muhammad Shafique. SwiftTron: An efficient hardware accelerator for quantized transformers. In 2023 International Joint Conference on Neural Networks (IJCNN), pages 1–9. IEEE, 2023.
[3] Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. I-BERT: Integer-only BERT quantization. In Proceedings of the 38th International Conference on Machine Learning, pages 5506–5518. PMLR, 2021.
[4] Zhikai Li and Qingyi Gu. I-ViT: Integer-only quantization for efficient vision transformer inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17065–17075. IEEE/CVF, 2023.
[5] Gamze İslamoğlu, Moritz Scherer, Gianna Paulin, Tim Fischer, Victor J. B. Jung, Angelo Garofalo, and Luca Benini. ITA: An energy-efficient attention and softmax accelerator for quantized transformers. In 2023 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), pages 1–6. IEEE, 2023.
[6] Jacob R. Stevens, Rangharajan Venkatesan, Steve Dai, Brucek Khailany, and Anand Raghunathan. Softermax: Hardware/software co-design of an efficient softmax for transformers. In 2021 58th ACM/IEEE Design Automation Conference (DAC), pages 469–474. IEEE, 2021.
[7] Meiqi Wang, Siyuan Lu, Danyang Zhu, Jun Lin, and Zhongfeng Wang. A high-speed and low-complexity architecture for softmax function in deep learning. In 2018 IEEE Asia Pacific Conference on Circuits and Systems (APCCAS), pages 223–226. IEEE, 2018.
[8] Hongxi Dong, Manzhen Wang, Yuanyong Luo, Muhan Zheng, Mengyu An, Yajun Ha, and Hongbing Pan. PLAC: Piecewise linear approximation computation for all nonlinear unary functions. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 28(9):2014–2027, 2020.
[9] Xiao Wu, Shuang Liang, Meiqi Wang, and Zhongfeng Wang. ReAFM: A reconfigurable nonlinear activation function module for neural networks. IEEE Transactions on Circuits and Systems II: Express Briefs, 70(7):2660–2664, 2023.
[10] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415, 2016.
[11] Tianyang Li, Fan Zhang, Guangwei Xie, Xitian Fan, Yanzhao Gao, and Mingqian Sun. A high speed reconfigurable architecture for softmax and GELU in vision transformer. Electronics Letters, 59(5), 2023.
[12] Christodoulos Peltekis, Kosmas Alexandridis, and Giorgos Dimitrakopoulos. Reusing softmax hardware unit for GELU computation in transformers. In 2024 IEEE 6th International Conference on AI Circuits and Systems (AICAS), pages 159–163. IEEE, 2024.
[13] Maxim Milakov and Natalia Gimelshein. Online normalizer calculation for softmax. arXiv preprint arXiv:1805.02867, 2018.
[14] Qi-Xian Wu, Shu-Sian Teng, Ming-Der Shieh, Chih-Tsun Huang, and Juin-Ming Lu. A low-cost reconfigurable architecture for efficient softmax and GELU in transformers. In 2025 International VLSI Symposium on Technology, Systems and Applications (VLSI-TSA), pages 1–4. IEEE, 2025.
[15] Qi-Xian Wu, Shu-Sian Teng, Ming-Der Shieh, Chih-Tsun Huang, and Juin-Ming Lu. A low-complexity and reconfigurable design for nonlinear function approximation in transformers. IEEE Transactions on Circuits and Systems II: Express Briefs, 72(12):1882–1886, 2025.
[16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics, 2019.
[17] Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3–11, 2018.
[18] Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
[19] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
[20] Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023.