在2012年的ImageNet大規模視覺辨識大賽(ImageNet Large Scale Visual Recognition Competition, ILSVRC)中， AlexNet首度採用深度學習(deep learning)並獲得冠軍，使卷積神經網路(convolution neural network, CNN)開始受到矚目，卷積神經網路在往後的圖像辨識和其他電腦視覺應用都有不錯的成果。但隨著網路模型深度越深，計算成本也越高，對於有能量限制的裝置是一個需要克服的問題，因此許多研究致力於實現可以降低運算複雜度的卷積神經網路模型。
本篇論文基於AlexNet的架構，將卷積層(convolution layer)改為定點數(fixed-point number)運算，並使用較少位元數運算的結果，即低精度的權重與激發(activations)的卷積結果，判斷經過線性整流函數(rectified linear unit, ReLU)前的數值是否為負數來尋找理想的閥(閾)值(threshold)，力求訓練一個在硬體中能降低運算量，但能維持原本卷積神經網路之準確度的神經網路模型。
關鍵字：深度學習、卷積神經網路、AlexNet

After AlexNet adopted deep learning and won the champion in the 2012 ImageNet Large Scale Visual Recognition Competition (ILSVRC), convolution neural network (CNN) began to attract attention. CNN has also achieved good results in image recognition and other computer vision applications. However, as the network model becomes deeper, the computational cost becomes higher, which is a problem that needs to be overcome for devices with energy limitations. Hence, many studies are dedicated to implementing the convolution neural network model which can reduce the computational complexity.
This paper is based on the AlexNet architecture. We quantify the input data of the convolution layer to fixed-point number, use computational result of less bits, the convolution result of low bitwidth weights and activations, and determine the value before rectified linear unit (ReLU) is negative or not to find the ideal threshold. We strive to train a neural network model to reduce the amount of calculation in the hardware and achieve same accuracy level of the original convolution neural network.
Keywords: deep learning, convolution neural network, AlexNet

[1] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet large scale visual recognition challenge” in International Journal of Computer Vision, pp. 211-252, 2015.
[2] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driess- che, J. Schrittwieser, I. Antonoglou, V. Panneer shelvam, M. Lanctot, et al., “Mastering the game of go with deep neural networks and tree search” in Nature, vol. 529, no. 7587, pp. 484-489, 2016.
[3] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition", in International Conference on Learning Representations (ICLR), pp. 1-14, 2015.
[4] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition” in IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
[5] J. Redmon, S. Divvalal, R. Girshick and A. Farhadi, "You only look once: Unified real-time object detection", in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779-788, 2016.
[6] J. Redmon and A. Farhadi, "YOLO9000: Better faster stronger", in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7263-7271, 2017.
[7] A. Krizhevsky, I. Sutskever, and G. Hinton, “Imagenet classification with deep convolutional neural networks” in Advances in Neural Information Processing Systems (NIPS), pp. 1097-1105, 2012.
[8] M. Courbariaux, Y. Bengio, and J.-P. David, “BinaryConnect: Training deep neural networks with binary weights during propagations,” in Advances in Neural Information Processing Systems (NIPS), pp. 3123–3131, 2015.
[9] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, “XNOR-Net: ImageNet classification using binary convolutional neural networks,” in European Conference on Computer Vision (ECCV), pp. 525–542, 2016.
[10] F. Li and B. Liu, “Ternary weight networks,” in Proc. NIPS Workshop Efficient Methods Deep Neural Network, 2016.
[11] C. Zhu, S. Han, H. Mao, and W. J. Dally, “Trained ternary quantization,” in International Conference on Learning Representations (ICLR), 2017.
[12] M. Courbariaux and Y. Bengio, “Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or −1,” 2016, [Online] Available: https://arxiv.org/ abs/1602.02830
[13] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, “Quantized neural networks: Training neural networks with low precision weights and activations,” 2016, [Online] Available: https:// arxiv.org/abs/1609.07061
[14] Z. Cai, X. He, J. Sun, and N. Vasconcelos, “Deep learning with low precision by halfwave Gaussian quantization,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[15] S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou, “DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients,” 2016, [Online]. Available: https://arxiv.org/ abs/1606.06160
[16] P. Gysel, M. Motamedi and S. Ghiasi, “Hardware-oriented approximation of convolutional neural networks,” in International Conference on Learning Representations (ICLR), 2016.
[17] Y. Ma, N. Suda, Y. Cao, J.-S. Seo and S. Vrudhula, "Scalable and modularized RTL compilation of convolutional neural networks onto FPGA", in Proc. FPL, pp. 1-8, 2016.
[18] E. H. Lee, D. Miyashita, E. Chai, B. Murmann, and S. S. Wong, “LogNet: Energy-efficient neural networks using logrithmic computations,” in Proc. ICASSP, pp. 5900-5904, 2017.
[19] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” in Proc. ICLR, 2016.
[20] V. Sze, Y. H. Chen, T. J. Yang, and J. S. Emer, “Efficient processing of deep neural networks: A tutorial and survey,” in Proc. IEEE, pp. 2295-2329, 2017.
[21] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” in Proc. IEEE, 1998, pp. 2278-2324.
[22] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in Proc. ICLR, 2015.
[23] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Proc. CVPR, 2015, pp. 1-9.
[24] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. CVPR, 2016, pp. 770-778.
[25] Y. LeCun, et al. (1998). The MNIST database. [Online]. Available: http://yann.lecun.com/exdb/mnist/
[26] Sekitoshi Kanai, Yasuhiro Fujiwara, Yuki Yamanaka, and Shuichi Adachi, “Sigsoftmax: Reanalysis of the softmax bottleneck,” in Advances in Neural Information Processing Systems (NIPS), 2018.
[27] 蕭人豪 (2020)。利用串列輸入與預測門閥降低類神經網路卷積層能量消耗之硬體實現。國立成功大學工程科學研究所碩士論文，台南市。