近年來在人工智慧、機器學習、影像辨識等領域愈加成熟，有越來越多的研究者投入此領域，也開始大量使用繪圖處理器來處理這些應用。本實驗室所開發的CASLab-GPGPU平台的目標應用即為上述領域的模型運算，而影像辨識中常使用卷積神經網路作為其模型，因此本論文對卷積神經網路的在運行CASLab-GPGPU情況分析評估，找出效能瓶頸的地方並對其優化。
本論文透過檢視CASLab-GPGPU執行卷積神經網路時的情況，發現在執行全連接層的表現並不理想，因此在Next-Line Prefetching的基礎上設計了Block-Base Prefetching來優化全連接層的執行效能，並且在CASLab-GPGPU的L2-Cache中實現。Next-Line Prefetching在增加Prefetch Degree時有可能會發生預取到重覆資料的缺陷，因此只能對下一個Cache Line進行預取，而本論文設計的Block-Base Prefetching透過將記憶體切分為多個記憶體區塊，將區塊中的存取進度紀錄於Memory Block History Table，有了存取進度紀錄，就算提高Prefetch Degree也能透過查詢紀錄來避免預取到重覆的資料。而在全連接層的執行測試結果中，使用Block-Base Prefetching的效能可以高達原本的2.6倍之多，L2-Cache Hit Rate也提升將近45%。而在各卷積神經網路的測試中，使用Block-Base Prefetching也可以帶來平均70%的效能增加，與Next-Line Prefetching相比可以說是效果卓越。

The target of CASLab-GPGPU is AI (Artificial Intelligence) application, and CNN (Convolutional Neural Network) is a kind of popular network in image recognition. We analyze the execution of classical CNN LeNet-5 on CASLab-GPGPU. Fully connected layer take a large part in the total execution time of LeNet-5, and L2-cache hit rate of fully connected layer is much lower than other layers of LeNet-5. It spends plenty of time to access DRAM because of low L2-cache hit rate. We propose a Block-Base prefetching scheme to increase the L2-cache hit rate for improve the performance of fully connected layer, which can prefetch more than one cache line once and avoid prefetching repeated cache line into cache. Any access including prefetching access will be recorded into a memory block history table. The prefetcher can get access the memory block via memory block history table to avoid prefetching the cache line already accessed before. The experimental results for fully connected layer show that Block-Base prefetching can increase 160% performance. And the experimental results of different CNN show that Block-Base prefetching can bring over 60% performance improvement. All these experimental results show Block-Base prefetching can make CASLab-GPGPU more efficient when executing CNN model.

[1] Y. Lecun, L. Bottou, Y. Bengio, and P.Haffner, “Gradient-based learning applied to document recognition,” in Proc. IEEE, vol. 86, no. 11, 1998.
[2] A. Krizhevsky, I. Sutskever, and G.E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,”, Adv. Neural Inf. Process. Syst., 2012.
[3] K. Simonyan, and A. Zisserman, "Deep Convolutional Networks for Large-Scale Image Recognition,” in ICLR, 2015.
[4] A. G. Howard, M. Zhu,. B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. “MobileNet : Efficient Convolutional Neural Networks for Mobile Applications,” arXiv:1704.04861, 2017
[5] S. Ding, F. Long, H. Fan, L. Liu, and Y. Wang, “A novel YOLOv3-tiny network for unmanned airship obstacle detection,” in IEEE 8th Data Driven Control Learn. Syst. Conf. DDCLS, 2019
[6] He K, Zhang X, Ren S, et al. “Deep residual learning for image recognition,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016
[7] J. Lee, N. B. Lakshminarayana, H. Kim, and R. Vuduc, “Many-thread aware prefetching mechanisms for gpgpu applications,” in 43rd Annual IEEE/ACM International Symposium on Microarchitecture, 2010
[8] A. Sethia, G. Dasika, M. Samadi, and S. Mahlke, “APOGEE: Adaptive prefetching on GPUs for energy efficiency,” in Proceedings of the 22nd International Conference on Parallel Architectures and Compilation Techniques, 2013
[9] K.J. Nesbit, J.E. Smith, “Data Cache Prefetching Using a Global History Buffer,” in International Symposium on High Performance Computer Architecture (HPCA'04), 2004
[10] J. W. C. Fu, J. H. Patel, and B. L. Janssens, “Stride directed prefetching in scalar processors,” in Proceedings the 25th Annual International Symposium on Microarchitecture (MICRO 25), 1992
[11] T-F. Chen and J.-L. Baer, “Effective hardware-based data prefetching for high-performance processors,” in IEEE Transactions on Computers, 1995
[12] A. Lashgar, E. Salehi, A. Baniasadi, “Understanding Outstanding Memory Request Handling Resources in GPGPUs,”in The Sixth International Symposium on Highly Efficient Accelerators and Reconfigurable Technologies (HEART 2015), 2015
[13] S.-Y. Lee, C.-J. Wu, “Characterizing the latency hiding ability of GPUs,” in IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), 2014
[14] H. Guo, L. Huang, Y. Y. Lü, J. Ma, C. Qian, S. Ma, Z. Wang, “Accelerating BFS via Data Structure-Aware Prefetching on GPU,” in IEEE Access, 2018
[15] V. Narasiman, M. Shebanow, C. J. Lee, R. Miftakhutdinov, O. Mutlu, Y. N. Patt, "Improving GPU performance via large warps and two-level warp scheduling," in 44th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2011
[16] W. Jia, K. A. Shaw, M. Martonosi, “Characterizing and improving the use of demand-fetched caches in GPUs,” in 26th ACM international conference on Supercomputing, 2012
[17] A. Bakhoda, G. L. Yuan, W. W. L. Fung, H. Wong, T. M. Aamodt, “Analyzing CUDA workloads using a detailed GPU simulator,” in IEEE International Symposium on Performance Analysis of Systems and Software, 2009
[18] “THE MNIST DATABASE,” [Online]. Available: http://yann.lecun.com/exdb/mnist/
[19] “The CIFAR-10 dataset,” [Online]. Available: https://www.cs.toronto.edu/~kriz/cifar.html
[20] D. Joseph, D. Grunwald, “Prefetching Using MarkovPredictors,” in IEEE Transactions on Computer Systems, 1999
[21] G.B. Kandiraju, A. Sivasubramaniam, “Going the distance for TLB prefetching: an application-driven study,” in Proceedings 29th Annual International Symposium on Computer Architecture, 2002