深度卷積神經網路的剪枝方法被廣泛地應用於降低深度學習模型在運算預算受限之儀器上的推理時間花費與記憶體消耗。目前大部分的剪枝演算法都基於一個特定的剪枝標準來估計模型內的一個通道或一層卷積層內之所有權重的聯合重要性，並移除其中冗餘的通道或卷積層。但是，在下一層卷積層中，與此冗餘之通道相乘的卷積核也會一併的丟失，而卻沒有符合所設定的剪枝標準。因此，該研究提出一種具恢復性的剪枝演算法，利用了同一層卷積層中的其他另外一個通道來還原被移除的卷積核以近似下一層卷積層原本的輸出值。在所提出的演算法中，使用了同一層中任兩個通道之間的皮爾森相關係數來決定具有最高線性相似性的通道作為還原之通道。藉由計算該組通道之間的回歸直線，剪枝通道上的權重值可以被還原通道的權重值近似。初步的結果顯示，在準確率、浮點數運算量和參數量下降上都有改善。在CIFAR-10資料集上，VGG-11模型在減少了75%浮點數運算量與85%參數量的情況下，準確率達到92.2%。

Pruning of deep neural networks is commonly used to reduce the inference time cost and memory consumption of deep learning models such that they can be run on devices with only limited resource budgets. Current pruning algorithms estimate the importance of each set of weights, typically channel or layer, in the model according to a certain criterion, and then prune the redundant channel. However, the kernel maps multiplied by these pruned channels in the next layer are also lost without meeting the pruning criterion. Accordingly, this paper proposes a restorative pruning algorithm in which another channel in the same layer is used to restore the removed kernel maps to approximate the original output of the next layer. In the proposed algorithm, the channel which is used to restore is determined by the Pearson Correlation coefficient between two channels with highest linear similarity in the same layer. By calculating the regression line of these two channels, the values of the kernel maps on one channel can be used to restore the values on the other. The preliminary result demonstrates an improvement on accuracy, number of FLOPs and parameter reduction. On VGG-11, with 75% FLOPs reduction by removing 85% of the parameters, the accuracy achieves 92.2% on CIFAR-10 dataset.

Reference
1. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” NIPS, 2012.
2. C. Szegedy et al., "Going deeper with convolutions," CVPR. 2015.
3. K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition," CVPR, 2015.
4. G. E. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” CoRR, 2015.
5. M. Moczulski, M. Denil, J. Appleyard, and N. de Freitas, “Acdc: A structured efficient linear layer,” ICLR, 2016.
6. Y. Cheng, F. X. Yu, R. Feris, S. Kumar, A. Choudhary, and S.-F. Chang, “An exploration of parameter redundancy in deep networks with circulant projections,” ICCV, 2015.
7. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” CoRR, 2014.
8. J.-H. Luo, J. Wu, and W. Lin, “Thinet: A filter level pruning method for deep neural network compression,” ICCV, 2017.
9. P. Molchanov, A. Mallya, S. Tyree, I. Frosio and J. Kautz, "Importance Estimation for Neural Network Pruning," CVPR, 2019.
10. W. Chen, J. Wilson, S. Tyree, K. Q. Weinberger, and Y. Chen, “Compressing neural networks with the hashing trick,” JMLR Workshop and Conference Proceedings, 2015.
11. L. Chen, G. Papandreou, I. Kokkinos, K. Murphy and A. L. Yuille, "DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs," IEEE, 2017.
12. R. Yu, A. Li, C.-F. Chen, J.-H. Lai, V. I. Morariu, X. Han, M. Gao, C.-Y. Lin, and L. S. Davis, “NISP: Pruning networks using neuron importance score propagation,” CVPR, 2017.
13. Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang, “Learning efficient convolutional networks through network slimming,” ICCV, 2017.
14. M. Denil, B. Shakibi, L. Dinh, M. Ranzato, and N. D. Freitas, “Predicting parameters in deep learning,” NIPS, 2013.
15. R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” CVPR, 2014.
16. Ross Girshick, “Fast r-cnn,” IEEE, 2015.
17. S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” NIPS, 2015.
18. W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li, “Learning structured sparsity in deep neural networks,” NIPS, 2016.
19. V. Lebedev and V. Lempitsky, “Fast convnets using group-wise brain damage,” CVPR, 2016.
20. Zhichao Lu, Gautam Sreekumar, Erik Goodman, Wolfgang Banzhaf, Kalyanmoy Deb, Vishnu Naresh Boddeti, “Neural Architecture Transfer,” 2020.
21. Catherine Wong, Neil Houlsby, Yifeng Lu, Andrea Gesmundo: “Transfer Learning with Neural AutoML”, NIPS, 2018.
22. H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” ICLR, 2017.
23. Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, Dmitry Kalenichenko, “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference,” IEEE, 2018.
24. J. Ye, X. Lu, Z. Lin, and J. Z. Wang. “Rethinking the smallernorm-less-informative assumption in channel pruning of convolution layers,” ICLR, 2018.
25. Y. Chauvin, “A back-propagation algorithm with optimal use of hidden units,” NIPS, 1989.
26. Z. Wu, T. Nagarajan, A. Kumar, S. Rennie, L. S. Davis, K. Grauman, and R. Feris, “Blockdrop: Dynamic inference paths in residual networks,” CVPR, 2018.
27. Y. He and S. Han, “Adc: Automated deep compression and acceleration with reinforcement learning,” 2018.
28. Y. LeCun, J. S. Denker, S. Solla, R. E. Howard, and L. D. Jackel, “Optimal brain damage,” NIPS, 1990.
29. P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz, “Pruning convolutional neural networks for resource efficient transfer learning,” ICLR, 2017.
30. Darryl D. Lin, Sachin S. Talathi, V. Sreekanth Annapureddy: “Fixed Point Quantization of Deep Convolutional Networks,” 2015.
31. Wikipedia contributors. "Pearson correlation coefficient." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 17 Jul. 2020. Web. 2 Aug. 2020.
32. Babajide O. Ayinde, Jacek M. Zurada, “Building Efficient ConvNets using Redundant Feature Pruning,” ICLR, 2018.
33. Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,” CVPR, 2016.
34. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer, “Automatic differentiation in pytorch,” NIPS, 2017.
35. C. Zhao, B. Ni, J. Zhang, Q. Zhao, W. Zhang and Q. Tian, "Variational Convolutional Neural Network Pruning," CVPR, 2019.