在本篇論文中，我們提出兩種結合卷積神經網路 (Convolutional neural networks, CNN) 以及迭代反向投影 (Iterative back projection, IBP) 的方法來強化圖像超解析度。第一個方法針對多重放大倍率的圖像超解析度，一般基於卷積神經網路的超解析度技術中，對於每一個圖像放大倍率都要個別訓練一個卷積神經網路的模組，例如要實現兩倍放大的圖像超解析度，在訓練模組時就必須針對兩倍的放大圖像做訓練。在此篇論文中，我們只利用一個卷積神經網路模組實現不同放大倍率的圖像超解析度，換句話說，在訓練時以一個模組同時訓練不同的放大倍率，再進一步利用迭代反投影技術強化輸出的高解析度圖像；第二個方法利用卷積神經網路技術，改良迭代反向投影演算法中誤差補償的部分。我們進一步將此基於卷積神經網路改良的迭代反向投影方法 (CNN-based iterative back projection, CIBP) 與現存的超解析度卷積神經網路 (Super-resolution convolutional neural networks, SRCNN) 做結合，得到更好的超解析度重建圖像。根據實驗結果，我們所提出的兩種方法都能夠得到優於其他方法的超解析度效果。

In this thesis, we investigate two approaches to enhance image super-resolution by combining the deep Convolutional Neural Networks (CNN) with Iterative Back Projection (IBP). For the first approach, a CNN model is learned from mapping from the low resolution images into the high resolution ones. Unlike conventional methods, we train several scaling factors in the same network. Then, the IBP is utilized to further reduce the artefacts in the generated high resolution image. This approach enables us process different scaling factors using only a single CNN model. For the second approach, we use an additional error-compensated CNN model (eCNN) to more accurately compensate the residual in the IBP, which is further combined with the super-resolution CNN (SRCNN). The experiment results show that our methods significantly enhance the restoration quality, and outperform the state-of-the-art super-resolution methods.

[1] S. Farsiu, M. D. Robinson, M. Elad, and P. Milanfar, “Fast and robust multiframe super resolution,” IEEE Transactions on Image processing, vol. 13, no. 10, pp. 1327–1344, 2004.
[2] D. Glasner, S. Bagon, and M. Irani, “Super-resolution from a single image,” IEEE 12th International Conference on Computer Vision, pp. 349–356, 2009.
[3] C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” Computer Vision-ECCV, pp. 184-199, 2014
[4] M. Irani and S. Peleg, “Improving resolution by image registration,” CVGIP: Graphical models and image processing, vol. 53, no. 3, pp. 231–239, 1991.
[5] M. Irani and S. Peleg, “Motion analysis for image enhancement: Resolution, occlusion, and transparency,” Journal of Visual Communication and Image Representation, vol. 4, no. 4, pp. 324–335, 1993.
[6] J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE Transactions on Image Processing, vol. 19, no. 11, pp. 2861–2873, 2010.
[7] C. E. Duchon, “Lanczos filtering in one and two dimensions,” Journal of Applied Meteorology, vol. 18, no. 8, pp. 1016–1022, 1979.
[8] X. Li and M. T. Orchard, “New edge-directed interpolation,” IEEE Transactions on Image Processing, vol. 10, no. 10, pp. 1521–1527, 2001.
[9] G. Freedman and R. Fattal, “Image and video upscaling from local self-examples,” ACM Transactions on Graphics (TOG), vol. 30, no. 2, pp. 12, 2011.
[10] J. Yang, Z. Lin, and S. Cohen, “Fast image super-resolution based on in-place example regression,” IEEE Conference on Computer Vision and Pattern Recognition, pp. 1059–1066, 2013.
[11] H. Chang, D.-Y. Yeung, and Y. Xiong, “Super-resolution through neighbor embedding,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. I–I, 2004.
[12] M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi-Morel, “Low-complexity single-image super-resolution based on nonnegative neighbor embedding,” British Machine Vision Conference, BMVC, pp. 135.1–135.10, 2012.
[13] R. Timofte, V. De, and L. Van Gool, “Anchored neighborhood regression for fast example-based super-resolution,” IEEE International Conference, ICCV, pp. 1920–1927, 2013.
[14] K. Fukushima, "Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position," Biological Cybernetics, vol. 36, no. 4, pp. 193-202, 1980.
[15] Rosenblatt, Frank, The perceptron, a perceiving and recognizing automaton, Cornell Aeronautical Laboratory, 1957.
[16] M. Minsky and S. Papert, “Perceptrons,” Cambridge: MIT Press, 1969.
[17] Williams, DE Rumelhart GE Hinton RJ, and G. E. Hinton. "Learning representations by back-propagating errors," Nature 323, 533-536, 1986.
[18] Cortes, Corinna, and Vladimir Vapnik. "Support-vector networks," Machine learning 20.3, 273-297, 1995.
[19] Freund, Yoav, and Robert E. Schapire. "Experiments with a new boosting algorithm," ICML, vol. 96, pp. 148-156, 1996.
[20] Hinton, Geoffrey E., and Ruslan R. Salakhutdinov. "Reducing the dimensionality of data with neural networks," Science 313.5786 , pp. 504-507, 2006.
[21] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
[22] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems, pp. 1097–1105, 2012.
[23] D. Eigen, D. Krishnan, and R. Fergus, “Restoring an image taken through a window covered with dirt or rain,” IEEE International Conference, ICCV, pp. 633–640, 2013.
[24] P. Smolensky, "Information processing in dynamical systems: Foundations of harmony theory," Parallel distributed processing: explorations in the microstructure of cognition, vol. 1, pp. 194-281, 1986.
[25] G. E. Hinton, S. Osindero and Y. W. Teh, "A fast learning algorithm for deep belief nets," Neural Computation, vol. 18, pp. 1527-1554, 2006.
[26] J. Gao, Y. Guo, and M. Yin, “Restricted Boltzmann machine approach to couple dictionary training for image super-resolution,” IEEE International Conference, ICIP, pp. 499–503, 2013.
[27] Y. Zhou, Y. Qu, Y. Xie, and W. Zhang, “Image Super-Resolution Using Deep Belief Networks,” International Conference on Internet Multimedia Computing and Service, pp. 28, 2014.
[28] Hubel, David H., and Torsten N. Wiesel. "Receptive fields, binocular interaction and functional architecture in the cat's visual cortex," The Journal of physiology 160.1, pp 106-154, 1962.
[29] Barnard, Etienne, and David Casasent. "Shift invariance and the neocognitron,"Neural Networks 3.4, pp 403-410, 1990.
[30] M. N. Bareja and C. K. Modi, “An effective iterative back projection based single image super resolution approach,” International Conference, CSNT, pp. 95–99, 2012.
[31] R. R. Makwana and N. D. Mehta, “Single Image Super-Resolution VIA Iterative Back Projection Based Canny Edge Detection and a Gabor Filter Prior,” International Journal of Soft Computing and Engineering (IJSCE), vol. 3, no. 1, pp. 2231–2307, 2013.
[32] Q. Zhou, S. Chen, J. Liu, and X. Tang, “Edge-preserving single image super-resolution,” ACM international conference on Multimedia, pp. 1037–1040, 2011.
[33] V. Jain and S. Seung, “Natural image denoising with convolutional networks,” Neural Information Processing Systems, pp. 769–776, 2009.
[34] L. Bottou, “Large-scale machine learning with stochastic gradient descent,” COMPSTAT, pp. 177–186, 2010.
[35] J. Bouvrie, “Notes on convolutional neural networks,” Technical report, 2006.
[36] Wilson, D. Randall, and Tony R. Martinez. "The general inefficiency of batch training for gradient descent learning," Neural Networks 16.10, pp. 1429-1451, 2003.
[37] R. Zeyde, M. Elad, and M. Protter, “On single image scale-up using sparse-representations,” Curves and Surfaces, pp. 711–730, 2012.
[38] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 5, pp. 898–916, 2011.
[39] Jia, Yangqing, et al. "Caffe: Convolutional architecture for fast feature embedding." ACM International Conference on Multimedia , pp. 675-678, 2014.
[40] Y. Liang, J. Wang, S. Zhou, Y. Gong, and N. Zheng, “Incorporating image priors with deep convolutional neural networks for image super-resolution,” Neurocomputing, pp.340-347, 2016.
[41] Wang, Zhou, et al. "Image quality assessment: from error visibility to structural similarity," IEEE transactions on image processing 13.4, pp. 600-612, 2004.
[42] Sheikh, Hamid R., Alan C. Bovik, and Gustavo De Veciana. "An information fidelity criterion for image quality assessment using natural scene statistics," IEEE Transactions on image processing 14.12, pp. 2117-2128, 2005.
[43] Zhang, Lin, et al. "FSIM: a feature similarity index for image quality Assessment," IEEE transactions on Image Processing 20.8 , pp. 2378-2386, 2011.
[44] Zhang, Lin, et al. "A comprehensive evaluation of full reference image quality assessment algorithms."IEEE International Conference on Image Processing, pp.1477-1480, 2012.