超解析技術主要的目的是利用低解析度影像產生高解析度影像，並盡可能地不遺失影像資訊。在本論文中，我們提出一個利用卷積神經網路與動作補償的視訊超解析演算法。首先對目標幀的相鄰幀做動態補償使其接近目標幀同時保有目標幀和相鄰幀之間的時間域資訊，其結果作為訓練卷積神經網絡的一部分輸入資料。接著利用二階導數為基礎的空間濾波器截取目標幀的高頻細節資訊並作為訓練類神經網路的一部分輸入資料。在訓練卷積神經網絡後得到一個低解析度影像與高解析度影像的非線性映射關係，在本論文的卷積神經網絡中使用週期性洗牌算子應用在重建高解析度影像的步驟使本論文的演算法得到速度上的提升。
實驗結果顯示，本論文所提出的演算法相較於其他方法，在運行時間、主觀影像品質與客觀影像評估標準上皆有較好的表現。

The main object of the super-resolution method is to produce the high-resolution (HR) images from the low- resolution (LR) images. In the proposed method, a convolutional neural network (CNN)-based algorithm which utilizes the motion compensation method and detail acquirement method as preprocessing is proposed. First, motion compensation makes adjacent frames approximate to the target frame and contain the temporal information. Second, extract the high-pass information from target frame by using the second-order derivative-based spatial filter. The results of above two steps are the inputs of CNN. A non-linear mapping relationship function between LR image and HR image is calculated by training the convolutional neural network. The periodic shuffling operator is employed in the reconstruction layer to improve the processing speed.
The experimental results show that the proposed approach has better performance than other methods on subjective visual quality, objective measurement and processing speed.

[1] R. Y. Tsai and T. S. Huang, “Multiframe image restoration and registration,” in Advances in Computer Vision and Image Processing, greenwich, JAI Press, 1984, pp. 317-339.
[2] H. S.Hou andH. C.Andrews, “Cubic Splines for Image Interpolation and Digital Filtering,” IEEE Trans. Acoust., vol. 26, no. 6, pp. 508–517, 1978.
[3] C. E.Duchon, “Lanczos Filtering in One and Two Dimensions,” Journal of Applied Meteorology, vol. 18, no. 8. pp. 1016–1022, 1979.
[4] X.Li andM. T.Orchard, “New edge-directed interpolation,” IEEE Trans. Image Process., vol. 10, no. 10, pp. 1521–1527, 2001.
[5] X.Zhang andX.Wu, “Image interpolation by adaptive 2-D autoregressive modeling and soft-decision estimation,” IEEE Trans. Image Process., vol. 17, no. 6, pp. 887–896, 2008.
[6] S. D. S.Dai, M. H. M.Han, Y. W. Y.Wu, andY. G. Y.Gong, “Bilateral Back-Projection for Single Image Super Resolution,” Multimed. Expo, 2007 IEEE Int. Conf., pp. 1039–1042, 2007.
[7] W.Dong, L.Zhang, G.Shi, X.Wu, andIeee, “Nonlocal Back-Projection for Adaptive Image Enlargement,” 2009 16th Ieee Int. Conf. Image Process. Vols 1-6, pp. 349–352, 2009.
[8] Y. C.Wee andH. J.Shin, “A novel fast fractal super resolution technique,” IEEE Trans. Consum. Electron., vol. 56, no. 3, pp. 1537–1541, 2010.
[9] S.Mallat andG.Yu, “Super-Resolution With Sparse Mixing Estimators,” vol. 19, no. 11, pp. 2889–2900, 2010.
[10] W. T.Freeman, T. R.Jones, andE. C.Pasztor, “Learning low-level vision,” IEEE Int. J. Comput. Vis., vol. 40, no. 1, pp. 25–47, 2000.
[11] Hong Chang, Dit-Yan Yeung, andYimin Xiong, “Super-resolution through neighbor embedding,” Proc. 2004 IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognition, 2004. CVPR 2004., vol. 1, pp. 275–282, 2004.
[12] S. H.Hong, R. H.Park, S.Yang, andJ. Y.Kim, “Image interpolation using interpolative classified vector quantization,” Image Vis. Comput., vol. 26, no. 2, pp. 228–239, 2008.
[13] A.Kappeler, S.Yoo, Q.Dai, andA. K.Katsaggelos, “Video Super-Resolution With Convolutional Neural Networks,” IEEE Trans. Comput. Imaging, vol. 2, no. 2, pp. 109–122, 2016.
[14] W.Shi et al., “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network,” 2016.
[15] B. D.Lucas and T.Kanade, “An Iterative Image Registration Technique with an Application to Stereo Vision,” Imaging, vol. 130, no. x, pp. 674–679, 1981.
[16] G.Farneb, “Two-Frame Motion Estimation Based on Polynomial Expansion,” Lect. Notes Comput. Sci., vol. 2749, no. 1, pp. 363–370, 2003.
[17] R. C. Gonzalez and R. E. Woods, Digital Image Processing (3rd Edition), NJ, USA: Prentice-Hall, Inc.,2006.
[18] Y.Lecun, Y.Bengio, andG.Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015.
[19] A.Krizhevsky, I.Sutskever, andG. E.Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” Adv. Neural Inf. Process. Syst., pp. 1–9, 2012.
[20] R.Hahnloser, R.Sarpeshkar, M.Mahowald, R.Douglas, andH. S.Seung, “Digital Selection and Analogue Amplification Coexist in a Cortex-inspired Silicon Circuit,” NATUREnature.com Deep. Res. II Of®ce Nav. Res. New Zeal. J. Zoo. Crosnier, A.) Mem. Mus. Natl Hist. Nat. Prog. Ocean., vol. 405, no. 31, pp. 517–567, 2000.
[21] K.Fukushima, “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,” Biol. Cybern., vol. 36, no. 4, pp. 193–202, 1980.
[22] Z.Wang, A. C.Bovik, H. R.Sheikh, andE. P.Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, 2004.
[23] Free 4K Demo Footage Centerh : ttps://www.harmonicinc.com/free-4k-demo-footage/
[24] Jacob + Katie Schwarz’s youtube channel :
https://www.youtube.com/channel/UCYq-iAOSZBvoUxvfzwKIZWA
[25] Around The World 4K’s youtube channel :
https://www.youtube.com/channel/UC4F-DMVCe-UbIsweL6kIEfg
[26] A.Kappeler, S.Yoo, Q.Dai, andA. K.Katsaggelos, “Video Super-Resolution With Convolutional Neural Networks,” IEEE Trans. Comput. Imaging, vol. 2, no. 2, pp. 109–122, 2016.
[27] J.Caballero et al., “Real-time video super-resolution with spatio-temporal networks and motion compensation,” Proc. - 30th IEEE Conf. Comput. Vis. Pattern Recognition, CVPR 2017, vol. 2017–January, pp. 2848–2857, 2017.
[28] OpenCV library: https://opencv.org/
[29] tegg89’s github: https://github.com/tegg89/SRCNN-Tensorflow
[30] drakelevy’s github: https://github.com/drakelevy/ESPCN-TensorFlow