由於受限於硬體技術，影像放大(image enlargement)通常經由軟體來完成，並且在影像處理中成為一項重要的研究課題。傳統上，影像內插(image interpolation)只能從單一影像來完成影像放大，影像品質因此受到很大的限制。超解析度放大(super-resolution enlargement)演算法則可將多張影像視為額外的資訊，來估算出一張高解析度影像。假設有足夠的觀察低解析度影像序列(observed low-resolution image sequence)，且在影像序列中包含有次像素移動(subpixel shifts)，那麼高解析影像就可以估算出來。在影像序列放大中會遇到兩個主要的問題，一個是移動估計(motion estimation)，另一個是超解析度演算法，用以將觀察影像序列及移動估計的結果當作輸入，產生一張高解析度影像。此法利用影像序列中之空間與時間資訊以製造出更高解析度的影像。
在本篇論文中，我們根據超解析度放大演算法來提出一個完善的系統，以處理影像序列放大的問題。這個系統主要包含三大步驟，第一個是改進的階層式區塊比對(hierarchical block matching)，此方法包含部分扭曲排除(partial distortion elimination)，第二個是改進的多層次放大，此方法是根據疊代式解析度加強(iterative resolution enhancement)演算法所改進的，第三個是強健式點對應(robust point matching)方法，此方法是用影像序列的中央影像與靜態影像來做影像對位。在本研究中，我們提出的多層次放大方法所產生的結果，比傳統方法更能夠有效加強影像序列之解析度，視覺上的以及量測數字上的評估均顯示顯著的改進，我們提出的系統在計算上也更為快速。藉由強健式點對應方法，我們也成功地將靜態影像合併到影像序列，並且得到更好的放大影像序列。

Due to hardware limitation, image enlargement (magnification) is mostly carried out by software and becomes an important research task in image processing. Traditionally, image interpolation can only be done with a single image frame. Thus, its quality is greatly constrained. Super-resolution enlargement algorithm is proposed that uses multiple frames as additional information to estimate the high-resolution image. Given enough observed low-resolution image frames with sub-pixel shifts, the construction of high-resolution image can be computed. Two major problems in super-resolution enlargement are the motion estimation and the super-resolution algorithm, which takes the observed frames and the estimated motion as the inputs and produces a high-resolution image. In other words, it utilizes both the spatial and temporal information in an image sequence in creating a higher resolution image.

In this thesis, a complete system based on super-resolution enlargement algorithm is proposed for image sequence enlargement. This system consists of three major steps, the modified hierarchical block matching method together with partial distortion elimination, the modified multilevel magnification based on an iterative resolution enhancement algorithm, and the robust point matching method used to register the center image frame and the still image. In this study, the proposed multilevel magnification method achieves better magnification quality than the traditional one. Besides, both the visual and quantitative improvements are significant, the proposed system is also faster in computation. The image registration by the robust point matching algorithm is also successful in combing a still high-resolution image to the given image sequence and obtains even better magnification results.

[1] R. Y. Tsai and T. S. Huang, “Multiframe Image Restoration and Registration,” Advances in Computer Vision and Image Processing (R. Y. Tsai, T. S. Huang, Eds.), vol. 1, pp. 317-339, JAI Press, London, 1984.

[2] R. R. Schultz and R. L. Stevenson, “A Bayesian Approach to Image Expansion for Improved definition,” IEEE Transactions on Image Processing, vol. 3, pp. 233-242, May 1994.

[3] R. R. Schultz and R. L. Stevenson, “Improved definition video frames from video sequence,” Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing, vol. 4, pp. 2169-2171, May 1995.

[4] R. R. Schultz and R. L. Stevenson, “Extraction of High-resolution Frame from Video Sequence,” IEEE Transactions on Image Processing, vol. 5, pp. 996-1011, June 1996.

[5] R. R. Schultz, L. Meng, and R. L. Stevenson, “Subpixel Motion estimation for Super-resolution Image Sequence Enhancement,” Journal of Visual Communication and Image Representation, special issue on High-Fidelity Media Processing, vol. 9, no. 1, pp. 38-50, Mar. 1998.

[6] P. H. Chang, J. J. Leou, and H. C. Hsieh, “A Genetic Algorithm Approach to Image Sequence Interpolation,” Signal Processing: Image Communication, vol. 16, pp. 507-520, 2001.

[7] S. Baker and T. Kanade, “Hallucinating faces,” Automatic Face and Gesture Recognition, 2000. Proceedings. Fourth IEEE International Conference on, pp. 83-88, 2000.

[8] S. Baker and T. Kanade, “Limits on super-resolution and how to break them,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, Issue: 9, pp. 1167-1183, Sept. 2002.

[9] J. Patti, M. I. Sezan, and A. M. Tekalp, “Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time,” IEEE Transactions on Image Processing, vol. 6, no. 8, pp. 1064-1076, Aug. 1997.

[10] P. E. Eren, M. I. Sezan, and A.M. Tekalp, “Robust, object-based high-resolution image reconstruction from low-resolution video,” IEEE Transactions on Image Processing, vol. 6, no. 10, pp. 1446-1451, Oct. 1997.

[11] Nimish R. Shah and Avideh Zakhor, “Resolution enhancement of color video sequences,” IEEE Transactions on Image Processing, vol. 8, Issue: 6, pp. 879-885, June 1999.

[12] Nimish R. Shah and Avideh Zakhor, “Multiframe spatial resolution enhancement of color video,” Image Processing, 1996. Proceedings. International Conference on, vol. 1, pp. 985-988, Sept. 1996.

[13] B. C. Tom and A. K. Katsaggelos, “Resolution enhancement of monochrome and color video using motion compensation,” IEEE Transactions on Image Processing, vol. 10, Issue: 2, pp. 278-287, Feb. 2001.

[14] A. K. Katsaggelos, “A multiple input image restoration approach,” Journal of Visual Communication and Image Representation, vol. 1, no. 1, pp. 93–103, Sept. 1990.

[15] M. Irani and S. Peleg, “Improving resolution by image registration,” CVGIP: Graphical Models and Image Processing, vol. 53, no. 3, pp. 231–239, May 1991.

[16] M. Irani and S. Peleg, “Motion analysis for image enhancement: Resolution, occlusion and transparency,” Journal of Visual Communications and Image Representation, vol. 4, pp. 324-335, Dec. 1993.

[17] A. Zomet and S. Peleg, “Efficient Super-Resolution and Applications to Mosaics,” International Conference on Pattern Recognition (ICPR'00), vol. 1, pp. 3-8, Sept. 2000.

[18] A. Zomet, A. Rav-Acha, and S. Peleg, “Robust super-resolution,” in Proceedings of the Int. Conf. on Computer Vision and Patern Recognition (CVPR), vol. 1, pp. 645-650, Dec. 2001.

[19] A. Zomet and S. Peleg, “Multi-sensor super-resolution,” Applications of Computer Vision, 2002. (WACV 2002). Proceedings. Sixth IEEE Workshop on, pp. 27-31, 2002.

[20] H. Greenspan, G. Oz, N. Kiryati, and S. Peled, “MRI inter-slice reconstruction using super-resolution,” Magnetic Resonance Imaging, vol. 20, pp.437-446, 2002.

[21] H. Greenspan, G. Oz, N. Kiryati, and S. Peled, “Super-resolution in MRI,” Biomedical Imaging, 2002. Proceedings. 2002 IEEE International Symposium on, pp. 943-946, 2002.

[22] J. B. A. Maintz and M. A. Viergever, “A survey of medical image processing,” Medical Image Analysis, vol. 2, no. 1, pp. 1-36, 1998.

[23] P. Besl and N. McKay, “A method for registration of 3-D shapes,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, no. 2, pp. 239-256, 1992.

[24] A. Rangarajan, H. Chui, E. Mjolsness, S. Pappu, L. Davachi, P. Goldman-Rakic, and J. Duncan., “A robust point matching algorithm for autoradiograph alignment,” Medical Image Analysis, vol. 4, no. 1, pp. 379-398, 1997.

[25] A. Rangarajan and H. Chui., “Rigid Point Feature Registration Using Mutual Information,” Medical Image Analysis, vol.4, pp. 1-17, 1999.

[26] M. Bierling and R. Thoma, “Motion Compensating Field Interpolation Using a Hierarchically Structured Displacement Estimator,” Signal Processing, vol. 11, no. 4, pp. 387-404, Dec. 1986.

[27] B. K. P. Horn and B.G. Shunck, “Determining Optical Flow,” Artificial Intelligence 17, pp. 185-203, 1981.

[28] B. C. Tom and A. K. Katsaggelos, “An iterative algorithm for improving the resolution of video sequence,” in Proc. SPIE Visual Communication Image Processing, pp. 1430-1438, Mar. 1996.

[29] M. Elad and A. Feuer, “Superresolution Restoration of Image Sequence: Adaptive Filtering Approach,” IEEE Transactions on Image Processing, vol. 8, no.3, pp. 387-395, Mar. 1999.

[30] V. Avrin and I. Dinstein, “Local motion estimation resolution enhancement video sequences,” Pattern Recognition, 1998. Proceedings. Fourteenth International Conference on, vol. 1, no. 16-20, pp. 539 -541, Aug. 1998.

[31] C. H. Lee and L. H. Chen, “A Fast Motion Estimation Algorithm Based on the Block Sum Pyramid,” IEEE Transactions on Image Processing, vol. 6, no. 11, pp. 1587-1591, Nov. 1997.

[32] H. S. Wang and R. M. Mersereau, “Fast algorithms for the estimation of motion vectors,” IEEE Transactions on Image Processing, vol. 8, Issue: 3, pp. 435-438, Mar. 1999.

[33] E. Shechtman, Y. Caspi, and M. Irani, “Increasing Space-Time Resolution in Video,” ECCV 2002, pp.753-768, 2002.

[34] J. Canny, “A computational approach to edge detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 8, no. 6, pp. 679-698, Nov. 1986.

[35] P. J. M. van Laarhoven and E. H. L. Aarts, Simulated Anneling: Theory and Applications. D. Reidel Publishing Company, 1987.

[36] R. Sinkhorn, “A relationship between arbitrary positive matrices and doubly stochastic matrices,” Ann. Math. Statist., vol. 35, pp. 876-879, 1964.

[37] S. Gold, C. P. Lu, A. Rangarajan, S. Pappu, and E. Mjolsness, “New algorithms for 2-D and 3-D point matching: pose estimation and correspondence,” In Tesauro, G., Touretzky, D. and Alspector, J. (editors), Advances in Neural Information Processing Systems, vol. 7, pp. 957-964. MIT Press, Cambridge, MA. 1995.

[38] Y. Caspi and M. Irani, “Spatio-temporal alignment of sequences,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 11, pp. 1409-1424, Nov. 2002.