| 研究生: |
方志偉 Fang, Chih-Wei |
|---|---|
| 論文名稱: |
通過多重解析度依據影像內容結構有方向性的填補移除物體之區域 Image Completion System Using Multi-Resolution Patch-Based Directional and Non-Directional Texture Synthesis |
| 指導教授: |
連震杰
Lien, Jenn-Jier James |
| 學位類別: |
博士 Doctor |
| 系所名稱: |
電機資訊學院 - 資訊工程學系 Department of Computer Science and Information Engineering |
| 論文出版年: | 2010 |
| 畢業學年度: | 98 |
| 語文別: | 英文 |
| 論文頁數: | 91 |
| 外文關鍵詞: | Eigenspace, Hessian Matrix, Image Completion, Texture Analysis, Texture Synthesis |
| 相關次數: | 點閱:69 下載:4 |
| 分享至: |
| 查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報 |
本論文提出了一種快速的圖像填補系統,包括一個訓練(或分析)過程和圖像填補(或合成)的過程。我們開發的系統包括兩個模組:紋理分析模組(Texture Analyses)和紋理合成模組(Texture Synthesis)。紋理分析模組能夠分析輸入的影像和在訓練過程中使用這個影像資料。根據訓練非週期或週期的紋理,我們使用不同的取樣方法,有著不同數量的區塊,以減少合成影像災的接縫。此外,主成份分析(Principal Component Analysis,PCA)的特性是用來降低資料的維度,來表達和重組的外觀特徵(即特徵向量,Eigenvectors)。然後,向量量化(vector quantization,VQ)演算法來減少比對區塊上的所用時間。對於合成模組,訓練資料用於合成一個較大的紋理影像,或者是用來取代被移除區域的圖像。
本系統採用多重解析度(Multi-Resolution)的方法,這不僅提高了合成過程的收斂速度,而且還提供了能夠處理較大的移除地區,而無需指派隨機的初始值或近似值。在訓練過程中,降維取樣(Down-Sampling)的方法是創建一個適用於修補的紋理特徵空間基於在多重解析度下的背景區域資料。在圖像填補過程中,升維取樣的方法(Up-Sampling)給於在合成替代前景地區的初始值。為了確保合成替代前景區域和原有的背景區域之間連續性的幾何紋理結構,我們發展了定向和非定向的方法(Directional and Non-Directional image Completion)是用來在較低解析度(Lower-Resolution)重建整體的幾何結構和在較高解析度(Higher-Resolution)提高細微的特徵。在圖像填補的過程中,合成個別區塊的優先順序和選擇相對的填補計劃(即定向或非定向)都是依照Hessian矩陣決定值(Hessian Matrix Decision Value,HMDV)參數。為了消除邊緣效應的合成影像,該修補程序的比對過程是基於只有在每個補丁邊界內的像素,定義為一個O形圖案(O-shape pattern)而不是所有的像素在整個修補程序。最後,紋理精細化的過程,提供一個較高解析度的合成結果。因此,我們的系統能迅速獲得高影像品質和充滿希望的結果。
This thesis presents a rapid image completion system comprising a training (or analysis) process and an image completion (or synthesis) process. We developed a system including two modules: the texture analysis module and the texture synthesis module. The analysis module is capable of analyzing an input image and performing the training process by using this image data. According to the training non-periodic or periodic pattern, we use different sampling methods to have different amount of patches in order to reduce the emergences of the seams of the output synthesized image. In addition, the properties of principal component analysis (PCA) are used to reduce the dimensions of the data representation and to recombine the appearance of the features (i.e. eigenvectors). Then the vector quantization (VQ) algorithm is employed to reduce the time spent on matching comparison. For the synthesis module, the training data is used to synthesize a large output texture, or is employed to replace the removed regions of an image.
The proposed system adopts a multi-resolution approach, which not only improves the convergence rate of the synthesis process, but also provides the ability to deal with large replaced regions without needing to assign initial random values or approximate values. In the training process, a down-sampling approach is applied to create a patch-based texture eigenspace based on multi-resolution background region information. In the image completion process, an up-sampling approach is applied to synthesize the replaced foreground regions. To ensure the continuity of the geometric texture structure between the original background scene regions and the replaced foreground regions, directional and non-directional image completion approaches are developed to reconstruct the global geometric structure and to enhance the local detailed features of the replaced foreground regions in the lower- and higher-resolution level images, respectively. In the image completion process, the synthesis priority order of the individual patches and the appropriate choice of completion scheme (i.e. directional or non-directional) are both determined in accordance with a Hessian matrix decision value (HMDV) parameter. To eliminate the rim effect in the synthesized image, the patch-based matching process is based only upon the pixels within the border of each patch defined as an O-shape pattern rather than all the pixels in the entire patch. Finally, a texture refinement process is performed to optimize the resolution of the synthesized result. Therefore, our system can rapidly obtain a high image quality and promising result.
[1] M. Ashikhmin, “Synthesizing Natural Textures,” ACM Symposium Interactive 3D Graphics, pp. 217-226, 2001.
[2] C. Ballester, V. Caselles, and J. Verdera, “Disocclusion by Joint Interpolation of Vector Fields and Gray Levels,” SIAM Journal Multiscale Modelling and Simulation, Vol. 2, pp. 80-123, 2003.
[3] C. Ballester, V. Caselles, J. Verdera, M. Bertalmio, and G. Sapiro, “A Variational Model for Filling-in Gray Level and Color Images,” IEEE International Conference on Computer Vision, pp. 10-16, 2001.
[4] C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, and J. Verdera, “Filling-In by Joint Interpolation of Vector Fields and Gray Levels,” IEEE Transactions on Image Processing, Vol.10, No.8, pp.1200-1211, 2001.
[5] Z. Bar-Joseph, R. El-Yaniv, D. Lischinski, and M. Werman, “Texture Mixing and Texture Movie Synthesis using Statistical Learning,” IEEE Transactions on Visualization and Computer Graphics, Vol. 7, No. 2, pp. 120-135, 2001.
[6] M. Bertalmio, “Strong-Continuation, Contrast-Invariant Inpainting with a Third-Order Optimal PDE,” IEEE Transactions on Image Processing, Vol.15, No. 7, pp. 1934-1938, 2006.
[7] M. Bertalmio, A. L. Bertozi, and G. Sapiro, “Navier-Stokes, Fluid Dynamics, and Image and Video Inpainting,” IEEE Conference on Computer Vision and Pattern Recognition, pp.355-362, 2001.
[8] M. Bertalmio, G. Sapiro, C. Ballester, and V. Caselles, “Image Inpainting,” ACM SIGGRAPH, pp. 417-424, 2000.
[9] M. Bertalmio, L. Vese, G. Sapiro, and S. Osher, “Simultaneous Structure and Texture Image Inpainting,” IEEE Transactions on Image Processing, Vol. 12, No. 8, pp. 882-889, 2003.
[10] S. Borikar, K. K. Biswas, and S. Pattanaik, “Fast Algorithm for Completion of Images with Natural Scenes,” CS Technical Report, University of Central Florida, April 2004.
[11] R. Bornard, E. Lecan, L. Laborelli, and J. H. Chenot, “Missing Data Correction in Still Images and Image Sequences,” ACM Multimedia, pp. 355-361, 2002.
[12] R.J. Cant, and C.S. Langensiepen, “A Multiscale Method for Automated Inpainting,” 17th European Simulation Multiconference, 2003.
[13] T. F. Chan, S. H. Kang, and J. Shen, “Euler’s Elastica and Curvature-Based Image Inpainting,” SIAM Journal on Applied Mathematics, Vol. 63, No. 2, pp. 564-592, 2002.
[14] T. F. Chan, and J. Shen, “Non-Texture Inpainting by Curvature-Driven Diffusions (CDD),” Journal of Visual Communication and Image Representation, Vol. 12, No. 4, pp. 436-449, 2001.
[15] T. F. Chan, and J. Shen, “Mathematical Models for Local Nontexture Inpaintings,” SIAM Journal on Applied Mathematics, Vol. 62, No. 3, pp. 1019-1043, 2001.
[16] T. F. Chan, and J. Shen, “Morphologically Invariant PDE Inpaintings,” UCLA CAM Report, 2001.
[17] W. H. Cheng, S. K. Lin, C. W. Wang, C. W. Hsieh, and J. L. Wu, "Effective Exemplar-Based Image Inpainting," 18th IPPR Conference on Computer Vision, Graphics and Image Processing (CVGIP 2005), pp.1337-1344, 2005.
[18] A. Criminisi, P. Perez, and K. Toyama, “Object Removal by Exemplar-Based Inpainting,” IEEE Conference on Computer Vision and Pattern Recognition, pp. 721--728, 2003.
[19] A. Criminisi, P. Perez, and K. Toyama, “Region Filling and Object Removal by Exemplar-Based Image Inpainting,” IEEE Transactions on Image Processing, Vol. 13, No. 9, pp. 1200-1212, 2004.
[20] G. Doretto, and S. Soatto, “Editable Dynamic Textures,” IEEE Conference on Computer Vision and Pattern Recognition, pp. 137-142, 2003.
[21] I. Drori, D. Cohen-Or, and H. Yeshurun, “Fragment-Based Image Completion,” ACM Transactions on Graphics (ACM SIGGRAPH 2003), Vol. 22, No. 3, pp. 303-312, 2003.
[22] J. S. De Bonet, “Multiresolution Sampling Procedure for Analysis and Synthesis of Texture Images,” ACM SIGGRAPH, pp. 361-368, 1997.
[23] L. Demanet, B. Song, and T. Chan, “Image Inpainting by Correspondence Maps: a Deterministic Approach,” UCLA CAM Report 03-40, 2003.
[24] A. A. Efros and W. T. Freeman, “Image Quilting for Texture Synthesis and Transfer,” ACM SIGGRAPH, pp. 341-346, 2001.
[25] A. A. Efros and T. K. Leung, “Texture Synthesis by Non-parametric Sampling,” International Conference on Computer Vision, pp. 1033-1038, 1999.
[26] S. Esedoglu and J. Shen, “Digital Inpainting Based on the Mumford-Shah-Euler Image Model,” European Journal of Applied Mathematics, Vol. 13, pp. 353-370, 2002.
[27] H. Fang, and J. C. Hart, "Textureshop: Texture Synthesis as a Photograph Editing Tool," ACM Transactions on Graphics (ACM SIGGRAPH 2004), Vol. 23, No. 3, pp. 354-359, 2004.
[28] C. W. Fang, and J. J. Lien, “Fast Image Replacement Using Multi-resolution Approach,” Asian Conference on Computer Vision, pp. 509-520, 2006.
[29] A. W. Fitzgibbon, “Stochastic Rigidity: Image Registration for Nowhere-Static Scenes,” International Conference on Computer Vision, pp. 662-669, 2001.
[30] W. T. Freeman, T. R. Jones, and E. C. Pasztor, “Example-Based Super-Resolution,” IEEE Computer Graphics and Applications, Vol. 22, No. 2, pp. 56-65, 2002.
[31] W. T. Freeman, E. Pasztor, and O. Carmichael, “Learning Low-Level Vision,” International Journal of Computer Vision, Vol. 40, No. 1, pp. 25-47, 2000.
[32] T. Gevers, “Image Segmentation and Similarity of Color-Texture Objects,” IEEE Transactions on Multimedia, Vol. 4, No. 4, pp. 509-516, 2000.
[33] S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, “The Lumigraph,” ACM SIGGRAPH, pp. 43-54, 1996.
[34] J.B. Greer, A.L. Bertozzi, “H1 Solutions of a Class of Fourth Order Nonlinear Equations for Image Processing,” Discrete and Continuous Dynamical Systems, Vol. 10, No. 1&2, pp. 349-366, 2004.
[35] H. Grossauer, “A Combined PDE and Texture Synthesis Approach to Inpainting,” 8th European Conference on Computer Vision (ECCV), Part II, pp. 214-224, 2004.
[36] C. E. Guo, S. C. Zhu, and Y. N. Wu, “Towards a Mathematical Theory of Primal Sketch and Sketchability,” International Conference on Computer Vision, pp. 1228-1235, 2003.
[37] P. Harrison, "A Non-hierarchical Procedure for Re-Synthesize of Complex Textures," International Conference in Central Europe Computer Graphics, Visualization, and Computer Vision (WSCG 2001), pp. 190-197, 2001.
[38] D. J. Heeger and J. R. Bergen, “Pyramid-Based Texture Analysis/Synthesis,” ACM SIGGRAPH, pp. 229-233, 1995.
[39] H. J. Hsu, J. F. Wang, and S. C. Liao, “A Hybrid Algorithm with Artifact Detection Mechanism for Region Filling After Object Removal From a Digital Photograph,” IEEE Transactions on Image Processing, Vol. 16, No. 6, pp. 1611-1622, 2007.
[40] H. Igehy and L. Pereira, “Image Replacement through Texture Synthesis,” IEEE International Conference on Image Processing, Vol. 3, pp. 186-189, 1997.
[41] J. Jia and C. K. Tang, “Image Repairing: Robust Image Synthesis by Adaptive ND Tensor Voting,” IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1, pp. 643-650, 2003.
[42] J. Jia and C. K. Tang, “Inference of Segmented Color and Texture Description by Tensor Voting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26, No. 6, pp. 771-786, 2004.
[43] V. Kwatra, A. Schödl, I. Essa, G. Turk, and A. Bobick, “Graphcut Textures: Image and Video Synthesis Using Graph Cuts,” ACM Transactions on Graphics (ACM SIGGRAPH 2003), Vol. 22, No. 3, pp. 277-286, 2003.
[44] N. Komodakis, and G. Tziritas, “Image Completion Using Global Optimization,” IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1, pp.442-452, 2006.
[45] N. Komodakis, and G. Tziritas, “Image Completion Using Efficient Belief Propagation via Priority Scheduling and Dynamic Pruning,” IEEE Transactions on Image Processing, Vol. 16, No. 11, pp. 2649-2661, 2007.
[46] N. Kozhekin, V. Savchenko, M. Senin, and I. Hagiwara, “An Approach to Surface Retouching and Mesh Smoothing,” The Visual Computer, Vol. 19, pp. 549-564, 2003.
[47] S. Lefebvre, and H. Hoppe, "Parallel Controllable Texture Synthesis," ACM Transactions on Graphics (ACM SIGGRAPH 2005), Vol. 24, No. 3, pp. 777-786, 2005.
[48] A. Levin, A. Zomet and Y. Weiss, “Learning How to Inpaint from Global Image Statistics”, IEEE International Conference on Computer Vision, Vol.1, pp.305-312, 2003.
[49] L. Liang, C. Liu, Y. Xu, B. Guo, and H. Y. Shum, “Real-Time Texture Synthesis by Patch-Based Sampling,” ACM Transactions on Graphics, Vol. 20, No. 3, pp. 127-150, 2001.
[50] W. C. Lin, J. Hays, C. Wu, V. Kwatra, and Y. Liu, " A Comparison Study of Four Texture Synthesis Algorithms on Near-regular Textures," ACM Transactions on Graphics (ACM SIGGRAPH 2004), Vol. 23, No. 3, pp.16-16, 2004.
[51] W. C. Lin, J. Hays, C. Wu, V. Kwatra, and Y. Liu, "A Comparison Study of Four Texture Synthesis Algorithms on Regular and Near-Regular Textures," Technology Report CMU-RI-TR-04-01, Robotics Institute, Carnegie Mellon University, 2004.
[52] Q. Liu, Z. Yao, Y.Y. Ke, “Solutions of Fourth-Order Partial Differential Equations in a Noise Removal Model,” Electronic Journal of Differential Equations, Vol. 2007, No. 120, pp. 1-11, 2007.
[53] Y. Liu and W. C. Lin, "Deformable Texture: the Irregular-Regular-Irregular Cycle," The Third International Workshop on Texture Analysis and Synthesis, 2003.
[54] Y. Liu and W. C. Lin, "Deformable Texture: the Irregular-Regular-Irregular Cycle," Technology Report CMU-RI-TR-03-26, Robotics Institute, Carnegie Mellon University, 2003.
[55] Y. Liu, W. C. Lin, and J. Hays, “Near-Regular Texture Analysis and Manipulation,” ACM Transactions on Graphics (ACM SIGGRAPH 2004), Vol. 23, No. 3, pp. 368-376, 2004.
[56] Y. Liu, Y. Tsin, and W. C. Lin, "The Promise and Perils of Near-Regular Texture," International Journal of Computer Vision, Vol. 62, No. 1-2, pp. 145-159, 2005.
[57] Z. Liu, C. Liu, H. Y. Shum, and Y. Yu, “Pattern-based Texture Metamorphosis,” 10th Pacific Conference on Computer Graphics and Applications, pp. 184-193, 2002.
[58] B. Mustafa, V. Trajkovik, and D. Davcev, "Missing Data Correction in Still Images Using Multi-Resolution Analysis," Journal of Computing and Information Technology, Vol. 15, No. 1, pp. 1-5, 2007.
[59] A. Nealen, and M. Alexa, “Fast and High Quality Overlap Repair for Patch-Based Texture Synthesis,” Computer Graphics International, pp. 582-585, 2004.
[60] M. M. Oliveira, B. Bowen, R. McKenna and Y. S. Chang, “Fast digital image inpainting,” International Conference on Visualization, Imaging and Image Processing, pp. 261-266, 2001.
[61] S. C. Pei, Y. C. Zeng, and C.H. Chang, “Virtual Restoration of Ancient Chinese Paintings Using Color Contrast Enhancement and Lacuna Texture Synthesis,” IEEE Transactions on Image Processing, Vol. 13, No. 3, pp. 416-429, 2004.
[62] J. Pearl, “Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference,” Morgan Kaufmann Publishers, San Mateo, California, 1988.
[63] P. Perez, M. Gangnet, and A. Blake, “PatchWorks: Example-Based Region Tiling for Image Editing,” Microsoft Research Report MSR-TR-2004-04, pp.190-197, 2004.
[64] J. Portilla, and E. P. Simoncelli, “Texture Modeling and Synthesis using Joint Statistics of Complex Wavelet Coefficients,” IEEE Workshop on Statistical and Computational Theories of Vision, 1999.
[65] J. Portilla, and E. P. Simoncelli, “A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients,” International Journal of Computer Vision, Vol. 40, No. 1, pp. 49-71, 2000.
[66] A. Rares, M. J. T. Reinders, and J. Biemond, “Image Sequence Restoration in The Presence of Pathological Motion and Severe Artifacts,” IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2002), Vol. 4, pp. 3365-3368, 2002.
[67] V. Savchenko, N. Kojekine, and H. Unno, “A Practical Image Retouching Method,” 1st International Symposium on Cyber Worlds (CW 2002), IEEE Computer Society, pp. 480-487, 2002.
[68] J. Shen, “Inpainting and the fundamental problem of image processing,” SIAM News, Vol. 36, No. 5, pp. 110-121, 2003.
[69] E. P. Simoncelli, and P. Javier, “Texture Characterization via Joint Statistics of Wavelet Coefficient Magnitudes,” Fifth International Conference on Image Processing, Vol. 1, pp. 62-66, 1998.
[70] J. Sun, L. Yuan, J. Jia, and H. Y. Shum, “Image Completion with Structure Propagation,” ACM Transactions on Graphics (ACM SIGGRAPH 2005), Vol. 24, No. 3, pp. 861-868, 2005.
[71] M. Szummer, and R. W. Picard, “Temporal Texture Modeling,” International Conference on Image Processing, Vol. 3, pp. 823-826, 1996.
[72] P. Tan, S. Lin, L. Quan, and H. Y. Shum, “Highlight Removal by Illumination- Constrained Inpainting,” IEEE Conference on Computer Vision, Vol. 1, pp. 164-169, 2003.
[73] Y. Tsin, Y. Liu, and V. Ramesh, "Texture Replacement in Real Images," IEEE Conference on Computer Vision and Pattern Recognition, pp. 539-544, 2001.
[74] J. F. Wang, H. J. Hsu, and S. C. Liao, “A Novel Framework for Object Removal from Digital Photograph,” IEEE Conference on Image Processing, Vol.2, pp. 73-76, 2005.
[75] L. Y. Wei and M. Levoy, “Fast Texture Synthesis Using Tree-structured Vector Quantization,” ACM SIGGRAPH, pp. 479-488, 2000.
[76] L. Y. Wei, “Texture Synthesis from Multiple Sources,” ACM Transactions on Graphics (ACM SIGGRAPH 2003), Vol. 22, No. 3, pp. 1-1, 2003.
[77] Y. Wexler, E. Shechtma, and M. Irani, “Space-Time Video Completion,” IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1, pp. 120-127, 2004.
[78] Q. Wu and Y. Yu, “Feature Matching and Deformation for Texture Synthesis,” ACM Transactions on Graphics (ACM SIGGRAPH 2004), Vol. 23, No. 3, pp. 364-367, 2004.
[79] H. Yamauchi, J. Haber, and H. P. Seidel, “Image Restoration Using Multiresolution Texture Synthesis and Image Inpainting,” Computer Graphics International, pp. 120-125, 2003.
[80] A. Zalesny, V. Ferrari, G. Caenen, and L. Van Gool, “Parallel Composite Texture Synthesis," Texture 2002 Workshop in conjunction with ECCV 2002, pp. 151-155, 2002.
[81] A. Zalesny, V. Ferrari, G. Caenen, and L. Van Gool, "Composite Texture Synthesis," International Journal of Computer Vision, Vol. 62, No. 1-2, pp.161-176, 2005.
[82] S. Zelinka, and M. Garland, “Towards Real-Time Texture Synthesis with the Jump Map,” Eurographics Organization, pp. 99-104, 2002.