三維場景在現今有非常多的應用，無論是虛擬實境、電腦遊戲、物理模擬或是醫學用途等等。由二維影像進行三維場景重建是非常熱門的問題，其中又可以分為多張影像與單一影像的三維場景重建。多張影像的場景重建已有許多成熟的方法可以實現。單一影像的三維場景重建因缺少太多必要的資訊，是一個非常有挑戰性的問題，至今此方向的研究仍有許多限制。

本研究會提出方法，在兩個條件下，使用單張影像重建出一個三維場景。為了確保重建後場景中的物體能有完整的形狀，我們需要第一個條件：可以取得該類似該類型物體的三維模型作為樣板模型(template model) 輸入；為了合成出完整的貼圖，我們需要第二個條件：輸入影像中具有重複的同類型物體，且這些物體有稍微不同的面向。例如一朵蓮花的照片，有許多形狀相似但角度不同的花瓣，此時我們只要在網路上找到一個花瓣的三維模型作為樣板，並透過不同角度的花瓣合成每個花瓣的貼圖，即可重建出三維場景。

隨網際網路資源日益豐富，可以自由取得的模型資料也愈來愈多，本研究因而使用樣版模型進行三維重建。隨著時間推進，使用者可以愈來愈輕鬆的從網際網路上取得三維模型，需要輸入樣板模型的條件要求也會隨時間降低。

本研究會將問題劃分為三個子問題，分別是三維模組與影像上物體的形狀匹配、網格變形與貼圖合成。解決這些問題之後，使用者即可建立出一個三維的場景。

Many applications, such as virtual reality, computer games, physics simulations, and medical applications, use 3D scenes. Reconstructing 3D scenes from 2D images is common and can be done in one of two ways: (1) from multiple images and (2) from a single image. While many well-developed methods exist for 3D reconstruction from multiple images, a lack of essential information means that 3D reconstruction from single images remains a challenge.

We propose a method for reconstructing 3D scenes from a single image under two conditions. To complete the shapes of the objects in the reconstructed scene, we need the first condition: 3D template models similar to objects in the input image are available. To synthesize a complete texture, we need the second condition: the input image contains duplicate objects; these objects are repeated in slightly different directions. For example, take an image of a lotus, which has many similar petals that lie in slightly different directions. We can reconstruct the 3D scene from this image if we find a 3D model of a petal from the Internet and synthesize the texture of each petal by considering information from other petals.

In the present study, we use template models to reconstruct 3D scenes because 3D model files are now more publicly available on the Internet than in the past.

The entire work is divided into three sub-problems: (1) model fitting, (2) mesh deformation, and (3) texture synthesis. We can reconstruct the 3D scene after resolving these problems.

[1] D. Hoiem, A. A. Efros, and M. Hebert, “Automatic photo pop-up,” ACM transactions on graphics (TOG), vol. 24, no. 3, pp. 577–584, 2005.
[2] A. Saxena, M. Sun, and A. Y. Ng, “Make3d: Learning 3d scene structure from a single still image,” IEEE transactions on pattern analysis and machine intelligence, vol. 31, no. 5, pp. 824–840, 2009.
[3] O. Barinova, V. Konushin, A. Yakubenko, K. Lee, H. Lim, and A. Konushin, “Fast automatic single-view 3-d reconstruction of urban scenes,” in European Conference on Computer Vision, pp. 100–113, Springer, 2008.
[4] T. Igarashi, S. Matsuoka, and H. Tanaka, “Teddy: a sketching interface for 3d freeform design,” in Acm siggraph 2007 courses, p. 21, ACM, 2007.
[5] B. M. Oh, M. Chen, J. Dorsey, and F. Durand, “Image-based modeling and photo edit- ing,” in Proceedings of the 28th annual conference on Computer graphics and interac- tive techniques, pp. 433–442, ACM, 2001.
[6] F. Yan, M. Gong, D. Cohen-Or, O. Deussen, and B. Chen, “Flower reconstruction from a single photo,” in Computer Graphics Forum, vol. 33, pp. 439–447, Wiley Online Library, 2014.
[7] R. Guo, C. Zou, and D. Hoiem, “Predicting complete 3d models of indoor scenes,”arXiv preprint arXiv:1504.02437, 2015.
[8] C.-K. Yeh, S.-Y. Huang, P. K. Jayaraman, C.-W. Fu, and T.-Y. Lee, “Interactive high- relief reconstruction for organic and double-sided objects from a photo,” IEEE trans- actions on visualization and computer graphics, vol. 23, no. 7, pp. 1796–1808, 2017.
[9] P. J. Besl, N. D. McKay, et al., “A method for registration of 3-d shapes,” IEEE Trans- actions on pattern analysis and machine intelligence, vol. 14, no. 2, pp. 239–256, 1992.
[10] P. Pérez, M. Gangnet, and A. Blake, “Poisson image editing,” in ACM Transactions on graphics (TOG), vol. 22, pp. 313–318, ACM, 2003.
[11] D. Hoiem, A. A. Efros, and M. Hebert, “Geometric context from a single image,” in Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conference on, vol. 1, pp. 654–661, IEEE, 2005.
[12] D. Hoiem, A. A. Efros, and M. Hebert, “Recovering surface layout from an image,”International Journal of Computer Vision, vol. 75, no. 1, p. 151, 2007.
[13] M. Prasad and A. Fitzgibbon, “Single view reconstruction of curved surfaces,” in Com- puter Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 2, pp. 1345–1354, IEEE, 2006.
[14] N. Kholgade, T. Simon, A. Efros, and Y. Sheikh, “3d object manipulation in a single photograph using stock 3d models,” ACM Transactions on Graphics (TOG), vol. 33, no. 4, p. 127, 2014.
[15] L. Quan, P. Tan, G. Zeng, L. Yuan, J. Wang, and S. B. Kang, “Image-based plant modeling,” in ACM Transactions on Graphics (TOG), vol. 25, pp. 599–604, ACM, 2006.
[16] P. Tan, G. Zeng, J. Wang, S. B. Kang, and L. Quan, “Image-based tree modeling,” in ACM Transactions on Graphics (TOG), vol. 26, p. 87, ACM, 2007.
[17] D. Sỳkora, L. Kavan, M. Čadík, O. Jamriška, A. Jacobson, B. Whited, M. Simmons, and O. Sorkine-Hornung, “Ink-and-ray: Bas-relief meshes for adding global illumination effects to hand-drawn characters,” ACM Transactions on Graphics (TOG), vol. 33, no. 2, p. 16, 2014.
[18] A. Rivers, T. Igarashi, and F. Durand, “2.5 d cartoon models,” in ACM Transactions on Graphics (TOG), vol. 29, p. 59, ACM, 2010.
[19] C.-K. Yeh, P. K. Jayaraman, X. Liu, C.-W. Fu, and T.-Y. Lee, “2.5 d cartoon hair modeling and manipulation,” IEEE transactions on visualization and computer graphics, vol. 21, no. 3, pp. 304–314, 2015.
[20] Y. Iwashita, R. Kurazume, K. Konishi, M. Nakamoto, M. Hashizume, and T. Hasegawa,“Fast alignment of 3d geometrical models and 2d grayscale images using 2d distance maps,” Systems and Computers in Japan, vol. 38, no. 14, pp. 52–62, 2007.
[21] V. Kraevoy, A. Sheffer, and M. van de Panne, “Contour-based modeling using deformable 3d templates,” tech. rep., Tech Report TR-2007-13, CS, 2007.
[22] Y. Kho and M. Garland, “Sketching mesh deformations,” in Acm siggraph 2007 courses, p. 41, ACM, 2007.
[23] A. Nealen, O. Sorkine, M. Alexa, and D. Cohen-Or, “A sketch-based interface for detailpreserving mesh editing,” in ACM SIGGRAPH 2007 courses, p. 42, ACM, 2007.
[24] M. Bertalmio, A. L. Bertozzi, and G. Sapiro, “Navier-stokes, fluid dynamics, and image and video inpainting,” in Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, vol. 1, pp. I–I, IEEE, 2001.
[25] A. Telea, “An image inpainting technique based on the fast marching method,” Journal of graphics tools, vol. 9, no. 1, pp. 23–34, 2004.
[26] A. Criminisi, P. Pérez, and K. Toyama, “Region filling and object removal by exemplarbased image inpainting,” IEEE Transactions on image processing, vol. 13, no. 9, pp. 1200–1212, 2004.
[27] C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman, “Patchmatch: A randomized correspondence algorithm for structural image editing,” ACM Trans. Graph., vol. 28, no. 3, pp. 24–1, 2009.
[28] K. Zhou, X. Wang, Y. Tong, M. Desbrun, B. Guo, and H.-Y. Shum, “Texturemontage,” ACM Transactions on Graphics (TOG), vol. 24, no. 3, pp. 1148–1155, 2005.
[29] J. Jeon, Y. Jung, H. Kim, and S. Lee, “Texture map generation for 3d reconstructed scenes,” The Visual Computer, vol. 32, no. 6-8, pp. 955–965, 2016.
[30] Y. Boykov and V. Kolmogorov, “An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision,” IEEE transactions on pattern analysis and machine intelligence, vol. 26, no. 9, pp. 1124–1137, 2004.
[31] S. Suzuki et al., “Topological structural analysis of digitized binary images by border following,” Computer vision, graphics, and image processing, vol. 30, no. 1, pp. 32–46, 1985.
[32] P. Alliez, S. Tayeb, and C. Wormser, “CGAL 4.10 - 3D Fast Intersection and Distance Computation (AABB Tree).” http://doc.cgal.org/latest/AABB_tree/index.html, 2017.
[33] H. W. Kuhn, “The hungarian method for the assignment problem,” Naval Research Logistics (NRL), vol. 2, no. 1-2, pp. 83–97, 1955.
[34] M. Meyer, M. Desbrun, P. Schröder, and A. H. Barr, “Discrete differential-geometry operators for triangulated 2-manifolds,” in Visualization and mathematics III, pp. 35–57, Springer, 2003.
[35] O. Sorkine and M. Alexa, “As-rigid-as-possible surface modeling,” in Symposium on Geometry processing, vol. 4, 2007.
[36] A. Sheffer, E. Praun, K. Rose, et al., “Mesh parameterization methods and their applications,” Foundations and Trends® in Computer Graphics and Vision, vol. 2, no. 2, pp. 105–171, 2007.