立體匹配演算法產生的立體影像序列，是針對兩台水平攝影機拍攝的影像作計算，找出左右影像的對應點，推算出物體的深度。由於，搜尋對應點的演算法會參考到目標點四周區域來做相似度的比較，將有最小匹配成本的選作為最佳的對應點。因此在做匹配時會因為影像中物體遮蔽、缺乏紋理等問題導致深度資訊的錯誤，而在影片序列上則是會遇到雜訊與光線變化的情況而影響最佳解的結果，導致相同對應點的深度在時間軸上的變化呈現跳動的現象。這些時間上的深度錯誤會嚴重減弱合成虛擬視角的品質。
本篇論文提出了應用單一像素深度軌跡與色度軌跡的特性改善遮蔽、雜訊的問題，之後利用牛頓運動定理校正單一像素在不同時間點上的深度值與利用Bilateral Filter做空間上保留邊緣去除雜訊。這個演算法分為四個部分，第一個部分是利用色度軌跡判斷深度軌跡中屬於遮蔽的區域，並加以改善，第二部分是將初始的深度軌跡與使用中值濾波器後的深度軌跡做比對，找出突出的雜訊，並針對此加以改善。第三的部分是利用牛頓定理推估以及校正深度值在時間軸上的變化，維持同一物體在時間軸上的深度變化的一致性。第四部分是在空間上對每個像素本身與鄰近的深度值做加權平均，去除空間上的雜訊。

Stereo matching algorithm estimate a disparity map based on spatial correspondence with two image taken from two horizontal cameras. All of the stereo matching methods suffer from textureless, occlusion region, and in video sequence, the matching cost of the same object can vary from its neighborhood and the video noise. If we combine these consecutive disparity maps into one depth video will cause temporal depth flickering. It will reduce the quality of synthesized views.

This paper proposes a novel method for occlusion、noise detection and modification based on the relationship of depth trajectory of single point and color intensity trajectory of single point. And we use Newton's Laws of Motion to give a constraint on variation of depth temporally. Finally we use a bilateral filter to reduce the noise of disparity map on a same object spatially.

[1] K.-J. Yoon and I.-S. Kweon.” Adaptive support-weight approach for correspondence search. ” PAMI 2006.

[2] K. Zhang, J. Lu, and G. Lafruit. “Cross-based local stereo matching using orthogonal integral images.” IEEE TCSVT 2009.

[3] A. Hosni, M. Bleyer, M. Gelautz, and C. Rhemann.” Local stereo matching using geodesic support weights. “ICIP 2009.

[4] Dengfeng Chai and Qunsheng Peng. “Bilayer Stereo Matching.” IEEE International Conference on Computer Vision (ICCV 2007), Oct. 2007.

[5] Jong Dae Oh, Siwei Ma and C-CJ Kuo. “Stereo Matching via Disparity Estimation and Surface Modeling.” IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2007), Jun. 2007.

[6] Yi Deng, Qiong Yang, Xueyin Lin and Xiaoou Tang. “Stereo correspondence with Occlusion Handling in a Symmetric Patch-Based Graph-Cuts Model.” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29 (6) pp. 1068 - 1079, 2007.

[7] QingXiong Yang, Liang Wang, Ruigang Yang, H Stewenius and David Nister. “Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling.” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31 (3) pp. 492 - 504, 2009.

[8] Wan-Yu Chen, Yu-Lin Chang, Shyh-Feng Lin, Li-Fu Ding and Liang-Gee Chen, "Efficient Depth Image Based Rendering with Edge Dependent Depth Filter and Interpolation." IEEE International Conference on Multimedia and Expo, 2005. ICME 2005, pp. 1314-1317, 2005

[9] Sergey Smirnov, Atanas Gotchev, Karen Egiazarian, “A Memory-efficient and Time-consistentFiltering of Depth Map Sequences”, In Proceedings of SPIE Image Processing Algorithms andSystems, 7532, 75317, 2010.

[10] Hosni A., C. Rhemann, M. Bleyer and M. Gelautz, Temporally Consistent Disparity and Optical Flow via Efficient Spatio-temporal Filtering, Pacific-Rim Symposium on Image and Video Technology (PSIVT) 2011

[11] Christian Richardt, Douglas Orr, Ian Davies, Antonio Criminisi, Neil A. Dodgson, “Real-time Spatiotemporal Stereo Matching Using the Dual-Cross-Bilateral Grid”, In Proceedings of the 11th European conference on computer vision conference on Computer vision, pp.510-523, 2010.

[12] Asmaa Hosni, Christoph Rhemann, Michael Bleyer, Margrit Gelautz,“Temporally Consistent Disparity and Optical Flow via Efficient Spatiotemporal Filtering”, In Proceedings of the Fifth Pacific-Rim Symposium on Image and Video Technology, 2011.

[13] Hai Tao, Harpreet S. Sawhney, Rakesh Kumar, “Dynamic depth recovery from multiple synchronized video streams”, In Proceedings of IEEE Computer Society Conference on CVPR, vol. 1, pp.118-124, 2001.

[14] Guofeng Zhang, Jiaya Jia, Tien-Tsin Wong, Hujun Bao, “Consistent Depth Maps Recovery from a Video Sequence”, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 6, pp.974-988, 2009.

[15] E. Scott Larsen, Philippos Mordohai, Marc Pollefeys, Henry Fuchs, “Temporally Consistent Reconstruction from Multiple Video Streams Using Enhanced Belief Propagation”, In Proceedings of IEEE 11th International Conference on Computer Vision, pp.1-8, 2007.

[16] Shaojie Qiao, Changjie Tang, Huidong Jin, Teng Long, Shucheng Dai,Yungchang Ku,Michael Chau: PutMode: prediction of uncertain trajectories in moving objects databases

[17] Li Hong and George Chen. “Segment-based stereo matching using graph cuts.” IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2004), Jul. 2004, Washington, DC, USA.
[18] Yuri Boykov, Olga Veksler and Ramin Zabih. “Fast approximate energy minimization via graph cuts.” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23 (11) pp. 1222-1239, 2001.
[19] Goce Trajcevski,Ouri Wolfson,Fengli Zhang, and Sam Chamberlain,” the geometry of uncertainty in moving objects databases” n: EDBT, Vol. 2287Springer (2002) , p. 233-250.

[20] Teng Long, Shaojie Qiao, Changjie Tang, Liangxu Liu, Taiyong Li, and Jiang Wu, “E3TP: A Novel Trajectory Prediction Algorithm in Moving Objects Databases” : PAISI 2009, LNCS 5477, pp. 76–88, 2009.

[21] Dongbo Min, Sehoon Yea and Anthony Vetro , “TEMPORALLY CONSISTENT STEREO MATCHING USING COHERENCE FUNCTION,” Proc. IEEE 3DTV Conference, June 2010

[22] Chun-Rong Huang , Chu-Song Chen, and Pau-Choo Chung, “Contrast Context Histogram – A Discriminating Local Descriptor for Image Matching,” In Proceedings of 18 th IAPR International Conference on Pattern Recognition , ICPR'06, vol.4, pp. 53-56, Hong Kong, Aug., 2006.

[23] Li Yao, , Dong-Xiao Li, , Ming Zhang, "A Temporally Streamlined Optimization Method for Stereo Video Correspondence", IJACT: International Journal of Advancements in Computing Technology, Vol. 4, No. 2, pp. 238 ~ 246, 2012

[24] Min Ki Park, Ji-Ho Cho, In yeop Jang, Seung Joo Lee and Kwan H. Lee, "An Iterative Joint Bilateral Filtering for Depth Refinement of a 3D Model", ACM SIGGRAPH Asia 2011 Poster, 12-15 December 2011, Hong Kong.

[25] L. Wang, M. Liao, M. Gong, R. Yang, and D. Nistér, “High-quality real-time stereo using adaptive cost aggregation and dynamic programming.” 3rd International Symposium on 3D Data Processing, Visualization, and Transmission, 3DPVT 2006, pp. 798-805, 2006

[26] C. Tomasi and R. Manduchi, “ Bilateral Filtering for Gray and Color Images,” Proceedings of the 1998 IEEE International Conference on Computer Vision, Bombay, India, 1998.