為了節省生產成本，現代的數位相機大多採用單感光元件架構，透過色彩濾波陣列在每個取樣點記錄不同的色彩資訊，再藉由色彩濾波陣列插補法或稱解馬賽克技術重建出全彩的圖像。近年來，許多基於深度學習的解馬賽克方法也被提出，基於深度學習的解馬賽克方法多採用端對端的網絡架構，輸入的色彩濾波陣列圖像經深度神經網絡運算後，便可重建出全彩圖像。然而，為了得到更高品質的解馬賽克結果，神經網絡的架構不斷加深，使網絡難以訓練並大幅增加模型推論所需的時間。
本論文將傳統演算法中的色差平面概念導入深度神經網絡，設計一用於解馬賽克之三階段殘差網絡架構。網絡的第一階段負責重建圖像的綠色通道，第二階段負責重建圖像的紅色與藍色通道，第三階段則藉由融合前兩階段重建的紅綠藍三個通道間的資訊，對整體網絡進行微調，並產生最終的重建圖像。所提出的網絡架構中，包含了獨創的色差殘差單元與色差導引單元，前者使用色差平面資訊重建色彩通道，避免解馬賽克殘差網絡中常見的初始誤差；後者則使用重建的綠色通道引導網絡的第二階段，提取有效的色差平面特徵。透過這兩種獨創單元，所提出的網絡能明確地利用色差平面中的資訊，從而在無需建構極深的網絡架構的同時，達到優良的解馬賽克品質。
實驗結果顯示，所提出的方法與參數量相近的網絡架構相比，在Kodak標準測試集可提升約0.23分貝的CPSNR，並減少約84.6%的模型推論時間，在兼顧重建圖像品質的同時，大幅降低網絡運算成本。

To reduce the cost, most modern digital cameras adopt the architecture of single image sensor. At each sample point, the intensity value of a specific color channel is recorded based on the pattern of color filter array (CFA). Consequently, CFA interpolation, which is also called image demosaicking, is performed to reconstruct the full-color image. In recent years, many deep learning-based image demosaicking methods had been proposed. Most of these methods adopted an end-to-end architecture, which simply took an input CFA image and generated the reconstructed full-color image. However, the structures of the deep neural networks were getting deeper and deeper to obtain high-quality demosaicking results. As a result, the training processes of these networks were more difficult and the required inference time significantly increased.
In this thesis, we introduce the concepts of color difference planes in traditional demosaicking algorithms to residual network and propose a color difference-based three-stage residual demosaicking network (CDTSRDN). The first stage of the proposed network reconstructs the green (G) channel of image. The second stage of the proposed network reconstructs the red (R) and blue (B) channels of image. The third stage of the proposed network fuses the information between the reconstructed RGB channels to fine-tune the full network, and the full-color image is obtained. In the proposed network, the novel color difference residual units (CDRUs) and color difference guidance unit (CDGU) are designed. The CDRUs utilize the information of color difference planes to recover the missing colors, thus preventing the initial errors from the residual network. The CDGU extracts the color difference features for the second stage of the proposed network with the guidance of the reconstructed G channel. With the CDRUs and CDGU, the CDTSRDN can reconstruct high-quality full-color image without stacking extremely deep network structure.
The experimental results show that the CDTSRDN can improve the CPSNR by 0.23 dB on the Kodak dataset compared with previously proposed network with a similar number of parameters. Moreover, the inference time can be reduced by 84.6%. The proposed network can not only produce high-quality demosaicking results, but also significantly reduce the computational loads.

[1] B. E. Bayer, “Color imaging array,” U.S. Patent 3 971 065, Jul. 1976.
[2] B. K. Gunturk, Y. Altunbasak and R. M. Mersereau, “Color plane interpolation using alternating projections,” in IEEE Transactions on Image Processing, vol. 11, no. 9, pp. 997-1013, Sep. 2002.
[3] S.-C. Pei and I.-K. Tam, “Effective color interpolation in CCD color filter arrays using signal correlation,” IEEE Trans. Circuits Syst. Video Technol., vol. 13, no. 6, pp. 503-513, Jun. 2003.
[4] L. Zhang, X. Wu, A. Buades, and X. Li, “Color demosaicking by local directional interpolation and nonlocal adaptive thresholding,” J. Electron. Imag., vol. 20, no. 2, Apr. 2011, Art. no. 023016.
[5] S.-L. Chen and E.-D. Ma, “VLSI implementation of an adaptive edge-enhanced color interpolation processor for real-time video applications,” IEEE Trans. Circuits Syst. Video Technol., vol. 24, no. 11, pp. 1982–1991, Nov. 2014.
[6] D. Kiku, Y. Monno, M. Tanaka, and M. Okutomi, “Residual interpolation for color image demosaicking,” in Proc. IEEE Int. Conf. Image Process. (ICIP), pp. 2304–2308, Sep. 2013.
[7] D. Kiku, Y. Monno, M. Tanaka, and M. Okutomi, “Minimized-Laplacian residual interpolation for color image demosaicking,” in Proc. SPIE, vol. 9023, pp. 90230L-1-90230L-8, Mar. 2014.
[8] D. Kiku, Y. Monno, M. Tanaka, and M. Okutomi, “Beyond color difference: Residual interpolation for color image demosaicking,” IEEE Trans. Image Process., vol. 25, no. 3, pp. 1288–1300, Mar. 2016.
[9] Y. Monno, D. Kiku, M. Tanaka, and M. Okutomi, “Adaptive residual interpolation for color and multispectral image demosaicking,” Sensors, vol. 17, no. 12, p. 2787, Dec. 2017.
[10] C.-Y. Lien, F.-J. Yang, P.-Y. Chen, and Y.-W. Fang, “Efficient VLSI architecture for edge-oriented demosaicking,” IEEE Trans. Circuits Syst. Video Technol., vol. 28, no. 8, pp. 2038–2047, Aug. 2018.
[11] Y. Niu, J. Ouyang, W. Zuo and F. Wang, “Low Cost Edge Sensing for High Quality Demosaicking,” IEEE Transactions on Image Processing, vol. 28, no. 5, pp. 2415-2427, May 2019.
[12] T.-L. Lin, C.-W. Chang, J.-S. Tu, X.-F. Yue and J.-R. Yang, “Edge-Based Demosaicking Method Using Uncorrelatedness With Sensors for CFA,” IEEE Sensors Journal, vol. 19, no. 19, pp. 8904-8912, Oct. 2019.
[13] K. Zhang, W. Zuo, S. Gu, and L. Zhang, “Learning deep CNN denoiser prior for image restoration,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., vol. 2, pp. 2808–2817, 2017.
[14] R. J. Tan, K. Zhang, W. M. Zuo, and L. Zhang, “Color image demosaicking via deep residual learning,” in Proc. IEEE Int. Conf. Multimedia Expo., pp. 1–5, Jul. 2017.
[15] K. Cui, Z. Jin, and E. Steinbach, “Color image demosaicking using a 3-Stage convolutional neural network structure,” in Proc. 25th IEEE Int. Conf. Image Process. (ICIP), pp. 2177–2181, Oct. 2018.
[16] Y. Zhang, K. Li, K. Li, B. Zhong, and Y. Fu, “Residual non-local attention networks for image restoration,” in Proc. Int. Conf. Learn Representations (ICLR), 2019.
[17] Y. Wang, S. Yin, S. Zhu, Z. Ma, R. Xiong, and B. Zeng, “NTSDCN: New three-stage deep convolutional image demosaicking network,” IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 9, pp. 3725–3729, Sep. 2021.
[18] C. Mou, J. Zhang, and Z. Wu, “Dynamic attentive graph learning for image restoration,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., pp. 4308–4317, 2021.
[19] W. Xing and K. Egiazarian, “Residual swin transformer channel attention network for image demosaicing,” in European Workshop on Visual Information Processing (EUVIP), pp. 1–6, 2022.
[20] X. Li, Y. Niu, B. Zhao, H. Shi, and Z. An, “Toward Moiré-free and detail-preserving demosaicking,” arXiv: 2305.08585, 2023.
[21] K. He, J. Sun, and X. Tang, “Guided image filtering,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 35, no. 6, pp. 1397–1409, 2013.
[22] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 770–778, Jun. 2016.
[23] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” in Proc. Int. Conf. Learn Represent., pp. 1–5, May 2016.
[24] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei, “Deformable convolutional networks,” in Proc. Int. Conf. Comput. Vis., pp. 764–773, 2017.
[25] W. Shi, J. Caballero, F. Huszár, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang, “Real-time single image and video super resolution using an efficient sub-pixel convolutional neural network,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 1874-1883, 2016.
[26] Kodak Image Dataset. Accessed: Jun. 20, 2021. [Online]. Available: http://r0k.us/graphics/kodak/
[27] McMaster Image Dataset. Accessed: Jul. 29, 2021. [Online]. Available: http://www4.comp.polyu.edu.hk/~cslzhang/CDM_Dataset.htm
[28] K. Ma et al., “Waterloo exploration database: New challenges for image quality assessment models,” IEEE Trans. Image Process., vol. 26, no. 2, pp. 1004–1016, Feb. 2017.
[29] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. Int. Conf. Learn Represent., May 2015, pp. 1–15.