高光譜影像具有豐富的光譜資訊，能夠提供各種圖像相關的任務更高的準確率，因此被廣泛應用。然而，由於必須面臨在空間分辨率和頻譜波段數量之間的取捨，取得擁有高解析度的高光譜影像是一項極具挑戰性的任務。我們融合了硬體和軟體的設計，以解決這一問題。更具體地說，硬體使用超穎介面驅動的裝置來取得一個四波段影像，而軟體則將其後處理，變成高品質的高光譜影像。在這篇論文中，我們提出了一個演算法，稱為「基於Transformer網絡的去模糊和使用CODE理論的高光譜重建（Deform-CORE）」，以實現軟體端的目標。首先，該演算法使用一個經過精心設計的網路（Deform），將從超穎介面拍攝出的模糊四波段影像去模糊過後轉換成乾淨的四波段影像。然後，頻譜重建模塊（CORE）能夠在不犧牲影像空間解析度的情況下產生對應的十八波段影像。CORE是透過將凸優化與深度學習兩個技巧的結合來實現的。透過深度學習的策略，我們能夠提取複雜的空間資訊並為凸優化提供正則化項。透過凸優化，我們降低了對大量數據的要求，只需要深度學習提供粗估解。最終，我們成功地利用有限的訓練數據（20張）恢復了高品質的高光譜影像，這使得從超穎介面所取得的影像具有實際應用的能力。

Hyperspectral image (HSI) has been widely used in image-related tasks because it provides a higher accuracy attributed to its abundant spectral information. However, acquiring high-resolution HSIs is a challenging mission and has a tradeoff between spatial resolution and the number of spectral bands. To solve this, we cooperate in designing hardware and software. In contrast, the hardware captures a four-bands image in a metasurface-driven device, and the software post-processes the image into a high-quality hyperspectral image. This thesis designs an algorithm, termed deblurring with Transformer-based network and hyperspectral reconstruction using CODE theory (Deform-CORE), to achieve the target of software. It first transforms the blurred four-band image obtained from the metasurface-empowered chip into a clean four-band image with a well-designed network, Deform. Then spectral reconstruction block, CORE, generates the corresponding eighteen-band image without sacrificing spatial resolution. CORE is achieved by combining convex optimization with deep learning. With deep learning, we can extract complicated spatial information and provide a regularization term for convex optimization. With convex optimization, we reduce the requirement for a large amount of data and only need deep learning to provide a rough estimate of the solution. In the end, we successfully recover a high-quality hyperspectral image with limited training data, which makes metasurface-acquired images feasible for practical applications.

[1] R. M. Haralick, K. Shanmugam, and I. H. Dinstein, “Textural features for image classification,” IEEE Transactions on Systems, Man, and Cybernetics, no. 6, pp. 610–621, 1973.
[2] G. Oliveira, X. Frazão, A. Pimentel, and B. Ribeiro, “Automatic graphic logo detection via fast region-based convolutional networks,” in 2016 International Joint Conference on Neural Networks (IJCNN), 2016, pp. 985–991.
[3] L. J. Rickard, R. W. Basedow, E. F. Zalewski, P. R. Silverglate, and M. Landers, “HYDICE: An airborne system for hyperspectral imaging,” in Imaging Spectrometry of the Terrestrial Environment, vol. 1937, 1993, pp. 173–179.
[4] S. Liu, D. Marinelli, L. Bruzzone, and F. Bovolo, “A review of change detection in multitemporal hyperspectral images: Current techniques, applications, and challenges,” IEEE Geoscience and Remote Sensing Magazine, vol. 7, no. 2, pp. 140–158, 2019.
[5] T. Vo-Dinh, “A hyperspectral imaging system for in vivo optical diagnostics,” IEEE Engineering in Medicine and Biology Magazine, vol. 23, no. 5, pp. 40–49, 2004.
[6] C.-H. Lin, J. M. B. Dias, T.-H. Lin, Y.-C. Lin, and C.-H. Kao, “A new hyperspectral compressed sensing method for efficient satellite communications,” in 2020 IEEE 11th Sensor Array and Multichannel Signal Processing Workshop (SAM), 2020, pp. 1–5.
[7] C.-H. Lin and T.-H. Lin, “All-addition hyperspectral compressed sensing for metasurface-driven miniaturized satellite,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–15, 2022.
[8] J. Hu, S. Bandyopadhyay, Y.-h. Liu, and L.-y. Shao, “A review on metasurface: from principle to smart metadevices,” Frontiers in Physics, vol. 8, p. 586087, 2021.
[9] N. Shlezinger, G. C. Alexandropoulos, M. F. Imani, Y. C. Eldar, and D. R. Smith, “Dynamic metasurface antennas for 6g extreme massive mimo communications,” IEEE Wireless Communications, vol. 28, no. 2, pp. 106–113, Apr. 2021.
[10] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang, and L. Shao, “Multi-stage progressive image restoration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2021, pp. 14 821–14 831.
[11] L. Chen, X. Lu, J. Zhang, X. Chu, and C. Chen, “HINet: Half instance normalization network for image restoration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Jun. 2021, pp. 182–192.
[12] J. Li, C. Wu, R. Song, Y. Li, and F. Liu, “Adaptive weighted attention network with camera spectral sensitivity prior for spectral reconstruction from RGB images,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June Jun. 2020, pp. 462–463.
[13] Y. Cai, J. Lin, Z. Lin, H. Wang, Y. Zhang, H. Pfister, R. Timofte, and L. Van Gool, “MST++: Multi-stage spectral-wise transformer for efficient spectral reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Jun. 2022, pp. 745–755.
[14] A. Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations, 2021.
[15] S. Mei, R. Jiang, X. Li, and Q. Du, “Spatial and spectral joint super-resolution using convolutional neural network,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 7, pp. 4590–4603, 2020.
[16] C.-H. Lin, Y.-C. Lin, and P.-W. Tang, “ADMM-ADAM: A new inverse imaging framework blending the advantages of convex optimization and deep learning,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–16, 2022.
[17] C.-Y. Chi, W.-C. Li, and C.-H. Lin, Convex Optimization for Signal Processing and Communications: From Fundamentals to Applications. Boca Raton, FL, USA: CRC Press, 2017.
[18] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Restormer: Efficient transformer for high-resolution image restoration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2022, pp. 5728–5739.
[19] C.-H. Lin, F. Ma, C.-Y. Chi, and C.-H. Hsieh, “A convex optimization-based coupled nonnegative matrix factorization algorithm for hyperspectral and multispectral data fusion,” IEEE Transactions on Geoscience and Remote Sensing, vol. 56, no. 3, pp. 1652–1667, Mar. 2018.
[20] X. Hu, Y. Cai, J. Lin, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. Van Gool, “HDNet: High-resolution dual-domain learning for spectral compressive imaging,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2022, pp. 17 542–17 551.
[21] J. M. Bioucas-Dias and J. M. P. Nascimento, “Hyperspectral subspace identification,” IEEE Transactions on Geoscience and Remote Sensing, vol. 46, no. 8, pp. 2435–2445, Aug. 2008.
[22] C.-H. Lin and J. M. Bioucas-Dias, “An explicit and scene-adapted definition of convex self-similarity prior with application to unsupervised Sentinel-2 super-resolution,” IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 5, pp. 3352–3365, May 2020.
[23] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.