侵入式腦機介面的發展旨在協助癱瘓的病人恢復生活日常之機能，透過對運動皮層的神經訊號與相對應時刻的運動狀態建模，由腦中神經訊號的活動來判斷該時刻受試者的運動意圖，利用腦機介面對外部裝置的控制，實現癱瘓病人的運動功能。然而對於侵入式腦機介面而言，無論是傳統的線性模型亦或是現代的神經網路模型，神經記錄條件的改變時常導致解碼模型的失效，為了保持模型的一致性與有效性，我們必須不斷地對模型進行校正，在腦機介面使用前大量收集當日所需的訓練資料，並重新對模型進行訓練，這個過程通常需要耗費大量的時間成本，因此本研究目標藉由深度學習領域中的遷移學習技巧，利用過去的實驗記錄來協助當日模型的校正。
我們使用一隻恆河猴共37個實驗區間的實驗記錄，動物行為實驗為隨機光標的到達任務，其大腦中運動皮層植有96通道的微電極陣列，記錄實驗過程中的神經活動，並同時記錄其手指的運動軌跡。我們首先觀察神經記錄條件的變化，並以t-SNE方法分析不同實驗區間神經活動的分布。對於校正方法，嘗試了兩種基於深度學習的域適應校正方法，先將前一個實驗區間的記錄定義為源域資料，而當前實驗區間的記錄為目標資料，透過模型訓練的過程，使源域和目標域特徵於神經網路中的隱藏狀態具有相似的分布，解決神經記錄條件所造成的差異，同時可以進行解碼的預測，另外透過遷移學習中模型的微調方法進行解碼器的校正，並與一般的模型校正方法做比較。
結果顯示，雖然兩種域適應方法確實可以使源域和目標域的資料特徵混合，但是對於解碼性能並沒有明顯的幫助，而模型微調的方法不但有助於性能的提升，還可以減少當日所需的訓練資料量，快速進行解碼器的校正，並且連續的微調校正有助於解碼器保持高性能的預測。

The development of intracortical brain-computer interface is designed to help paralyzed patients recover their daily functions of life. Through the modeling of the neural signals of the motor cortex and the movement state of the corresponding time, we can determine the subject’s movement intention at that moment by the activity of neural signals. Using the brain-computer interface to control the external device and realize the movement function of the paralyzed patient. However, for intracortical brain-computer interfaces, whether it is a traditional linear model or a modern neural network model, changes in neural recording conditions often lead to the failure of the decoding model. In order to maintain the consistency and effectiveness of the model, we must continuously calibrate the model. Before using the brain-computer interface, it is necessary to collect a large amount of training data of the day and retrain the model. This process usually takes a lot of time costs. Therefore, the goal of this research is to use the transfer learning techniques in deep learning to assist the model calibration of the day by using past experimental records.

We use the experimental records of a rhesus monkey with a total of 37 sessions. The animal behavior experiment is a random cursor reaching task. A 96-channel microelectrode array is implanted in the motor cortex of the brain to record the neural activity during the experiment and the movement trajectory of the finger at the same time. We first observe the changes in neural recording conditions, and use the t-SNE method to analyze the distribution of neural activity in different sessions. For the calibration method, we’ve tried two domain adaptation methods based on deep learning. First, the record of the previous session was defined as the source domain data, and the record of the current session was the target data. Through the process of model training, the source domain and target domain features in the hidden state of the neural network have similar distributions, and the differences caused by the neural recording conditions can be resolved. In addition, the decoder is calibrated by the fine-tuning method in the transfer learning, and compared with the general model calibration method.

The results show that although the two domain adaptation methods can indeed mix the data characteristics of the source and target domains, they do not significantly help the decoding performance. The method of model fine-tuning not only contributes to the improvement of performance, but also reduces the amount of training data required for the day, and quickly calibrates the decoder. Continuous fine-tuning calibration helps the decoder maintain high-performance predictions.

[1] Lago, N. and A. Cester, "Flexible and organic neural interfaces: a review," Applied Sciences, vol. 7, no. 12, pp. 1292, 2017.

[2] Chapin, J.K., et al., "Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex," Nature neuroscience, vol. 2, no. 7, pp. 664-670, 1999.

[3] Serruya, M.D., et al., "Instant neural control of a movement signal," Nature, vol. 416, no. 6877, pp. 141-142, 2002.

[4] Velliste, M., et al., "Cortical control of a prosthetic arm for self-feeding," Nature, vol. 453, no. 7198, pp. 1098-1101, 2008.

[5] Wessberg, J., et al., "Real-time prediction of hand trajectory by ensembles of cortical neurons in primates," Nature, vol. 408, no. 6810, pp. 361-365, 2000.

[6] Taylor, D.M., S.I.H. Tillery, and A.B. Schwartz, "Direct cortical control of 3D neuroprosthetic devices," Science, vol. 296, no. 5574, pp. 1829-1832, 2002.

[7] Hochberg, L.R., et al., "Neuronal ensemble control of prosthetic devices by a human with tetraplegia," Nature, vol. 442, no. 7099, pp. 164-171, 2006.

[8] Kim, S.-P., et al., "Neural control of computer cursor velocity by decoding motor cortical spiking activity in humans with tetraplegia," Journal of neural engineering, vol. 5, no. 4, pp. 455, 2008.

[9] Hochberg, L.R., et al., "Reach and grasp by people with tetraplegia using a neurally controlled robotic arm," Nature, vol. 485, no. 7398, pp. 372-375, 2012.

[10] Collinger, J.L., et al., "High-performance neuroprosthetic control by an individual with tetraplegia," The Lancet, vol. 381, no. 9866, pp. 557-564, 2013.

[11] Gilja, V., et al., "Clinical translation of a high-performance neural prosthesis," Nature medicine, vol. 21, no. 10, pp. 1142, 2015.

[12] Jarosiewicz, B., et al., "Virtual typing by people with tetraplegia using a self-calibrating intracortical brain-computer interface," Science translational medicine, vol. 7, no. 313, pp. 313ra179-313ra179, 2015.

[13] Slutzky, M.W., "Brain-machine interfaces: powerful tools for clinical treatment and neuroscientific investigations," The Neuroscientist, vol. 25, no. 2, pp. 139-154, 2019.

[14] Simeral, J., et al., "Neural control of cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical microelectrode array," Journal of neural engineering, vol. 8, no. 2, pp. 025027, 2011.

[15] Perge, J.A., et al., "Intra-day signal instabilities affect decoding performance in an intracortical neural interface system," Journal of neural engineering, vol. 10, no. 3, pp. 036004, 2013.

[16] Perge, J.A., et al., "Reliability of directional information in unsorted spikes and local field potentials recorded in human motor cortex," Journal of neural engineering, vol. 11, no. 4, pp. 046007, 2014.

[17] Sussillo, D., et al., "Making brain–machine interfaces robust to future neural variability," Nature communications, vol. 7, no. 1, pp. 1-13, 2016.

[18] Jarosiewicz, B., et al., "Advantages of closed-loop calibration in intracortical brain–computer interfaces for people with tetraplegia," Journal of neural engineering, vol. 10, no. 4, pp. 046012, 2013.

[19] Aflalo, T., et al., "Decoding motor imagery from the posterior parietal cortex of a tetraplegic human," Science, vol. 348, no. 6237, pp. 906-910, 2015.

[20] Zhang, P., et al., "Feature-Selection-based Transfer Learning for Intracortical Brain-Machine Interface Decoding," IEEE Transactions on Neural Systems and Rehabilitation Engineering, no., pp., 2020.

[21] Wu, D., "Online and offline domain adaptation for reducing BCI calibration effort," IEEE Transactions on human-machine Systems, vol. 47, no. 4, pp. 550-563, 2016.

[22] Zhang, P., et al., "Decoder calibration with ultra small current sample set for intracortical brain–machine interface," Journal of neural engineering, vol. 15, no. 2, pp. 026019, 2018.

[23] Makin, J.G., et al., "Superior arm-movement decoding from cortex with a new, unsupervised-learning algorithm," Journal of neural engineering, vol. 15, no. 2, pp. 026010, 2018.

[24] Quiroga, R.Q., Z. Nadasdy, and Y. Ben-Shaul, "Unsupervised spike detection and sorting with wavelets and superparamagnetic clustering," Neural computation, vol. 16, no. 8, pp. 1661-1687, 2004.

[25] Watkins, P.T., et al. Validation of adaptive threshold spike detector for neural recording. in The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 2004. IEEE.

[26] Semmaoui, H., et al., "Setting adaptive spike detection threshold for smoothed TEO based on robust statistics theory," IEEE transactions on biomedical engineering, vol. 59, no. 2, pp. 474-482, 2011.

[27] Rey, H.G., C. Pedreira, and R.Q. Quiroga, "Past, present and future of spike sorting techniques," Brain research bulletin, vol. 119, no., pp. 106-117, 2015.

[28] Gerstein, G. and W. Clark, "Simultaneous studies of firing patterns in several neurons," Science, vol. 143, no. 3612, pp. 1325-1327, 1964.

[29] Caro-Martín, C.R., et al., "Spike sorting based on shape, phase, and distribution features, and K-TOPS clustering with validity and error indices," Scientific reports, vol. 8, no. 1, pp. 1-28, 2018.

[30] Harris, K.D., et al., "Accuracy of tetrode spike separation as determined by simultaneous intracellular and extracellular measurements," Journal of neurophysiology, vol. 84, no. 1, pp. 401-414, 2000.

[31] Todorova, S., et al., "To sort or not to sort: the impact of spike-sorting on neural decoding performance," Journal of neural engineering, vol. 11, no. 5, pp. 056005, 2014.

[32] Trautmann, E.M., et al., "Accurate estimation of neural population dynamics without spike sorting," Neuron, vol. 103, no. 2, pp. 292-308. e4, 2019.

[33] Georgopoulos, A.P., A.B. Schwartz, and R.E. Kettner, "Neuronal population coding of movement direction," Science, vol. 233, no. 4771, pp. 1416-1419, 1986.

[34] Carmena, J.M., et al., "Learning to control a brain–machine interface for reaching and grasping by primates," PLoS biology, vol. 1, no. 2, pp. e42, 2003.

[35] Wu, W., et al., "Neural decoding of cursor motion using a Kalman filter," Advances in neural information processing systems, no., pp. 133-140, 2003.

[36] Wu, W., et al., "Modeling and decoding motor cortical activity using a switching Kalman filter," IEEE transactions on biomedical engineering, vol. 51, no. 6, pp. 933-942, 2004.

[37] Li, Z., et al., "Unscented Kalman filter for brain-machine interfaces," PloS one, vol. 4, no. 7, pp. e6243, 2009.

[38] Li, S., J. Li, and Z. Li, "An improved unscented kalman filter based decoder for cortical brain-machine interfaces," Frontiers in neuroscience, vol. 10, no., pp. 587, 2016.

[39] Gilja, V., et al., "A high-performance neural prosthesis enabled by control algorithm design," Nature neuroscience, vol. 15, no. 12, pp. 1752-1757, 2012.

[40] Vaskov, A.K., et al., "Cortical decoding of individual finger group motions using ReFIT Kalman filter," Frontiers in neuroscience, vol. 12, no., pp. 751, 2018.

[41] Hochreiter, S. and J. Schmidhuber, "Long short-term memory," Neural computation, vol. 9, no. 8, pp. 1735-1780, 1997.

[42] Chung, J., et al., "Empirical evaluation of gated recurrent neural networks on sequence modeling," arXiv preprint arXiv:1412.3555, no., pp., 2014.

[43] Hosman, T., et al. BCI decoder performance comparison of an LSTM recurrent neural network and a Kalman filter in retrospective simulation. in 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER). 2019. IEEE.

[44] Ahmadi, N., T.G. Constandinou, and C.-S. Bouganis. Decoding hand kinematics from local field potentials using long short-term memory (LSTM) network. in 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER). 2019. IEEE.

[45] Glaser, J.I., et al., "Machine learning for neural decoding," Eneuro, vol. 7, no. 4, pp., 2020.

[46] Barrese, J.C., et al., "Failure mode analysis of silicon-based intracortical microelectrode arrays in non-human primates," Journal of neural engineering, vol. 10, no. 6, pp. 066014, 2013.

[47] Balasubramanian, K., et al., "Changes in cortical network connectivity with long-term brain-machine interface exposure after chronic amputation," Nature communications, vol. 8, no. 1, pp. 1-10, 2017.

[48] Campbell, A. and C. Wu, "Chronically implanted intracranial electrodes: tissue reaction and electrical changes," Micromachines, vol. 9, no. 9, pp. 430, 2018.

[49] Fernández, E., et al., "Acute human brain responses to intracortical microelectrode arrays: challenges and future prospects," Frontiers in neuroengineering, vol. 7, no., pp. 24, 2014.

[50] Saxena, T., et al., "The impact of chronic blood–brain barrier breach on intracortical electrode function," Biomaterials, vol. 34, no. 20, pp. 4703-4713, 2013.

[51] Salatino, J.W., et al., "Glial responses to implanted electrodes in the brain," Nature biomedical engineering, vol. 1, no. 11, pp. 862-877, 2017.

[52] Gaire, J., et al., "The role of inflammation on the functionality of intracortical microelectrodes," Journal of neural engineering, vol. 15, no. 6, pp. 066027, 2018.

[53] Barrese, J.C., J. Aceros, and J.P. Donoghue, "Scanning electron microscopy of chronically implanted intracortical microelectrode arrays in non-human primates," Journal of neural engineering, vol. 13, no. 2, pp. 026003, 2016.

[54] Wellman, S.M., et al., "A materials roadmap to functional neural interface design," Advanced functional materials, vol. 28, no. 12, pp. 1701269, 2018.

[55] Dunlap, C.F., et al., "Classifying intracortical brain-machine interface signal disruptions based on system performance and applicable compensatory strategies: a review," Frontiers in Neurorobotics, vol. 14, no., pp. 76, 2020.

[56] Flint, R.D., et al., "Long term, stable brain machine interface performance using local field potentials and multiunit spikes," Journal of neural engineering, vol. 10, no. 5, pp. 056005, 2013.

[57] Wang, D., et al., "Long-term decoding stability of local field potentials from silicon arrays in primate motor cortex during a 2D center out task," Journal of Neural Engineering, vol. 11, no. 3, pp. 036009, 2014.

[58] Stavisky, S.D., et al., "A high performing brain–machine interface driven by low-frequency local field potentials alone and together with spikes," Journal of neural engineering, vol. 12, no. 3, pp. 036009, 2015.

[59] Zhang, P., et al., "Using high-frequency local field potentials from multicortex to decode reaching and grasping movements in monkey," IEEE Transactions on Cognitive and Developmental Systems, vol. 11, no. 2, pp. 270-280, 2018.

[60] Ahmadi, N., T. Constandinou, and C.-S. Bouganis, "Robust and accurate decoding of hand kinematics from entire spiking activity using deep learning," Journal of Neural Engineering, no., pp., 2021.

[61] Zhang, M., et al., "Extracting wavelet based neural features from human intracortical recordings for neuroprosthetics applications," Bioelectronic medicine, vol. 4, no. 1, pp. 1-14, 2018.

[62] Schwemmer, M.A., et al., "Meeting brain–computer interface user performance expectations using a deep neural network decoding framework," Nature medicine, vol. 24, no. 11, pp. 1669-1676, 2018.

[63] Yu, B.M., et al., "Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity," Journal of neurophysiology, vol. 102, no. 1, pp. 614-635, 2009.

[64] Churchland, M.M., et al., "Neural population dynamics during reaching," Nature, vol. 487, no. 7405, pp. 51-56, 2012.

[65] Gallego, J.A., et al., "Neural manifolds for the control of movement," Neuron, vol. 94, no. 5, pp. 978-984, 2017.

[66] Gallego, J.A., et al., "Cortical population activity within a preserved neural manifold underlies multiple motor behaviors," Nature communications, vol. 9, no. 1, pp. 1-13, 2018.

[67] Gallego, J.A., et al., "Long-term stability of cortical population dynamics underlying consistent behavior," Nature neuroscience, vol. 23, no. 2, pp. 260-270, 2020.

[68] Aghagolzadeh, M. and W. Truccolo, "Inference and decoding of motor cortex low-dimensional dynamics via latent state-space models," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 24, no. 2, pp. 272-282, 2015.

[69] Pandarinath, C., et al., "Latent factors and dynamics in motor cortex and their application to brain–machine interfaces," Journal of Neuroscience, vol. 38, no. 44, pp. 9390-9401, 2018.

[70] Pandarinath, C., et al., "Inferring single-trial neural population dynamics using sequential auto-encoders," Nature methods, vol. 15, no. 10, pp. 805-815, 2018.

[71] Haghi, B., et al., "Deep Multi-State Dynamic Recurrent Neural Networks Operating on Wavelet Based Neural Features for Robust Brain Machine Interfaces," bioRxiv, no., pp. 710327, 2019.

[72] Shaikh, S., et al., "Sparse Ensemble Machine Learning to improve robustness of long-term decoding in iBMIs," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 28, no. 2, pp. 380-389, 2019.

[73] Li, W., et al., "Multi-source domain adaptation for decoder calibration of intracortical brain-machine interface," Journal of neural engineering, vol. 17, no. 6, pp. 066009, 2020.

[74] Zheng, M. and B. Yang, "A deep neural network with subdomain adaptation for motor imagery brain-computer interface," Medical Engineering & Physics, vol. 96, no., pp. 29-40, 2021.

[75] O’Doherty, J.E., et al., "Nonhuman primate reaching with multichannel sensorimotor cortex electrophysiology," Zenodo http://doi. org/10.5281/zenodo, vol. 583331, no., pp., 2017.

[76] Maynard, E.M., C.T. Nordhausen, and R.A. Normann, "The Utah intracortical electrode array: a recording structure for potential brain-computer interfaces," Electroencephalography and clinical neurophysiology, vol. 102, no. 3, pp. 228-239, 1997.

[77] Bullard, A.J., et al., "Design and testing of a 96-channel neural interface module for the Networked Neuroprosthesis system," Bioelectronic medicine, vol. 5, no. 1, pp. 1-14, 2019.

[78] Srivastava, N., et al., "Dropout: a simple way to prevent neural networks from overfitting," The journal of machine learning research, vol. 15, no. 1, pp. 1929-1958, 2014.

[79] Ganin, Y., et al., "Domain-adversarial training of neural networks," The journal of machine learning research, vol. 17, no. 1, pp. 2096-2030, 2016.

[80] Tzeng, E., et al., "Deep domain confusion: Maximizing for domain invariance," arXiv preprint arXiv:1412.3474, no., pp., 2014.