非正交多重接取技術(NOMA)是未來通訊系統的核心接取技術。本論文回顧過去五年有關於NOMA的相關文獻，並且審視了與NOMA有關的資源分配的論文，同時也對於機器學習演算法進行整理並且介紹，然後在這些基礎上進行研究。本論文著中在使用機器學習演算法中的深度Q網路(DQN)作為優化算法框架來將蜂巢式網路用戶的整體系統頻譜效益進行最佳化。在NOMA系統中，基站以同一頻譜達到多重接入的效果，利用傳輸功率差異同時服務多個使用者，而在此同時，使用相同頻譜的使用者們將會對彼此造成干擾。為了減少干擾造成的損害以及提高使用相同頻譜的效率，我們會利用DQN來進行用戶配對的最優化計算，藉由最優化計算來提升整體性統頻譜效率。DQN算法藉由將類神經網路作為一種評價工具，訓練神經網路給用戶在不同的子載波傳送資訊時的頻譜效率評價，並且藉由神經網路給予的評價來做用戶子載波分配關係的選擇。我們同時也對於神經網路配置進行了大量的嘗試和錯誤來得到最適合本論文研究的神經網路配置，並且使用該配置對三種非正交多重接取技術候選者多用戶疊加碼傳輸(MUST), 模式區分多址接入(PDMA), 以及稀疏碼多址接入(SCMA)進行模擬分析出不同非正交多重接取技術在不同性能指標下之優劣。

The non-orthogonal multiple access (NOMA) is the key to multiple access in future communication. We survey papers related to NOMA in the last five years, We survey papers about resource allocation related to NOMA and introduce machine learning. This thesis focus on resource allocation framework utilizing deep Q-network (DQN) to optimize spectrum efficiency in a single cell, in which deep Q-network is one of machine learning. In a NOMA system, base stations utilize the same frequency for multiplexing. Meanwhile, users in the same frequency would cause inter-user interference. To reduce interference and improve the spectrum efficiency, we use DQN to optimize user assignment, by optimizing computing to enhance spectrum efficiency. DQN utilize the neural network as an evaluation tool, train neural network to evaluate spectrum efficiency of user transmit information in different subcarrier, and chose subcarrier to the user by evaluation of neural network. We also design neural network's configuration through trial-and-errors and use this configuration of the simulation shows the performance between different NOMA which are multi-user superposition transmission (MUST), pattern division multiple access (PDMA), and sparse code multiple access (SCMA).

[1] 3GPP TS 38.812 V16.00.0, “Technical Specification Group Radio Access Net-work;Study on Non-Orthogonal Multiple Access (NOMA) for NR,” 2018.
[2] 3GPP TS 36.331 V15.30.0, “Evolved universal terrestrial radio access (E-UTRA); radio resource control (RRC),” 2018.
[3] ITU-R M.2083-0, “IMT Vision — Framework and Overall Objectives of the Future Development of IMT for 2020 and Beyond,” tech. rep., Sept. 2015
[4] Z. Dawy, W. Saad, A. Ghosh, J. G. Andrews, and E.Yaacoub, “Toward massive machine type cellular communications,” IEEE Wirel. Commun., vol. 24, no. 1, pp. 120–128, Feb.2017.
[5] M. S. Ali, H. Tabassum, and E. Hossain, “Dynamic User Clustering and Power Allo- cation for Uplink and Downlink Non-Orthogonal Multiple Access (NOMA) Systems,” IEEE Access, vol. 4, pp. 6325–6343, 2016.
[6] Q. Sun, S. Han, and Z. Pan, “On the Ergodic Capacity of MIMO NOMA Systems,” IEEE Wirel. Commun. Lett., vol. 4, no. 4, pp. 405–408, 2015.
[7] A. E. Mostafa, Y. Zhou, V. W.S. Wong, “Connectivity maximization for narrowband IoT systems with NOMA,” IEEE ICC, DOI: 10.1109/ICC.2017.7996362, Jul. 2017.
[8] J. Shang, L. Gui, H. Zhou, M. Dong, J. Chen, “A group-based M2M multiple ac-cess scheme in massive MIMO MU-SCMA cellular networks,” IEEE WCSP, DOI:10.1109/WCSP.2016.7752657, Oct. 2016.
[9] K. Kang, Z. Pan, J. Liu, S. Shimamoto, “A game theory based power control algorithm for future MTC NOMA networks,” IEEE CCNC, DOI: 10.1109/CCNC.2017.7983106, Jul. 2017.
[10] T. Lv, Y. Ma, J. Zeng, P. T. Mathiopoulos, “Millimeter-wave NOMA transmission in cellular M2M communications for internet of things,” IEEE Internet Things J., vol. 5, no. 3, pp. 1989 - 2000, DOI: 10.1109/JIOT.2018.2819645, Mar. 2018
[11] H. S. Jang, H. Park, and D. K. Sung, “A Non-Orthogonal Resource Allocation Scheme in Spatial Group Based Random Access for Cellular M2M Communications,” IEEE Trans. Veh. Technol., vol. 66, no. 5, pp. 1–1, 2017.
[12] X. Shao, C. Yang, D. Chen, N. Zhao, and F. Richard Yu, “Dynamic IoT Device Clus-tering and Energy Management with Hybrid NOMA Systems,” IEEE Trans. Ind. Infor-matics, vol. 14, no. 10, pp. 4622–4630, 2018.
[13] A. Shahini and N. Ansari, “NOMA Aided Narrowband IoT for Machine Type Commu-nications With User Clustering,” IEEE Internet Things J., vol. 6, no. 4, pp. 7183–7191, 2019
[14] Z. Yang, W. Xu, Y. Pan, C. Pan, M. Chen, “Energy efficient resource alloca-tion in machine-to-machine communications with multiple access and energy har-vesting for IoT,” IEEE Internet Things J., vol. 5, no. 1, pp. 229 - 245, DOI:10.1109/JIOT.2017.2778766, Dec. 2017.
[15] D. Zhai, R. Zhang, L. Cai, B. Li, Y. Jiang, “Energy-efficient user scheduling and power allocation for NOMA-based wireless networks with massive IoT devices,” In-ternet Things J., vol. 5, no. 3, pp. 1857 - 1868, DOI: 10.1109/JIOT.2018.2816597, Mar. 2018.
[16] H. Mao, M. Alizadeh, I. Menache, and S. Kandula, “Resource management with deep reinforcement learning,” HotNets 2016 - Proc. 15th ACM Work. Hot Top. Networks, pp. 50–56, 2016.
[17] Watkins, C. J., Dayan, P. (1992). Q-learning. Machine learning, 8 (3-4), 279-292.
[18] M. Liu, T. Song, and G. Gui, “Deep cognitive perspective: Resource allocation for noma-based heterogeneous IoT with imperfect SIC,” IEEE Internet Things J., vol. 6, no. 2, pp. 2885–2894, 2019.
[19] N. Ye, X. Li, H. Yu, A. Wang, W. Liu, and X. Hou, “Deep Learning Aided Grant-Free NOMA Toward Reliable Low-Latency Access in Tactile Internet of Things,” IEEE Trans. Ind. Informatics, vol. 15, no. 5, pp. 2995–3005, 2019.
[20] L. Lei, “Learning-Assisted Optimization for Energy-Efficient Scheduling in Deadline- Aware NOMA Systems,” IEEE Trans. Green Commun. Netw., vol. 3, no. 3, pp. 615–627, 2019.
[21] Y. Fu, W. Wen, Z. Zhao, T. Q. S. Quek, S. Jin, and F. C. Zheng, “Dynamic Power Con- trol for NOMA Transmissions in Wireless Caching Networks,” IEEE Wirel. Commun. Lett., vol. 8, no. 5, pp. 1485–1488, 2019.
[22] Y. Sun, Y. Wang, J. Jiao, S.Wu, and Q. Zhang, “Deep Learning-Based Long-Term Power Allocation Scheme for NOMA Downlink System in S-IoT,” IEEE Access, vol. 7, pp. 86288–86296, 2019.
[23] S. Tseng and Y. Chen, “Deep-Learning-Aided Cross-Layer Resource Allocation of OFDMA / NOMA Video Communication Systems,” IEEE Access, vol. 7, pp. 157730–157740, 2019.
[24] I. Jung, S. Member, H. Kim, and S. Member, “An Enhanced Tabu Search Based Re-ceiver for Full-Spreading NOMA Systems,” IEEE Access, vol. 7, pp. 159899–159917, 2019.
[25] L. Xiao, Y. Li, C. Dai, H. Dai, and H. V. Poor, “Reinforcement Learning-Based NOMA Power Allocation in the Presence of Smart Jamming,” IEEE Trans. Veh. Technol., vol. 67, no. 4, pp. 3377–3389, 2018.
[26] G. Gui, H. Huang, Y. Song, and H. Sari, “Deep Learning for an Effective Nonorthog- onal Multiple Access Scheme,” IEEE Trans. Veh. Technol., vol. 67, no. 9, pp. 8440–8450, 2018.
[27] G. N. Systems, Y. Zhang, Q. Guo, Z. Wang, J. Xi, and N. Wu, “Block Sparse Bayesian Learning Based Joint User Activity Detection and Channel Estimation for,” vol. 67, no. 10, pp. 9631–9640, 2018.
[28] J. Cui, S. Member, Z. Ding, and S. Member, “Unsupervised Machine Learning-Based User Clustering in Millimeter-Wave-NOMA Systems,” vol. 17, no. 11, pp. 7425–7440, 2018.
[29] Y. Du, “Block-Sparsity-Based Multiuser Detection for Uplink Grant-Free NOMA,” IEEE Trans. Wirel. Commun., vol. 17, no. 12, pp. 7894–7899, 2018.
[30] M. Usama, “Unsupervised Machine Learning for Networking: Techniques, Applica- tions and Research Challenges,” IEEE Access, vol. 7, pp. 65579–65615, 2019.
[31] K. Wang, C. Gou, Y. Duan, Y. Lin, X. Zheng, and F. Y. Wang, “Generative adversarial networks: Introduction and outlook,” IEEE/CAA J. Autom. Sin., vol. 4, no. 4, pp. 588–598, 2017.
[32] S. Shah and M. Singh, “Comparison of a time efficient modified k-mean algorithm with K-mean and K-medoid algorithm,” Proc. - Int. Conf. Commun. Syst. Netw. Technol. CSNT 2012, pp. 435–437, 2012.
[33] J. A. Y. Liebowitz, “Expert systems: A short introduction,” Volume 50, Issues 5–6, March–April 1995, Pages 601-607.
[34] J.Ba and R.Caruana, “Do deep networks really need to be deep?,” Adv. Neural Inf. Process. Syst., pp. 2654–2662, 2014.
[35] A. Montoyo, R. Mu˜noz, and E. M´etais, Lecture Notes in Computer Science: Preface, vol. 3513. 2005.
[36] V. Sze, Y. H. Chen, T. J. Yang, and J. S.Emer, “Efficient Processing of Deep Neural Networks: A Tutorial and Survey,” Proc. IEEE, vol. 105, no. 12, pp. 2295–2329, 2017.
[37] R. Atienza, Advanced Deep Learning with Keras. 2018.
[38] [1] L.Liang, H.Ye, andG. Y.Li, “Spectrum Sharing in Vehicular Networks Based on Multi-Agent Reinforcement Learning,” IEEE J. Sel. Areas Commun., vol. 37, no. 10, pp. 2282–2292, 2019.
[39] Sutton, Richard S. and Barto, Andrew G., Reinforcement Learning: An Introduction. Second : The MIT Press, 2018.
[40] R. Ruby, S. Zhong, H. Yang, and K .Wu, “Enhanced Uplink Resource Allocation in Non-Orthogonal Multiple Access Systems,” IEEE Trans. Wirel. Commun., vol. 17, no. 3, pp. 1432–1444, 2018.
[41] 3GPP, “3rd Generation Partnership Project; Technical Specification Group Radio Ac- cess Network; Evolved Universal Terrestrial Radio Access (E-UTRA) and Evolved Universal Terrestrial Radio Access Network (E-UTRAN); Radio Frequency (RF) sys- tem scenarios (Release 12),” 3GPP TR 36.942 V14.0.0, Mar. 2017.
[42] M. A. Sedaghat and R. R. Muller, “On User Pairing in Uplink NOMA,” IEEE Trans. Wirel. Commun., vol. 17, no. 5, pp. 3474–3486, 2018.
[43] M. Zeng, N. -P. Nguyen, O. A.Dobre, Z. Ding, and H. V. Poor, “Spectral- and Energy- Efficient Resource Allocation for Multi-Carrier Uplink NOMA Systems,” IEEE Trans. Veh. Technol., vol. 68, no. 9, pp. 9293–9296, 2019.
[34] J.Ba and R.Caruana, “Do deep networks really need to be deep?,” Adv. Neural Inf. Process. Syst., pp. 2654–2662, 2014.
[35] A. Montoyo, R. Mu˜noz, and E. M´etais, Lecture Notes in Computer Science: Preface, vol. 3513. 2005.
[36] V. Sze, Y. H. Chen, T. J. Yang, and J. S.Emer, “Efficient Processing of Deep Neural Networks: A Tutorial and Survey,” Proc. IEEE, vol. 105, no. 12, pp. 2295–2329, 2017. [37] R. Atienza, Advanced Deep Learning with Keras. 2018.
[38] [1] L.Liang, H.Ye, andG. Y.Li, “Spectrum Sharing in Vehicular Networks Based on Multi-Agent Reinforcement Learning,” IEEE J. Sel. Areas Commun., vol. 37, no. 10, pp. 2282–2292, 2019.
[39] Sutton, Richard S. and Barto, Andrew G., Reinforcement Learning: An Introduction. Second : The MIT Press, 2018.
[40] R. Ruby, S. Zhong, H. Yang, and K .Wu, “Enhanced Uplink Resource Allocation in Non-Orthogonal Multiple Access Systems,” IEEE Trans. Wirel. Commun., vol. 17, no. 3, pp. 1432–1444, 2018.
[41] 3GPP, “3rd Generation Partnership Project; Technical Specification Group Radio Ac- cess Network; Evolved Universal Terrestrial Radio Access (E-UTRA) and Evolved Universal Terrestrial Radio Access Network (E-UTRAN); Radio Frequency (RF) sys-tem scenarios (Release 12),” 3GPP TR 36.942 V14.0.0, Mar. 2017.
[42] M. A. Sedaghat and R. R. Muller, “On User Pairing in Uplink NOMA,” IEEE Trans. Wirel. Commun., vol. 17, no. 5, pp. 3474–3486, 2018.
[43] M. Zeng, N. -P. Nguyen, O. A.Dobre, Z. Ding, and H. V. Poor, “Spectral- and Energy- Efficient Resource Allocation for Multi-Carrier Uplink NOMA Systems,” IEEE Trans. Veh. Technol., vol. 68, no. 9, pp. 9293–9296, 2019.
[53] J. Liu, M. Gong, Q. Miao, X. Wang, and H. Li, “Structure Learning for Deep Neural Networks Based on Multiobjective Optimization,” IEEE Trans. Neural Networks Learn. Syst., vol. 29, no. 6, pp. 2450–2463, 2018.
[54] L. Wang, H. Ye, L. Liang, and G. Y. Li, “Learn to Compress CSI and Allocate Re-sources in Vehicular Networks,” IEEE Trans. Commun., vol. 68, no. 6, pp. 1–1, 2020.
[55] Y. Xu, J. Yu, W. C. Headley, and R. M. Buehrer, “Deep Reinforcement Learning for Dynamic Spectrum Access in Wireless Networks,” Proc. - IEEE Mil. Commun. Conf. MILCOM, vol. 2019-October, no. 2, pp. 207–212, 2019.
[56] Y. Li, W. Zhang, C. X. Wang, J. Sun, and Y. Liu, “Deep Reinforcement Learning for Dynamic Spectrum Sensing and Aggregation in Multi-Channel Wireless Networks,” IEEE Trans. Cogn. Commun. Netw., vol. 6, no. 2, pp. 464–475, 2020.
[57] T. Yang, J. Li, H. Feng, N. Cheng, and W. Guan, “A Novel Transmission Scheduling Based on Deep Reinforcement Learning in Software-Defined Maritime Communica-tion Networks,” IEEE Trans. Cogn. Commun. Netw., vol. 5, no. 4, pp. 1155–1166, 2019.
[58] J. Tang et al., “Energy Minimization in D2D-Assisted Cache-Enabled Internet of Things: A Deep Reinforcement Learning Approach,” IEEE Trans. Ind. Informatics, vol. 16, no. 8, pp. 5412–5423, 2020.
[59] W. Lee, O. Jo, and M. Kim, “Intelligent Resource Allocation in Wireless Communica-tions Systems,” IEEE Commun. Mag., vol. 58, no. 1, pp. 100–105, 2020.
[60] S. Chen, B. Ren, Q. Gao, S. Kang, S. Sun, and K. Niu, “Pattern division multiple access- a novel non-orthogonal multiple access for 5G radio networks,” IEEE Transactions on Vehicular Technology, DOI: 10.1109/TVT.2016.2596438, Jul. 2016.
[61] J. Zeng, B. Li, X. Su, L. Rong, and R. Xing, “Pattern division multiple access (PDMA) for cellular future radio access,” International Conference on Wireless Communications and Signal Processing , DOI: 10.1109/WCSP.2015.7341229,Dec. 2015.
[53] J. Liu, M. Gong, Q. Miao, X. Wang, and H. Li, “Structure Learning for Deep Neural Networks Based on Multiobjective Optimization,” IEEE Trans. Neural Networks Learn. Syst., vol. 29, no. 6, pp. 2450–2463, 2018.
[54] L. Wang, H. Ye, L. Liang, and G. Y. Li, “Learn to Compress CSI and Allocate Re-sources in Vehicular Networks,” IEEE Trans. Commun., vol. 68, no. 6, pp. 1–1, 2020.
[55] Y. Xu, J. Yu, W. C. Headley, and R. M. Buehrer, “Deep Reinforcement Learning for Dynamic Spectrum Access in Wireless Networks,” Proc. - IEEE Mil. Commun. Conf. MILCOM, vol. 2019-October, no. 2, pp. 207–212, 2019.
[56] Y. Li, W. Zhang, C. X. Wang, J. Sun, and Y. Liu, “Deep Reinforcement Learning for Dynamic Spectrum Sensing and Aggregation in Multi-Channel Wireless Networks,”IEEE Trans. Cogn. Commun. Netw., vol. 6, no. 2, pp. 464–475, 2020.
[57] T. Yang, J. Li, H. Feng, N. Cheng, and W. Guan, “A Novel Transmission Scheduling Based on Deep Reinforcement Learning in Software-Defined Maritime Communica-tion Networks,” IEEE Trans. Cogn. Commun. Netw., vol. 5, no. 4, pp. 1155–1166, 2019.
[58] J. Tang et al., “Energy Minimization in D2D-Assisted Cache-Enabled Internet of Things: A Deep Reinforcement Learning Approach,” IEEE Trans. Ind. Informatics, vol. 16, no. 8, pp. 5412–5423, 2020.
[59] W. Lee, O. Jo, and M. Kim, “Intelligent Resource Allocation in Wireless Communica- tions Systems,” IEEE Commun. Mag., vol. 58, no. 1, pp. 100–105, 2020.
[60] S. Chen, B. Ren, Q. Gao, S. Kang, S. Sun, and K. Niu, “Pattern division multiple access- a novel non-orthogonal multiple access for 5G radio networks,” IEEE Transactions on Vehicular Technology, DOI: 10.1109/TVT.2016.2596438, Jul. 2016.
[61] J. Zeng, B. Li, X. Su, L. Rong, and R. Xing, “Pattern division multiple access (PDMA) for cellular future radio access,” International Conference on Wireless Communications and Signal Processing , DOI: 10.1109/WCSP.2015.7341229,Dec. 2015.
[62] W.Tang, S.Kang, B.Ren, X.Yue, and X.Zhang, “Uplink pattern division multiple access
in 5G systems ,” Institution of Engineering and Technology, DOI: 10.1049/ietcom.
2017.1143, May 2018.
[63] 3GPP Release 13, “Study on downlink multiuser superposition transmission (MUST)
for LTE,” TR36.859, May 2016.
[64] J. V. C. Evangelista, Z. Sattar, G. Kaddoum, and A. Chaaban, “Fairness and Sum-Rate
Maximization via Joint Subcarrier and Power Allocation in Uplink SCMA Transmission,”
IEEE Trans. Wirel. Commun., vol. 18, no. 12, pp. 5855–5867, 2019.
[65] H. Nikopour and H. Baligh, “Sparse code multiple access,” IEEE 24th Annual International
Symposium on Personal, Indoor, and Mobile Radio Communications, DOI:
10.1109/PIMRC.2013.6666156, Nov. 2013.
[66] M. Taherzadeh, H. Nikopour, A. Bayesteh , and H. Baligh, “SCMA codebook design,”
IEEE Vehicular Technology Conference, DOI: 10.1109/VTCFall.2014.6966170, Dec.
2014.
[67] Y. Wu, S. Zhang, and Y. Chen, “Iterative multiuser receiver in sparse code multiple
access systems,” IEEE International Conference on Communications, DOI:
10.1109/ICC.2015.7248770, Jun. 2015.
[68] K. Xiao, B. Xiao, S. Zhang, Z. Chen, and B. Xiao, “Simplified multiuser
detection for SCMA with sum-product algorithm,” IEEE WCSP, DOI:
10.1109/WCSP.2015.7341328, Dec. 2015.
[69] M. Moltafet and N. M. Yamchi, “Comparison Study Between PD-NOMA and SCMA,”
IEEE Trans. Veh. Technol., vol. 67, no. 2, pp. 1830–1834, 2018.
[70] CVX Research - CVX: Matlab software for disciplined convex program- ming, version
2.1. [Online]. Available: http://cvxr.com/cvx. Accessed on: Oct. 4, 2015.