近年來，有關無監督學習的研究越來越多。與有監督的學習相反，這是更常見的情況，我們無法訪問標籤信息，並且由於高註釋成本而無法負擔。 此外，訓練無監督嵌入模型對於下游任務具有很多優勢，例如預訓練權重和微調問題。針對視覺上有意義的嵌入應滿足靠近相似實例和分離異類的性質，我們分別提出了兩種基於鄰居信息的無監督嵌入學習方法。實驗表明，我們提出的基於鄰域信息的無監督嵌入學習可以在不同的基準數據集上實現最佳的性能。

Although the convenience of Deep learning in many areas, lacking annotation is still a significant drawback for training the model. However, unsupervised embedding learning can be regarded as a helper for the pre-training task even we do not have the information of labels. A visually meaningful embedding must satisfy the properties, closing the similar instances and separating those dissimilar. We proposed two approaches based on neighbor information in this paper, super-AND, and NB-DSCV. Considering neighbor information is critical in the unsupervised embedding learning task. Therefore, we leverage the neighbor's information to update the embedding by exploitation and exploration. We get a considerable improvement on unsupervised embedding learning tasks on our experiment result. In future work, we want to implement these approaches from visual data to the natural language process problem.

[1]Pierre Baldi. Autoencoders, unsupervised learning, and deep architectures. InProc. ofthe ICML workshop on unsupervised and transfer learning, pages 37–49, 2012.
[2]Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculumlearning. InProceedings of the 26th annual international conference on machine learn-ing, pages 41–48, 2009.
[3]Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculumlearning.
[4]Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weightuncertainty in neural networks, 2015.
[5]Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. InProc.oftheInternationalConferenceonMachineLearning-Volume70, pages 517–526.JMLR. org, 2017.
[6]Shaosheng Cao, Wei Lu, and Qiongkai Xu. Deep neural networks for learning graphrepresentations. InThirtieth AAAI conference on artificial intelligence, 2016.
[7]Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep cluster-ing for unsupervised learning of visual features. InProc. of the European Conferenceon Computer Vision, pages 132–149, 2018.
[8]Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep cluster-ing for unsupervised learning of visual features.Lecture Notes in Computer Science,page 139–156, 2018.
[9]Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simpleframework for contrastive learning of visual representations, 2020.
[10]Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networksin unsupervised feature learning. InProc. of the international conference on artificialintelligence and statistics, pages 215–223, 2011.
[11]Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representa-tion learning by context prediction. InProc. of the IEEE International Conference onComputer Vision, pages 1422–1430, 2015.33
doi:10.6844/NCKU202000764
[12]Qi Dong, Shaogang Gong, and Xiatian Zhu. Multi-task curriculum transfer deep learn-ing of clothing attributes.2017 IEEE Winter Conference on Applications of ComputerVision (WACV), Mar 2017.
[13]Timothy Dozat. Incorporating nesterov momentum into adam. 2016.
[14]Ian Goodfellow. Nips 2016 tutorial: Generative adversarial networks.arXiv preprintarXiv:1701.00160, 2016.
[15]Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. InProc. of the Advances in neural information processing systems, pages 2672–2680,2014.
[16]Gongde Guo, Hui Wang, David Bell, and Yaxin Bi. Knn model-based approach inclassification. 08 2004.
[17]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning forimage recognition. InProc. of the IEEE conference on computer vision and patternrecognition, pages 770–778, 2016.
[18]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning forimage recognition.2016IEEEConferenceonComputerVisionandPatternRecognition(CVPR), Jun 2016.
[19]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neuralnetwork.arXiv preprint arXiv:1503.02531, 2015.
[20]Jiabo Huang, Qi Dong, Shaogang Gong, and Xiatian Zhu. Unsupervised deep learningby neighbourhood discovery. InProc. of the International Conference on MachineLearning, pages 2849–2858, 2019.
[21]Xu Ji, João F. Henriques, and Andrea Vedaldi. Invariant information clustering forunsupervised image classification and segmentation.arXiv preprint arXiv:1807.06653,2018.
[22]T. Kanungo, D. M. Mount, N. S. Netanyahu, C. D. Piatko, R. Silverman, and A. Y.Wu. An efficient k-means clustering algorithm: analysis and implementation.IEEETransactions on Pattern Analysis and Machine Intelligence, 24(7):881–892, 2002.
[23]Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Noveldataset for fine-grained image categorization. InProc. of the First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and PatternRecognition, Colorado Springs, CO, June 2011.34
doi:10.6844/NCKU202000764
[24]Alex Krizhevsky. Learning multiple layers of features from tiny images.University ofToronto, 05 2012.
[25]Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technicalreport, Citeseer, 2009.
[26]Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification withdeep convolutional neural networks.Neural Information Processing Systems, 25, 012012.
[27]Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification withdeep convolutional neural networks. InProc. of the Advances in neural informationprocessing systems, pages 1097–1105, 2012.
[28]Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning.arXivpreprint arXiv:1610.02242, 2016.
[29]Steve Lawrence, C Lee Giles, Ah Chung Tsoi, and Andrew D Back. Face recogni-tion: A convolutional neural-network approach.IEEEtransactionsonneuralnetworks,8(1):98–113, 1997.
[30]Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutionalnetworks for semi-supervised learning. InThirty-Second AAAI Conferenceon ArtificialIntelligence, 2018.
[31]James M Lucas and Michael S Saccucci. Exponentially weighted moving average con-trol schemes: properties and enhancements.Technometrics, 32(1):1–12, 1990.
[32]Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew YNg. Reading digits in natural images with unsupervised feature learning. 2011.
[33]Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y.Ng. Reading digits in natural images with unsupervised feature learning. InNIPSWorkshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
[34]Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learn-ing to count. InProc. of the IEEE International Conference on Computer Vision, pages5898–5906, 2017.
[35]Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learningwith deep convolutional generative adversarial networks. InProc. of the InternationalConference on Machine Learning, 2016.
[36]Herbert E. Robbins. A stochastic approximation method.Annals of MathematicalStatistics, 22:400–407, 2007.35
doi:10.6844/NCKU202000764
[37]Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embed-ding for face recognition and clustering. InProc. of the IEEE conference on computervision and pattern recognition, pages 815–823, 2015.
[38]Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and JoellePineau. Building end-to-end dialogue systems using generative hierarchical neural net-work models. InThirtieth AAAI Conference on Artificial Intelligence, 2016.
[39]Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. InAd-vances in Neural Information Processing Systems, pages 1195–1204, 2017.
[40]Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-AntoineManzagol. Stacked denoising autoencoders: Learning useful representations in a deepnetwork with a local denoising criterion.J. Mach. Learn. Res., 11:3371–3408, Decem-ber 2010.
[41]C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical report, 2011.
[42]Jason Wang and Luis Perez. The effectiveness of data augmentation in image classi-fication using deep learning.Convolutional Neural Networks Vis. Recognit, page 11,2017.
[43]P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona.Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Instituteof Technology, 2010.
[44]Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learn-ing via non-parametric instance discrimination. InProc. of the IEEE Conference onComputer Vision and Pattern Recognition, pages 3733–3742, 2018.
[45]Tong Xiao, Shuang Li, Bochao Wang, Liang Lin, and Xiaogang Wang. Joint detec-tion and identification feature learning for person search.2017 IEEE Conference onComputer Vision and Pattern Recognition (CVPR), Jul 2017.
[46]Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clus-tering analysis. InProc. of the International conference on machine learning, pages478–487, 2016.
[47]Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graphneural networks?, 2018.36
doi:10.6844/NCKU202000764
[48]Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep rep-resentations and image clusters. InProc. of the IEEE Conference on Computer Visionand Pattern Recognition, pages 5147–5156, 2016.
[49]Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embeddinglearning via invariant and spreading instance feature. InProc. of the IEEE Conferenceon Computer Vision and Pattern Recognition, pages 6210–6219, 2019.
[50]Mang Ye, Xu Zhang, Pong C. Yuen, and Shih-Fu Chang. Unsupervised embeddinglearning via invariant and spreading instance feature.2019 IEEE/CVF Conference onComputer Vision and Pattern Recognition (CVPR), Jun 2019.
[51]Dejiao Zhang, Yifan Sun, Brian Eriksson, and Laura Balzano. Deep unsupervised clus-tering using mixture of autoencoders.arXiv preprint arXiv:1712.07788, 2017.
[52]Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsu-pervised learning by cross-channel prediction. InProc. of the IEEE Conference onComputer Vision and Pattern Recognition, pages 1058–1067, 2017.
[53]Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang,Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods andapplications, 2018