細粒度視覺分類是一項相當有難度的圖像分類任務，常見的特性有類別之間極為相似、類別之內有所差異、環境複雜等等，這些特性使得傳統的卷積神經網路無法順利地收斂，通常需要使用額外的標註或者運算方法才能完成此任務，造成人力成本居高不下、運算過程繁雜的問題。本研究使用Vision Transformer作為骨幹網路，提出了DAAT (Demonstrable self Attention Among Transformer)模型，透過改變Transformer架構讓其中的self attention能夠更加明確，並聚焦在關鍵的圖像位置之上。同時提出三個能夠加強效果的模組，分別是在將圖像映射成Token時使用較不容易損失局部資訊的的重疊分割方法，為了能讓Classification Token更好地學習分類資訊而延後加入的時間之運算方法，以及使用Contrastive Loss讓Token Embedding在Embedding Space內的分布更有利於細粒度視覺分類。整個過程不需要使用額外的標註資訊，能夠在只使用類別標註的情況下，順利地完成細粒度視覺分類任務。同時DAAT模型也不需要使用額外的運算方法和多個步驟的來進行訓練，可以進行端對端的訓練。在實驗部分，本研究使用了四個在細粒度視覺分類中相當熱門的數據集來進行圖像分類任務的訓練和驗證，並從結果證明了DAAT模型之有效性。

The fine-grained visual classification (FGVC) aims at recognizing objects from subcategories is very challenging because discriminative clues are usually fragmented. Most previous works tackle this problem by explicitly selecting the discriminative parts or integrating the attention mechanism via CNN-based approaches. Recently, Vision Transformer(ViT) shows its strong performance on general image recognition tasks. The self-attention links the information from every patch token to the classification token, making it perfectly suitable for FGVC. In this paper, we analyze the problems of using the CNN-based model on fine-grained visual classification and propose a new transformer architecture DAAT (Demonstrable self Attention Among Transformer)　with the select attention module, which aggregates the attention score of the transformer for guiding the network to accurately select the important tokens among image patches. Furthermore, overlap module, later CLS module, and extra loss module are applied to further improve the generalization ability of the network. To demonstrate the advantage of DAAT, we conduct evaluations on the four popular fine-grained benchmarks:CUB-200-2011, Stanford Cars, Stanford Dogs, NABirds where DAAT achieves state-of-the-art performance.

[1] Microsoft, “Flowerreco,” https://www.microsoft.com/en-us/research/ project/flowerreco-cn/.
[2] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in Computer Vision – ECCV 2014, pp. 818–833, Springer International Publishing, 2014.
[3] X. Zeng, W. Ouyang, J. Yan, H. Li, T. Xiao, K. Wang, Y. Liu, Y. Zhou, B. Yang, Z. Wang, H. Zhou, and X. Wang, “Crafting gbd-net for object detection,” vol. 40, pp. 2109–2123, 2018.
[4] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7132–7141, 2018.
[5] D. L. Ruderman and W. Bialek, “Statistics of natural images: Scaling in the woods,” vol. 73, pp. 814–817, 1994.
[6] B. A. Olshausen and D. J. Field, “Natural image statistics and efficient coding,” vol. 7, pp. 333–339, 1996.
[7] R. Liu, J. Lehman, P. Molino, F. P. Such, E. Frank, A. Sergeev, and J. Yosinski, “An intriguing failing of convolutional neural networks and the coordconv solution,” in Proceedings of the 32nd International Conference on Neural Information Processing Systems, p. 9628–9639, Curran Associates Inc., 2018.
[8] M. A. Islam, S. Jia, and N. D. B. Bruce, “How much position information do convo- lutional neural networks encode?,” 2020.
[9] A. Azulay and Y. Weiss, “Why do deep convolutional networks generalize so poorly to small image transformations?,” 2019.
[10] U. Asif, M. Bennamoun, and F. A. Sohel, “A multi-modal, discriminative and spatially invariant cnn for rgb-d object labeling,” vol. 40, pp. 2051–2065, 2018.
[11] CVPR, “inaturalist competition 2018,” https://sites.google.com/view/fgvc5/ competitions/inaturalist.
[12] L. Zhu and Y. Yang, “Inflated episodic memory with region self-attention for long- tailed visual recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[13] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” 2021.
[14] W. S. McCulloch and W. Pitts, “A logical calculus of the ideas immanent in nervous activity,” vol. 5, pp. 115–133, 1943.
[15] F. Rosenblatt, “The perceptron: a probabilistic model for information storage and or- ganization in the brain.,” vol. 65 6, pp. 386–408, 1958.
[16] P. Werbos and P. John, “Beyond regression : new tools for prediction and analysis in the behavioral sciences,” 1974.
[17] K. Fukushima, “Neocognitron: A self-organizing neural network model for a mech- anism of pattern recognition unaffected by shift in position,” vol. 36, pp. 193–202, 1980.
[18] M. I. Jordan, “Serial order: a parallel distributed processing approach. technical report, june 1985-march 1986,” 1986.
[19] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” vol. 9, pp. 1735–1780, 1997.
[20] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep con- volutional neural networks,” in Advances in Neural Information Processing Systems, vol. 25, Curran Associates, Inc., 2012.
[21] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 2015.
[22] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Van- houcke, and A. Rabinovich, “Going deeper with convolutions,” in 2015 IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 1–9, 2015.
[23] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” 2015.
[24] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” 2015.
[25] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” 2016.
[26] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2015.
[27] M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” 2019.
[28] A. Canziani, A. Paszke, and E. Culurciello, “An analysis of deep neural network mod- els for practical applications,” 2017.
[29] A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft, “Simple online and realtime tracking,” IEEE, 2016.
[30] N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in 2017 IEEE International Conference on Image Processing (ICIP), pp. 3645–3649, 2017.
[31] W. Li, Y. Xiong, S. Yang, S. Deng, and W. Xia, “Smot: Single-shot multi object track- ing,” 2020.
[32] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” p. 21–37, Springer International Publishing, 2016.
[33] G. Ning, Z. Zhang, C. Huang, Z. He, X. Ren, and H. Wang, “Spatially supervised recurrent convolutional neural networks for visual object tracking,” 2016.
[34] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” 2016.
[35] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang, “Bottom-up and top-down attention for image captioning and visual question answer- ing,” in CVPR, 2018.
[36] O. Abdel-Hamid, A.-r. Mohamed, H. Jiang, L. Deng, G. Penn, and D. Yu, “Convolu- tional neural networks for speech recognition,” vol. 22, pp. 1533–1545, 2014.
[37] Y. LeCun and Y. Bengio, “Convolutional networks for images, speech, and time se- ries,” in The Handbook of Brain Theory and Neural Networks, p. 255–258, 1998.
[38] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalch- brenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw au- dio,” 2016.
[39] K. Choi, G. Fazekas, M. Sandler, and K. Cho, “Convolutional recurrent neural net- works for music classification,” 2016.
[40] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. Wilson, “Cnn architectures for large-scale audio classification,” 2017.
[41] J. Salamon and J. P. Bello, “Deep convolutional neural networks and data augmenta- tion for environmental sound classification,” vol. 24, p. 279–283, Institute of Electrical and Electronics Engineers (IEEE), 2017.
[42] S. Lai, L. Xu, K. Liu, and J. Zhao, “Recurrent convolutional neural networks for text classification,” in Proceedings of the Twenty-Ninth AAAI Conference on Artificial In-telligence, p. 2267–2273, 2015.
[43] C. dos Santos and M. Gatti, “Deep convolutional neural networks for sentiment analy- sis of short texts,” in Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pp. 69–78, Dublin City University and Association for Computational Linguistics, 2014.
[44] K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio, “On the properties of neural machine translation: Encoder-decoder approaches,” 2014.
[45] M. Schlichtkrull, T. N. Kipf, P. Bloem, R. van den Berg, I. Titov, and M. Welling, “Modeling relational data with graph convolutional networks,” 2017.
[46] D. Duvenaud, D. Maclaurin, J. Aguilera-Iparraguirre, R. Gómez-Bombarelli, T. Hirzel, A. Aspuru-Guzik, and R. P. Adams, “Convolutional networks on graphs for learning molecular fingerprints,” 2015.
[47] J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun, “Spectral networks and locally con- nected networks on graphs,” 2014.
[48] T. Denk, “Text classification with separable convolutional neural networks,” 2018.
[49] T. Danel, P. Spurek, J. Tabor, M. Śmieja, Łukasz Struski, A. Słowik, and Łukasz Maziarka, “Spatial graph convolutional networks,” 2020.
[50] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” vol. 86, pp. 2278–2324, 1998.
[51] E. Simoncelli, W. Freeman, E. Adelson, and D. Heeger, “Shiftable multiscale trans- forms,” vol. 38, pp. 587–607, 1992.
[52] R. Zhang, “Making convolutional networks shift-invariant again,” in ICML, 2019.
[53] B. Irving, S. Suresh, B. Moshe, K. Peter, and F. Jozsef, “Subordinate-level object classification reexamined,” in Psychological research, vol. 62, pp. 131–153, 1999.
[54] A. Torralba, R. Fergus, and W. T. Freeman, “80 million tiny images: A large data set for nonparametric object and scene recognition,” vol. 30, pp. 1958–1970, 2008.
[55] J. Jeon, V. Lavrenko, and R. Manmatha, “Automatic image annotation and retrieval using cross-media relevance models,” in Proceedings of the 26th Annual Interna- tional ACM SIGIR Conference on Research and Development in Informaion Retrieval, p. 119–126, Association for Computing Machinery, 2003.
[56] H. Zhang, T. Xu, M. Elhoseiny, X. Huang, S. Zhang, A. Elgammal, and D. Metaxas, “Spda-cnn: Unifying semantic part detection and abstraction for fine-grained recogni- tion,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1143–1152, 2016.
[57] C. Huang and Z. He, “Task-driven progressive part localization for fine-grained recog- nition,” in 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–9, 2016.
[58] X. Shu, J. Tang, G.-J. Qi, Z. Li, Y.-G. Jiang, and S. Yan, “Image classification with tailored fine-grained dictionaries,” vol. 28, pp. 454–467, 2018.
[59] Y. Chai, V. Lempitsky, and A. Zisserman, “Symbiotic segmentation and part local- ization for fine-grained categorization,” in 2013 IEEE International Conference on Computer Vision, pp. 321–328, 2013.
[60] N. Zhang, R. Farrell, F. Iandola, and T. Darrell, “Deformable part descriptors for fine- grained recognition and attribute prediction,” in 2013 IEEE International Conference on Computer Vision, pp. 729–736, 2013.
[61] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan, “Object detection with discriminatively trained part-based models,” vol. 32, pp. 1627–1645, 2010.
[62] N. Zhang, J. Donahue, R. Girshick, and T. Darrell, “Part-based r-cnns for fine-grained category detection,” in Computer Vision – ECCV 2014, pp. 834–849, Springer Inter- national Publishing, 2014.
[63] J. R. R. Uijlings, K. E. A. van de Sande, T. Gevers, and A. W. M. Smeulders, “Selective search for object recognition,” vol. 104, p. 154–171, 2013.
[64] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587, 2014.
[65] T. Xiao, Y. Xu, K. Yang, J. Zhang, Y. Peng, and Z. Zhang, “The application of two- level attention models in deep convolutional neural network for fine-grained image classification,” 2014.
[66] X. Liu, T. Xia, J. Wang, Y. Yang, F. Zhou, and Y. Lin, “Fully convolutional attention networks for fine-grained recognition,” 2017.
[67] Z. Ge, C. McCool, C. Sanderson, and P. Corke, “Subset feature learning for fine- grained category classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2015.
[68] Z. Ge, A. Bewley, C. McCool, B. Upcroft, P. Corke, and C. Sanderson, “Fine-grained classification via mixture of deep convolutional neural networks,” 2015.
[69] Z. Wang, X. Wang, and G. Wang, “Learning fine-grained features via a cnn tree for large-scale classification,” 2017.
[70] D. Wang, Z. Shen, J. Shao, W. Zhang, X. Xue, and Z. Zhang, “Multiple granularity descriptors for fine-grained categorization,” in 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2399–2406, 2015.
[71] T.-Y. Lin, A. RoyChowdhury, and S. Maji, “Bilinear cnns for fine-grained visual recognition,” 2017.
[72] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, Curran Associates, Inc., 2017.
[73] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidi- rectional transformers for language understanding,” 2019.
[74] Q. Wang, B. Li, T. Xiao, J. Zhu, C. Li, D. F. Wong, and L. S. Chao, “Learning deep transformer models for machine translation,” 2019.
[75] J. He, J.-N. Chen, S. Liu, A. Kortylewski, C. Yang, Y. Bai, C. Wang, and A. Yuille, “Transfg: A transformer architecture for fine-grained recognition,” 2021.
[76] H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, and H. Jégou, “Going deeper with image transformers,” 2021.
[77] S. Abnar and W. Zuidema, “Quantifying attention flow in transformers,” 2020.
[78] S. Serrano and N. A. Smith, “Is attention interpretable?,” 2019.
[79] Y. Wang, V. I. Morariu, and L. S. Davis, “Learning a discriminative filter bank within a cnn for fine-grained recognition,” 2018.
[80] D. Zhou, B. Kang, X. Jin, L. Yang, X. Lian, Z. Jiang, Q. Hou, and J. Feng, “Deepvit: Towards deeper vision transformer,” 2021.
[81] Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A discriminative feature learning approach for deep face recognition,” in Computer Vision – ECCV 2016, pp. 499–515, Springer International Publishing, 2016.
[82] J. Deng, J. Guo, X. Niannan, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in CVPR, 2019.
[83] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an in- variant mapping,” in 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), vol. 2, pp. 1735–1742, 2006.
[84] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, “The caltech-ucsd birds-200-2011 dataset,” 2011. http://www.vision.caltech.edu/visipedia/ CUB-200-2011.html.
[85] J. Krause, M. Stark, J. Deng, and L. Fei-Fei, “3d object representations for fine- grained categorization,” in 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), 2013. http://ai.stanford.edu/~jkrause/cars/ car_dataset.html.
[86] A. Khosla, N. Jayadevaprakash, B. Yao, and L. Fei-Fei, “Novel dataset for fine- grained image categorization,” in First Workshop on Fine-Grained Visual Categoriza- tion, IEEE Conference on Computer Vision and Pattern Recognition, 2011. http://vision.stanford.edu/aditya86/ImageNetDogs/.
[87] G. Van Horn, S. Branson, R. Farrell, S. Haber, J. Barry, P. Ipeirotis, P. Perona, and S. Belongie, “Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection,” in 2015 IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2015. https://dl. allaboutbirds.org/nabirds.
[88] R. Wightman, “Pytorch image models,” GitHub, 2019. https://github.com/ rwightman/pytorch-image-models.
[89] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers and distillation through attention,” in Proceedings of the 38th International Conference on Machine Learning, vol. 139, pp. 10347–10357, PMLR, 2021.
[90] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” 2018.
[91] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strat- egy to train strong classifiers with localizable features,” in International Conference on Computer Vision (ICCV), 2019.
[92] GOOGLE, “Vision transformer models,” 2019. https://console.cloud.google. com/storage/browser/vit_models.
[93] A. Steiner, A. Kolesnikov, X. Zhai, R. Wightman, J. Uszkoreit, and L. Beyer, “How to train your vit? data, augmentation, and regularization in vision transformers,” 2021.
[94] A. Dubey, O. Gupta, P. Guo, R. Raskar, R. Farrell, and N. Naik, “Pairwise confu- sion for fine-grained visual classification,” in The European Conference on Computer Vision (ECCV), 2018.
[95] A. Dubey, O. Gupta, R. Raskar, and N. Naik, “Maximum entropy fine-grained classi-fication,” in Proceedings of the 32nd International Conference on Neural Information Processing Systems, p. 635–645, Curran Associates Inc., 2018.
[96] W. Luo, X. Yang, X. Mo, Y. Lu, L. S. Davis, and S.-N. Lim, “Cross-x learning for fine-grained visual categorization,” in ICCV, 2019.
[97] Y. Cui, Y. Song, C. Sun, A. Howard, and S. Belongie, “Large scale fine-grained cate- gorization and domain-specific transfer learning,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4109–4118, 2018.
[98] Y. Zhao, K. Yan, F. Huang, and J. Li, “Graph-based high-order relation discovery for fine-grained recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15079–15088, 2021.
[99] P. Zhuang, Y. Wang, and Y. Qiao, “Learning attentive pairwise interaction for fine- grained classification,” in AAAI, pp. 13130–13137, 2020.
[100] D. Korsch, P. Bodesheim, and J. Denzler, “Classification-specific parts for improving fine-grained visual categorization,” in German Conference on Pattern Recognition (GCPR), pp. 62–75, 2019.
[101] L. Zhang, S. Huang, W. Liu, and D. Tao, “Learning a mixture of granularity-specific experts for fine-grained categorization,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8330–8339, 2019.
[102] H. Touvron, A. Vedaldi, M. Douze, and H. Jégou, “Fixsenet,” in Advances in Neural Information Processing Systems (NeurIPS), 2019.
[103] X. Liu, L. Wang, and X. Han, “Transformer with peak suppression and knowledge guidance for fine-grained image recognition,” 2021.
[104] Z. Yang, T. Luo, D. Wang, Z. Hu, J. Gao, and L. Wang, “Learning to navigate for fine-grained classification,” in ECCV, 2018.
[105] Y. Gao, X. Han, X. Wang, W. Huang, and M. R. Scott, “Channel interaction networks for fine-grained image categorization,” 2020.
[106] C. Liu, H. Xie, Z. Zha, and L. Ma, “Filtration and distillation: Enhancing region at- tention for fine-grained visual categorization,” in Proceedings of the AAAI Conference on Artificial Intelligence, pp. 11555–11562, 2020.
[107] C. Liu, H. Xie, Z. Zha, L. Yu, Z. Chen, and Y. Zhang, “Bidirectional attention- recognition model for fine-grained object classification,” vol. 22, pp. 1785–1795, 2020.
[108] R. Du, D. Chang, A. K. Bhunia, J. Xie, Y.-Z. Song, Z. Ma, and J. Guo, “Fine-grained visual classification via progressive multi-granularity training of jigsaw patches,” in European Conference on Computer Vision, 2020.