一次性神經網路搜索的主要目標為從超網路中搜索在不同硬體資源限制之下的子網路。然而，現今的搜索方法對於 N 個不同的硬體資源限制需要重新搜索 N 次，這使得整個搜索的過程需要耗費非常大量的時間。在本論文中，我們提出一個新的搜索策略—架構生成器，通過架構生成器直接生成子網絡的方式來進行搜索，從而使搜索過程更加高效和靈活。給定不同的目標硬體資源限制作為經過預訓練後的架構生成器之輸入，便可以在一次前向傳遞當中針對 N 種不同硬體資源限制生成 N 個不同的網路架構，而這樣搜索的過程並不需要重新執行搜索策略或是重新訓練超級網路。此外，我們提出了一種新穎的單路徑超網路，稱為聯合超網路，以進一步提高搜索效率並減少架構生成器的 GPU 記憶體需求。透過架構生成器和聯合超網，我們提出一個靈活高效的一次性神經網路搜索框架，稱為 Searching by Generating NAS (SGNAS)。使用預訓練的超網路，SGNAS 對 N 個不同硬體資源限制的搜索時間僅為 5 個 GPU 小時，比最新的單路徑方法快 4N 倍。且經過從新開始訓練，SGNAS 在 ImageNet 上的 top1-accuracy 為 77.1%，與目前最優異的結果相當。

In one-shot NAS, sub-networks need to be searched from the supernet to meet different hardware constraints. However, the search cost is high and N times of searches are needed for N different constraints. In this work, we propose a novel search strategy called architecture generator to search sub-networks by generating them, so that the search process can be much more efficient and flexible. With the trained architecture generator, given target hardware constraints as the input, N good architectures can be generated for N constraints by just one forward pass without re-searching and supernet retraining. Moreover, we propose a novel single-path supernet, called unified supernet, to further improve search efficiency and reduce GPU memory consumption of the architecture generator. With the architecture generator and the unified supernet, we propose a flexible and efficient one-shot NAS framework, called Searching by Generating NAS (SGNAS). With the pre-trained supernt, the search time of SGNAS for N different hardware constraints is only 5 GPU hours, which is 4N times faster than previous SOTA single-path methods. After training from scratch, the top1-accuracy of SGNAS on ImageNet is 77.1%, which is comparable with the SOTAs.

[1] Mohamed S. Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas Donald Lane. Zerocost proxies for lightweight nas. In Proceedings of International Conference on Learning Representations, 2021.
[2] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architectures using reinforcement learning. In Proceedings of International Conference on Learning Representations, 2017.
[3] Gabriel Bender, PieterJan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc V. Le. Understanding and simplifying oneshot architecture search. In Proceedings of International Conference on Machine Learning, 2018.
[4] Maxim Berman, Leonid Pishchulin, Ning Xu, Matthew B. Blaschko, and Gerard Medioni. AOWS: Adaptive and optimal network width search with latency constraints. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[5] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Onceforall: Train one network and specialize it for efficient deployment. In Proceedings of International Conference on Learning Representations, 2020.
[6] Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In Proceedings of International Conference on Learning Representations, 2019.
[7] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019.
[8] Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the CIFAR datasets. CoRR, abs/1707.08819, 2017.
[9] Xiangxiang Chu, Xudong Li, Yi Lu, Bo Zhang, and Jixiang Li. Mixpath: A unified approach for oneshot neural architecture search. arXiv preprint arXiv:2001.05887, 2020.
[10] Xiangxiang Chu, Bo Zhang, Jixiang Li, Qingyuan Li, and Ruijun Xu. Scarletnas: Bridging the gap between scalability and fairness in neural architecture search. arXiv preprint arXiv:1908.06022, 2019.
[11] Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search. arXiv preprint arXiv:1907.01845, 2019.
[12] Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[13] Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, and Joseph E. Gonzalez. Fbnetv3: Joint architecture recipe search using neural acquisition function, 2020.
[14] Xiaoliang Dai, Peizhao Zhang, Bichen Wu, Hongxu Yin, Fei Sun, Yanghan Wang, Marat Dukhan, Yunqing Hu, Yiming Wu, Yangqing Jia, Peter Vajda, Matt Uyttendaele, and Niraj K. Jha. Chamnet: Towards efficient network design through platformaware model adaptation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[15] Xiyang Dai, Dongdong Chen, Mengchen Liu, Yinpeng Chen, and Lu Yuan. Danas: Data adapted pruning for efficient neural architecture search. In Proceedings of European Conference on Computer Vision, 2020.
[16] Jia Deng, Wei Dong, Richard Socher, LiJia Li, Kai Li, and Li FeiFei. Imagenet: A largescale hierarchical image database. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2009.
[17] Xuanyi Dong and Yi Yang. Oneshot neural architecture search via selfevaluated template network. In Proceedings of IEEE International Conference on Computer Vision, 2019.
[18] Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[19] Xuanyi Dong and Yi Yang. Nasbench201: Extending the scope of reproducible neural architecture search. In Proceedings of International Conference on Learning Representations, 2020.
[20] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 2019.
[21] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path oneshot neural architecture search with uniform sampling. In Proceedings of European Conference on Computer Vision, 2020.
[22] Andrew Howard, Mark Sandler, Grace Chu, LiangChieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. Searching for mobilenetv3. In Proceedings of IEEE International Conference on Computer Vision, 2019.
[23] Jie Hu, Li Shen, and Gang Sun. Squeeze and excitation networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[24] Yibo Hu, Xiang Wu, and Ran He. Tfnas: Rethinking three search freedoms of latency constrained differentiable neural architecture search. In Proceedings of European Conference on Computer Vision, 2020.
[25] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbelsoftmax, 2016.
[26] M. G. Kendall. A new measure of rank correlation. Biometrika, 1938.
[27] Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Proceedings of the Conference on Uncertainty in Artificial Intelligence, 2019.
[28] TsungYi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of IEEE International Conference on Computer Vision, 2017.
[29] TsungYi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2014.
[30] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In Proceedings of International Conference on Learning Representations, 2019.
[31] Jieru Mei, Yingwei Li, Xiaochen Lian, Xiaojie Jin, Linjie Yang, Alan Yuille, and Jianchao Yang. AtomNAS: Fine grained end to end neural architecture search. In Proceedings of International Conference on Learning Representations, 2020.
[32] Joseph Mellor, Jack Turner, Amos Storkey, and Elliot J. Crowley. Neural architecture search without training, 2020.
[33] Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions, 2018.
[34] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. In Proceedings of AAAI Conference on Artificial Intelligence, 2019.
[35] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, realtime object detection. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[36] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. UNet: Convolutional networks for biomedical image segmentation. In Proceedings of Medical Image Computing and ComputerAssisted Intervention, 2015.
[37] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and LiangChieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[38] Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Singlepath nas: Designing hardwareefficient convnets in less than 4 hours. In arXiv preprint arXiv:1904.02877, 2019.
[39] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. Mnasnet: Platformaware neural architecture search for mobile. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[40] Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of International Conference on Machine Learning, 2019.
[41] Mingxing Tan and Quoc V. Le. Mixconv: Mixed depthwise convolutional kernels. In Proceedings of British Machine Vision Conference, 2019.
[42] Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, Peter Vajda, and Joseph E. Gonzalez. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[43] Dilin Wang, Meng Li, Chengyue Gong, and Vikas Chandra. Attentivenas: Improving neural architecture search via attentive sampling. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[44] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardwareaware efficient convnet design via differentiable neural architecture search. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[45] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, GuoJun Qi, Qi Tian, and Hongkai Xiong. Pcdarts: Partial channel connections for memoryefficient architecture search. In Proceedings of International Conference on Learning Representations, 2020.
[46] Zhicheng Yan, Xiaoliang Dai, Peizhao Zhang, Yuandong Tian, Bichen Wu, and Matt Feiszli. Fpnas: Fast probabilistic neural architecture search. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[47] Zhaohui Yang, Yunhe Wang, Xinghao Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu. Cars: Continuous evolution for efficient neural architecture search. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[48] Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, and Changshui Zhang. Greedynas: Towards fast oneshot nas with greedy supernet. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[49] Jiahui Yu and Thomas S. Huang. Universally slimmable networks and improved training techniques. In Proceedings of IEEE International Conference on Computer Vision, 2019.
[50] Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel M. Bender, PieterJan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc V. Le. Bignas: Scaling up neural architecture search with big singlestage models. In Proceedings of European Conference on Computer Vision, 2020.
[51] Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In Proceedings of International Conference on Learning Representations, 2019.
[52] Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang, Chen Change Loy, Shuai Yi, Xuesen Zhang, and Wanli Ouyang. Econas: Finding proxies for economical neural architecture search. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2020.