文字風格轉換是一個改寫句子特定屬性的控制生成任務，此篇研究中，我們的方法
著重在正負面情緒的風格轉換，是將一句正面情緒的句子改寫為負面情緒，反之亦
然。兩兩成對的風格轉換句子在現實中是很難大量蒐集的，因此我們研究的是非成
對資料集上的風格轉換，其中並無成對句子當作模型的輸入輸出來直接訓練風格轉
換。文字填充 (Text Infilling) 是最近常用於非成對風格轉換任務的方法，它刪除句子
中與風格相關的詞，然後用另一種風格的詞填入這個被挖取的文句，這個方法具有
能保留大部分文字的優點。但是文字填充方法移除掉了一些字詞，其中的資訊也跟
著遺失，使得生成的句子和原始句子會有不同意思，內容資訊並沒有被保留住。為
了主旨一致性，我們的目標是在文字填充方法上同時提供原始句子的資訊。
我們在標準的 Transformer 模型 (編碼器­解碼器架構) 上使用變分自動編碼器 (Variational AutoEncoder) 技術從原始語句中提取內容資訊並得到語句表示 (representation)。
然後將原始句子的表示和被挖取的文句 (masked text) 同時輸入到模型中。實驗結果
顯示，跟之前的文字填充方法相比，我們的方法雖然降低了些許風格轉換的成功率，
卻保留住了更多原文內容，尤其是語意上的保留 (BERTScore)。我們的方法不僅具有
保留文字的優點，還可以一定程度上保留句子的原始語意。在轉換風格的同時維持
主旨一致性。

Text style transfer is a controllable generation task that revises certain attribute of a sentence. Our work focuses on Positive/Negative sentiment transfer, which revises a sentence with positive or negative sentiment into another. In the task, the dataset is non­parallel, which means that we do not have sentence pairs as input/output for straightforwardly training style transfer. Text infilling method is recently used on non­parallel text style transfer tasks. It removes style­relative words in a sentence and then infills the incomplete sentence with another style’s words. Because some words are removed, the information of the words also disappears. Some information is lost, so that the generated sentences often have different topic than the source sentences. For topic consistency, we aim to provide information on the original sentence to improve generation.
We use the Variational AutoEncoder(VAE) technique on a standard encoder­decoder Transformer model to extract content information from source sentences. Then a masked text and a representation are inputted into the generating model simultaneously. In our experimental results, our method can preserve more content especially semantic than previous Text Infilling methods with only a little lower transfer rate. We preserve not only words, which is the Text Infilling method’s advantage, but also preserve the semantics of the source sentence. The sentence topic consistency is maintained while transferring the style.

[1] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005.
[2] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. Advances in neural information processing systems, 29, 2016.
[3] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, pages 10–21. Association for Computational Linguistics (ACL), 2016.
[4] Kyunghyun Cho, Bart van Merriënboer, Çağlar Gu̇ lçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734, 2014.
[5] Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. Style transfer in text: Exploration and evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[6] Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one­class collaborative filtering. In WWW, 2016.
[7] Keith J Holyoak. Parallel distributed processing: explorations in the microstructure of cognition. Science, 236:992–997, 1987.
[8] Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. Disentangled representation learning for non­parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 424–434, 2019.
[9] Jacob Devlin Ming­Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre­training of deep bidirectional transformers for language understanding. In Proceedings of NAACL­HLT, pages 4171–4186, 2019.
[10] Diederik P Kingma and Max Welling. Auto­encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[11] Kalpesh Krishna, John Wieting, and Mohit Iyyer. Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 737–762, 2020.
[12] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising 42 sequence­to­sequence pre­training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, 2020.
[13] Juncen Li, Robin Jia, He He, and Percy Liang. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865–1874, 2018.
[14] Dayiheng Liu, Jie Fu, Yidan Zhang, Chris Pal, and Jiancheng Lv. Revision in continuous space: Unsupervised text style transfer without adversarial learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8376–8383, 2020.
[15] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
[16] Aman Madaan, Amrith Setlur, Tanmay Parekh, Barnabás Poczós, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W Black, and Shrimai Prabhumoye. Politeness transfer: A tag and generate approach. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1869–1881, 2020.
[17] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners.
[18] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text­to­text transformer. J. Mach. Learn. Res., 21(140):1–67, 2020.
[19] Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from nonparallel text by cross­alignment. Advances in neural information processing systems, 30, 2017.
[20] Xiaoyu Shen, Hui Su, Shuzi Niu, and Vera Demberg. Improving variational encoderdecoders in dialogue generation. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[21] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
[22] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[23] Xing Wu, Tao Zhang, Liangjun Zang, Jizhong Han, and Songlin Hu. Mask and infill: Applying masked language model to sentiment transfer.
[24] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2019.