在當今的數位時代，科技正重塑我們感知和參與世界的方式。本文探討了在虛擬環境中透過感官交換創造新的互動搜索體驗的可能性。主要研究問題是：透過在虛擬環境中實驗音效和視覺元素，能否建構一個互動搜索體驗？如果可以，這種發展如何透過感官交換創造新的搜索體驗？
本研究整合研究方法，結合歷史和當代觀點、實驗項目、參與者訪談和主題分析。開發了三個項目，最終項目「Neo-Sense」涉及視覺和聽覺的重新定位。參與者測試了三種感官交換組合的項目。
本研究從 3D 開發人員和聲音設計師的角度描述了創建這三個項目的經驗，並使用參與式行動研究方法評估參與者的體驗。研究結果表明，參與者探索出一套新的感官理解策略，能夠理解並欣賞趣味性和可變化的感官體驗。此外，比較分析表明，雖然使用者對於屏幕式的虛擬環境更為熟悉，但頭戴式顯示器（HMD）更適合創造直觀和創新的感官交換體驗。
這項研究有助於重新想像空間感知，強調探索和參與者參與作為在虛擬環境中互動改變感官搜索體驗中感官再學習和適應的重要方面。研究結果將為數位藝術家和設計師、VR 開發人員和跨學科創作者提供參考。

In the current digital age, technology is reshaping our perception and engagement with the world. This thesis explores sensory swapping in virtual environments to create new interactive search experiences. The main research question is: Can I create an interactive searching experience by experimenting with the audio and visual components in virtual environments? If so, how does the development help to create a new search experience by swapping senses?
Using a mixed-methods approach, I combine historical and contemporary perspectives, experimental projects, participant interviews, and thematic analysis. Three projects were developed, with the final project, Neo-Sense, involving the relocation of sight and hearing. Participants tested three variations of sense swapping.
I describe my experience of creating these projects from the perspective of a 3D developer and sound designer, using participatory experiments to assess participants’ experience. The results indicate that participants discovered new strategies to understand and appreciate playful and altered sensory experiences. Furthermore, comparative analysis showed that while screen based virtual environments are more familiar, a head mounted display (HMD) is more suitable for creating intuitive and innovative sense swapping experiences.
The research contributes to a reimagining of spatial perceptions, emphasising exploration and participant engagement as important aspects of relearning and adaptability of the senses within interactive altered sensory searching experiences in virtual environments. The research findings will serve as a reference for digital artists and designers, VR developers, and interdisciplinary creators.

Adams, R., Gibson, S., & Arisona, S. M. (Eds.). (2008). Transdisciplinary digital art: Sound, vision and the new screen (Vol. 7). Springer Science & Business Media.
Attride-Stirling, J. (2001). Thematic networks: an analytic tool for qualitative research. Qualitative research, 1(3), 385-405.
Aukstakalnis. (2017). Practical augmented reality : a guide to the technologies, applications, and human factors for AR and VR / Steve Aukstakalnis. Addison-Wesley.
Bacanlı, Dombaycı, M. A., Demir, M., & Tarhan, S. (2011). Quadruple Thinking: Creative Thinking. Procedia, Social and Behavioral Sciences, 12, 536–544. https://doi.org/10.1016/j.sbspro.2011.02.065
Bach-y-Rita, P., & Kercel, S. W. (2003). Sensory substitution and the human–machine interface. Trends in cognitive sciences, 7(12), 541-546.
Barrett, M. S., Creech, A., & Zhukov, K. (2021). Creative Collaboration and Collaborative Creativity: A Systematic Literature Review. Frontiers in Psychology, 12, 713445–713445. https://doi.org/10.3389/fpsyg.2021.713445
Berghege, M.C.J. (2023) Twilight of virtualities: Imagining and playfulness in an ambiguous virtual reality. University of Wolverhampton. http://hdl.handle.net/2436/625243
Bialkova, S., & Gisbergen, M. (2017). When sound modulates vision: VR applications for art and entertainment. 2017 IEEE 3rd Workshop on Everyday Virtual Reality (WEVR), 1-6. https://doi.org/10.1109/WEVR.2017.7957714
Biggs, M. (Michael A. R. ), & Karlsson, Henrik. (2011). The Routledge companion to research in the arts / edited by Michael Biggs and Henrik Karlsson ; in collaboration with Stiftelsen Riksbankens Jubileumsfond, Stockholm, www.rj.se. (1st ed.). Routledge.
Brailas, A., Tragou, E., & Papachristopoulos, K. (2023). Introduction to Qualitative Data Analysis and Coding with QualCoder. American Journal of Qualitative Research, 7(3), 19-31.
Borgmann, A. (1984). Technology and the character of contemporary life: A philosophical inquiry. University of Chicago Press.
Brinkmann, & Kvale, S. (2015). InterViews : learning the craft of qualitative research interviewing / Svend Brinkmann, Aalborg University, Denmark ; Steinar Kvale, University of Aarhus. (Third edition.). Sage Publications.
Buckmaster, L. (2023). ‘Even closing my eyes is an intense movement’: the VR experience that simulates a serious neurological condition. The Guardian Official Website. Retrieved from: https://www.theguardian.com/film/2023/aug/10/even-closing-my-eyes-is-an-intense-movement-the-vr-experience-that-simulates-a-serious-neurological-condition
Burnard, P., Gill, P., Stewart, K., Treasure, E., & Chadwick, B. (2008). Analysing and presenting qualitative data. British dental journal, 204(8), 429-432.
Buzan, T., & Buzan, B. (2006). The mind map book. Pearson Education.
Candy, L. (2006). Practice based research: A guide. CCS report, 1(2), 1-19.
Cardaioli, M., Scattolin, M., & Bisiacchi, P. (2017, September). Equilibrioception: A Method to Evaluate the Sense of Balance. In DCPD@ CHItaly (pp. 143-147).
Collins, H. (Hilary J. ). (2019). Creative research : the theory and practice of research for the creative industries / Hilary Collins. (Second edition.). Bloomsbury Visual Arts.
Conceição, S. C., Samuel, A., & Yelich Biniecki, S. M. (2017). Using concept mapping as a tool for conducting research: An analysis of three approaches. Cogent Social Sciences, 3(1), 1404753.
Cooper, N., Milella, F., Pinto, C., Cant, I., White, M., & Meyer, G. (2018). The effects of substitute multisensory feedback on task performance and the sense of presence in a virtual reality environment. _PloS one_, _13_(2), e0191846.
Cornelio, P., Velasco, C., & Obrist, M. (2021). Multisensory integration as per technological advances: A review. Frontiers in Neuroscience, 614.
Cowan, E. (2018). Research Methodologies for the Creative Arts & Humanities: Exegesis.
Creswell, J. W. (2012). Educational research : planning, conducting, and evaluating quantitative and qualitative research / John W. Creswell. (4th ed.). Pearson.
Creswell, & Creswell, J. D. (2023). Research design : qualitative, quantitative, and mixed methods approaches / John W. Creswell, J. David Creswell. (Sixth edition.). SAGE Publications, Inc.
Creswell, & Poth, C. N. (2018). Qualitative inquiry & research design : choosing among five approaches / John W. Creswell, University of Michigan, Cheryl N. Poth, University of Alberta. (Fourth edition.). SAGE.
Cross, N. (1999). Design research: A disciplined conversation. Design issues, 15(2), 5-10.
Csikszentmihalyi, M. (1997). Flow and the psychology of discovery and invention. HarperPerennial, New York, 39, 1-16.
Csikszentmihalyi, M. (2013). Flow: The psychology of happiness. Random House.
de Rooij, A., van Dartel, M., Ruhl, A., Schraffenberger, H., van Melick, B., Bontje, M., ... & Witter, M. (2018). Sensory augmentation: toward a dialogue between the arts and sciences. In _Interactivity, Game Creation, Design, Learning, and Innovation: 6th International Conference, ArtsIT 2017, and Second International Conference, DLI 2017, Heraklion, Crete, Greece, October 30–31, 2017, Proceedings 6_ (pp. 213-223). Springer International Publishing.
Deroy, O. (Ed.). (2017). Sensory blending: On synaesthesia and related phenomena. Oxford University Press.
Dieck, Jung, T. H., & Loureiro, S. M. C. (Eds.). (2021). Augmented reality and virtual reality : new trends in immersive technology / edited by M. Claudia tom Dieck, Timothy H. Jung, Sandra M. C. Loureiro. Springer Nature Switzerland. https://doi.org/10.1007/978-3-030-68086-2
Dunne, A., & Raby, F. (2013). Speculative everything: design, fiction, and social dreaming. MIT press.
Eagleman, D. (2020). Livewired: The inside story of the ever-changing brain. Canongate Books.
Eagleman, D. M., & Perrotta, M. V. (2023). The future of sensory substitution, addition, and expansion via haptic devices. _Frontiers in Human Neuroscience_, _16_, 1055546.
Earnshaw, R., Liggett, S., Excell, P., & Thalmann, D. (2020). Technology, Design and the Arts-Opportunities and Challenges.
Faulkner, M. (Ed.). (2006). VJ: Audio-Visual Art and VJ Culture: Includes DVD. Laurence King Publishing.
Floridi, L. (2014). The fourth revolution: How the infosphere is reshaping human reality. OUP Oxford.
Frayling, C. (1993). Research in art and design. Royal College of Art research papers, 1, 1-5.
Garner, A. R., & Keller, G. B. (2022). A cortical circuit for audio-visual predictions. Nature neuroscience, 25(1), 98-105.
Ghosh, Souvick & Rath, Manasa & Shah, Chirag. (2018). Searching as Learning: Exploring Search Behavior and Learning Outcomes in Learning-related Tasks. 22-31. 10.1145/3176349.3176386.
Girju, & Lambert, C. (2021). Inter-Sense: An Investigation of Sensory Blending in Fiction. arXiv.org. https://doi.org/10.48550/arxiv.2110.09710
Gray, & Malins, J. (2004). Visualising research : a guide to the research process in art and design / Carole Gray and Julian Malins. Ashgate.
Grimshaw. (2014). The Oxford handbook of virtuality / edited by Mark Grimshaw. Oxford University Press.
Hamilton, Jillian & Jaaniste, Luke (2009) Content, structure and orientations of the practice-led exegesis. In Melles, G (Ed.) Art, Media, Design: Writing Intersections: Conference Proceedings. Swinburne University of Technology, Faculty of Design, Australia, pp. 184- 195. https://eprints.qut.edu.au/29703/
Harbisson, N. (2019). Hearing colors: My life experience as a cyborg. In _Creativity, imagination and innovation: perspectives and inspirational stories_ (pp. 117-125).
Haseman, Brad (2006) A Manifesto for Performative Research. Media International Australia incorporating Culture and Policy, theme issue "Practice-led Research"(no. 118):pp. 98-106.
Hashim, H. T., & Alexiou, A. (2022). Drugs and Consciousness. In The Psychology of Consciousness: Theory and Practice (pp. 81–93). Springer International Publishing AG. https://doi.org/10.1007/978-3-030-90692-4_6
Heywood, I., & Howes, D. (Eds.). (2017). Sensory Arts and Design. Bloomsbury Publishing.
Iaconesi, S. (2019). Approaches, methods and tools for Speculative Design. Retrieved from : https://speculativeedu.eu/approaches-methods-and-tools-for-speculative-design/
Isaacson, W. (2014). The innovators: How a group of inventors, hackers, geniuses and geeks created the digital revolution. Simon and Schuster.
Jouhki, H., & Oksanen, A. (2022). To Get High or to Get Out? Examining the Link between Addictive Behaviors and Escapism. Substance Use & Misuse, 57(2), 202–211. https://doi.org/10.1080/10826084.2021.2002897
Kennedy, I. (2018). The Inaudibility of Contemporary Technics: Carsten Nicolai's Telefunken. Discourse, 40(3), 308-330.
Kapralos, Collins, K., & Uribe-Quevedo, A. (2017). The senses and virtual environments. The Senses & Society, 12(1), 69–75. https://doi.org/10.1080/17458927.2017.1268822
Kean, S., Hall, J. C., Perry, P., & Webb, J. (2011). Meet the Kinect: An introduction to programming natural user interfaces. New York, NY, USA:: Apress.
Kim, C. S. (2013). Christine Sun Kim. Soundings: A Contemporary Score, New York:: The Museum of Modern Art, 38-41.
Kim, Y. J., & Zhong, C.-B. (2017). Ideas rise from chaos: Information structure and creativity. Organizational Behavior and Human Decision Processes, 138, 15–27. https://doi.org/10.1016/j.obhdp.2016.10.001
Klempe. (2022). Sound and Reason: Synesthesia As Metacognition (1st ed.). Palgrave Macmillan. https://doi.org/10.1007/978-981-19-2340-1
Koureas, G. (Ed.). (2017). " Art, History and the Senses": 1830 to the Present. Routledge.
Kristjánsson, Á., Moldoveanu, A., Jóhannesson, Ó. I., Balan, O., Spagnol, S., Valgeirsdóttir, V. V., & Unnthorsson, R. (2016). Designing sensory-substitution devices: Principles, pitfalls and potential 1. _Restorative neurology and neuroscience_, _34_(5), 769-787.
Kuniavsky, Mike. (2003). Observing the user experience : a practitioner’s guide to user research / Mike Kuniavsky. Morgan Kaufmann Publishers.
Langlitz, N. (2013). Neuropsychedelia : the revival of hallucinogen research since the decade of the brain / Nicolas Langlitz. University of California Press.
Laurent, B (2021). Multisensory stimulation. https://labs.ullo.fr/author/brigitte_laurent/
LaValle, S. M. (2023). Virtual reality. Cambridge university press.
Liu, Hu, H., Zhou, C., Bian, Y., & Liu, J. (2021). Exploring the Role of Distraction in Weak Flow–Performance Link Based on VR Searching Tasks. Applied Sciences, 11(13), 5799–. https://doi.org/10.3390/app11135799
Lopes, D. M. M. (2000). What is it like to see with your ears? The representational theory of mind. Philosophical and Phenomenological Research, 439-453.
Massiceti, D., Hicks, S. L., & van Rheede, J. J. (2018). Stereosonic vision: Exploring visual-to-auditory sensory substitution mappings in an immersive virtual reality navigation paradigm. PloS one, 13(7), e0199389.
Macpherson, F. (2018). Sensory substitution and augmentation: An introduction.
Maeda, John. (2004). Creative code / John Maeda ; foreword by Red Burns. Thames & Hudson.
Makhlin, Jessica, "Chromesthesia as Phenomenon: Emotional Colors" (2014). Writing Programs. 12. https://digitalcommons.lmu.edu/arc_wp/12
Martin, & Hanington, B. M. (2012). Universal methods of design : 100 ways to research complex problems, develop innovative ideas, and design effective solutions / Bella Martin, Bruce Hanington. (Digital ed.). Rockport Publishers.
Maxwell, J. A. (2012). Qualitative research design: An interactive approach. Sage publications.
McCreery, M. P., Schrader, P. G., Krach, S. K., & Boone, R. (2013). A sense of self: The role of presence in virtual environments. _Computers in Human Behavior_, _29_(4), 1635-1640.
McMenemy, & Ferguson, S. (2007). A hitchhiker’s guide to virtual reality / Karen McMenemy, Stuart Ferguson. A K Peters, Ltd.
McNiff, S. (Ed.). (2013). Art as research : opportunities and challenges / edited by Shaun McNiff. Intellect.
Merzenich, M. M., Van Vleet, T. M., & Nahum, M. (2014). Brain plasticity-based therapeutics. Frontiers in human neuroscience, 8, 385.
Moody, N., Fells, N., & Bailey, N. (2006). Motion as the connection between audio and visuals. In ICMC.
Mortari, L. (2015). Reflectivity in research practice: An overview of different perspectives. International journal of qualitative methods, 14(5), 1609406915618045.
Musante, K., & DeWalt, B. R. (2010). Participant observation: A guide for fieldworkers. Rowman Altamira.
Nijhof, E. (2011). Searching? Or actually trying to find something?–The comforts of searching versus the challenges of finding. World Patent Information, 33(4), 360-363.
O’Dwyer, L. M., & Bernauer, J. A. (2014). Quantitative research for the qualitative researcher / Laura M. O’Dwyer, James A. Bernauer. SAGE.
Olman, C. (2022). Sensory Substitution, Introduction to Sensation and Perception https://pressbooks.umn.edu/sensationandperception/chapter/sensory-substitution/
Park, & Alderman, J. (2018). Designing across senses : a multimodal approach to product design / Christine W. Park and John Alderman.. (First edition.). O’Reilly Media.
Pascual-Leone, A., Amedi, A., Fregni, F., & Merabet, L. B. (2005). The plastic human brain cortex. Annu. Rev. Neurosci., 28, 377-401.
Prakash, S. (2022). _Psychedelic Virtual Reality: A Mixed Methods Study of Psychological Transformation_. California Institute of Integral Studies.
Richardson, J. (2012). An eye for music [electronic resource] : popular music and the audiovisual surreal / John Richardson. Oxford University Press.
Rigoulot, S., & Pell, M. D. (2012). Seeing emotion with your ears: emotional prosody implicitly guides visual attention to faces. PloS one, 7(1),
Rudenko, S., McDonnell, M., Layden, T., & Haahr, M. (2022). Exploring Classical Music Narratives Through Multimodality in AR/VR Experiences. In International Conference on Interactive Digital Storytelling (pp. 395-404). Cham: Springer International Publishing.
Stanton, C., & Clune, J. (2016). Curiosity search: producing generalists by encouraging individuals to continually explore and acquire skills throughout their lifetime. PloS one, 11(9), e0162235.
Schiffman. (2000). Sensation and perception : an integrated approach / Harvey Richard Schiffman. (5th ed.). Wiley.
Smith, H., & Dean, R. T. (2009). Practice-led research, research-led practice in the creative arts / [edited by] Hazel Smith and Roger T. Dean. Edinburgh University Press.
Smith, M. M. (2007). Sensing the past: Seeing, hearing, smelling, tasting, and touching in history. Univ of California Press.
SoundSelf (n.d.), SoundSelf Research, Retrieved October 19, 2020, from the SoundSelf Website, https://soundself.com/research/
Stanton, C., & Clune, J. (2016). Curiosity search: producing generalists by encouraging individuals to continually explore and acquire skills throughout their lifetime. PloS one, 11(9), e0162235.
Stigchel. (2016). How Attention Works : finding your way in a world full of distraction / Stefan Van der Stigchel ; translated by Danny Guinan. (Guinan, Trans.). The MIT Press.
Street. (2017). Sound poetics : interaction and personal identity / by Seán Street. Springer International Publishing. https://doi.org/10.1007/978-3-319-58676-2
Sweller, J. (2011). Cognitive load theory. In Psychology of learning and motivation (Vol. 55, pp. 37-76). Academic Press.
Tenner, E. (1997). Why things bite back: Technology and the revenge of unintended consequences. Vintage.
Trivedi, B. (2010). Sensory hijack: rewiring brains to see with sound. Science, 2773, 42-45. https://www.newscientist.com/article/mg20727731-500-sensory-hijack-rewiring-brains-to-see-with-sound/
Tucker, T. (2021). Virtual Sensory Interfaces. In NIME 2021. PubPub.
Tullett, W. (2021). State of the field: sensory history. History, 106(373), 804-820.
Tuthill, J. C., & Azim, E. (2018). Proprioception. Current Biology, 28(5), R194-R203.
Tyurina, T., & Stavkova, S. (2020). Harmonization of the Activity of the Left and Right Cerebral Hemispheres-an Important Component of the Spiritual and Mental Health of Individual and Humanity. _Mental Health: Global Challenges Journal_, _3_(2), 45-49.
Verma, & Paul, S. (Eds.). (2022). Advances in augmented reality and virtual reality / edited by Jitendra Kumar Verma, Sudip Paul. Springer Singapore. https://doi.org/10.1007/978-981-16-7220-0
Virginia Tech (2021). Expanding virtual reality with a blend of art, technology, and experimentation. Retrieved from : https://news.vt.edu/articles/2021/03/icat-virtualsensoryconvergance.html#:~:text=Expanding%20virtual%20reality%20with%20a,Photo%20by%20David%20Franusich
Walters, H., & Pereira, P. H. (2022). The sound of color: Neil Harbisson's talk visualized. Ideas. ted. com, 11. https://ideas.ted.com/the-sound-of-color-neil-harbissons-talk-visualized/
Ward, J., & Meijer, P. (2010). Visual experiences in the blind induced by an auditory sensory substitution device. Consciousness and cognition, 19(1), 492-500.
Weinel, J. (2018). Inner sound: altered states of consciousness in electronic music and audio-visual media. Oxford University Press.
Woolfe, (2023). Virtual Reality Simulates Psychedelic Experiences. Psychedelic Support. https://psychedelic.support/resources/virtual-reality-simulates-psychedelic-experiences
Vygotsky, L. S., & Cole, M. (1978). Mind in society: Development of higher psychological processes. Harvard university press.
Yin, R. K. (2012). Applications of case study research / Robert K. Yin. (3rd ed.). SAGE.
Zerzan, J. (2009). Running on emptiness: the failure of symbolic thought. Insurgent Desire-The Online Green Anarchy Archive.