Where Digital Meets Physical: A Conceptual Framework for Exploring Aesthetic Potential of Mixed Reality Interface
Abstract
This paper aims to conceptually explore the aesthetic potential phenomenon of a mixed reality interface. The research is intended to create a novel conceptual framework that includes characteristics of mixed reality interface and approaches to studying the aesthetic potential phenomenon. The framework is based on the synthesis of knowledge and integration of human-computer interaction concepts and methodologies from design, philosophy, psychology, and media culture. This paper offers a new expanded view of the aesthetic potential phenomenon. This includes seeing the aesthetic potential in the way the mixed reality interface manifests itself as a meaningful part of our lifeworld. Furthermore, the next research steps are outlined under a new perspective of studying the interface and its future.
Downloads
References
Arnheim, R. (2004). Art and Visual Perception: A Psychology of the Creative Eye. University of California Press.
Baudrillard, J. (1994). Simulacra and Simulation. University of Michigan Press.
Berning, M., Kleinert, D., Riedel T., & Beigl M. (2014). A study of depth perception in hand-held augmented reality using autostereoscopic displays. IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 93–98. https://doi.org/10.1109/ISMAR.2014.6948413
Bertelsen, O. W. (2004). Transparency by Tertiary Artefactness. In Workshop on Aesthetic Approaches to Human-Computer Interaction, DAIMI Report Series, 33(572), 1–4. https://doi.org/10.7146/dpb.v33i572.7129
Bleecker, J. (2009) Design Fiction: A short essay on design, science, fact and fiction. Near Future Laboratory.
Blythe, M. (2014). Research through design fiction. Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems – CHI ’14. https://doi.org/10.1145/2556288.2557098
Carroll, J. M. (2010). Conceptualizing a possible discipline of human–computer interaction. Interacting with Computers, 22(1), 3–12. https://doi.org/10.1016/j.intcom.2009.11.0
Chen, S., & Duh, H. (2019). Interface of mixed reality: from the past to the future. CCF Trans. Pervasive Comp. Interact., 1, 69–87. https://doi.org/10.1007/s42486-018-0002-8
Christensen, M. S. (2004). Introducing Excitability! In Workshop on Aesthetic Approaches to Human-Computer Interaction, DAIMI Report Series, 33(572), 10–13. https://doi.org/10.7146/dpb.v33i572.7129
Chuah, S. H-W. (2019). Wearable XR-technology: literature review, conceptual framework and future research directions. International Journal of Technology Marketing, 13(3/4), 205–259. https://doi.org/10.1504/IJTMKT.2019.104586
Costanza, E., Kunz, A., & Fjeld, M. (2009). Mixed Reality: A Survey. In Lalanne, D., & Kohlas, J. (Eds.), Human Machine Interaction, 47–68. Springer https://doi.org/10.1007/978-3-642-00437-7_3
Coulton, P., Burnett, D., and Gradinar, A. (2016). Games as Speculative Design: Allowing Players to Consider Alternate Presents and Plausible Features. In Lloyd, P. & Bohemia, E. (Eds.), Future Focused Thinking – DRS International Conference 2016. https://doi.org/10.21606/drs.2016.15
Coulton, P., & Hook, A. (2017). Games design research through game design practice. In Lankoski, P., & Holopainen J. (Eds.), Game Design Research: An Introduction to Theory and Practice, 97–116. ETC Press.
Delbridge, R., & Fiss, P. C. (2013). Editors’ comments: Styles of theorizing and the social organization of knowledge. Academy of Management Review, 38(3), 325–331. https://doi.org/10.5465/amr.2013.0085
Dourish, P., & Bell, G. (2013). “Resistance is futile”: reading science fiction alongside ubiquitous computing. Personal and Ubiquitous Computing, 18(4), 769–778. https://doi.org/10.1007/s00779-013-0678-7
Dunne, A. (1999). Hertzian Tales: Electronic Products, Aesthetic Experience, and Critical Design. RCA CRD Research
El Jamiy, F., & Marsh, R. (2019). Survey on depth perception in head mounted displays: distance estimation in virtual reality, augmented reality, and mixed reality. IET Image Processing, 13, 707-712. https://doi.org/10.1049/iet-ipr.2018.5920
Ermi, L., & Mäyrä, F. (2005). Player-Centered Game Design: Experiences in Using Scenario Study to Inform Mobile Game Design. Game Stud., 5.
Figueiredo L.S., Pinheiro M., Neto E.V., Chaves T., & Teichrieb V. (2015). Sci-Fi Gestures Catalog. In Abascal J., Barbosa S., Fetter M., Gross T., Palanque P., Winckler M. (Eds.), Human-Computer Interaction – INTERACT 2015. INTERACT 2015. Lecture Notes in Computer Science, vol 9297. Springer, Cham. https://doi.org/10.1007/978-3-319-22668-2_30
Gray, B. (1950). The Social Effects of the Film. The Sociological Review, a42(1), 135–144. https://doi.org/10.1111/j.1467-954x.1950.tb02467.x
Guberman, S. (2017). Gestalt Theory Rearranged: Back to Wertheimer. Frontiers in Psychology, 8. https://doi.org/10.3389/fpsyg.2017.01782
Guerrilla Games. (2017). Horizon Zero Dawn (PC version) [Video game]. Sony Interactive Entertainment.
Hallnäs, L., & Redström, J. (2001). Slow Technology – Designing for Reflection. Personal and Ubiquitous Computing, 5(3), 201–212. https://doi.org/10.1007/pl00000019
Hallnäs, L., & Redström, J. (2002). From Use to Presence: On the Expression and Aesthetics of Everyday Computational Things. ACM Transaction on Computer-Human Interaction, 9(2), 106-124. https://doi.org/10.1145/513665.513668
Hammady, R., Strathearn, C., & Ma, M. (2019). User Experience Design for Mixed Reality: A Case Study of HoloLens in Museum. International Journal of Technology Marketing. https://doi.org/10.1504/ijtmkt.2019.10025609
Hammel, M. (2004). The aesthetics of Use. In Workshop on Aesthetic Approaches to Human-Computer Interaction, DAIMI Report Series, 33(572), 17–19. https://doi.org/10.7146/dpb.v33i572.7129
Hartmann, J., Sutcliffe, A., & De Angeli, A. (2008). Towards a theory of user judgment of aesthetics and user interface quality. ACM Transactions on Computer-Human Interaction, 15(4), 1–30. https://doi.org/10.1145/1460355.1460357
Heinrich, F., Bornemann, K., Lawonn, K., & Hansen, C. (2019). Depth Perception in Projective Augmented Reality: An Evaluation of Advanced Visualization Techniques. 25th ACM Symposium on Virtual Reality Software and Technology (VRST ’19), 26, 1–11. https://doi.org/10.1145/3359996.3364245
Hirschheim, R. (2008). Some guidelines for the critical reviewing of conceptual papers. Journal of the Association for Information Systems, 9(8), 432–441. https://aisel.aisnet.org/jais/vol9/iss8/21
Hochheiser, H., & Lazar, J. (2007). HCI and Societal Issues: A Framework for Engagement. International Journal of Human-Computer Interaction, 23(3), 339–374. https://doi.org/10.1080/10447310701702717
Jaakkola, E. (2020). Designing conceptual articles: four approaches. AMS Rev 10, 18–26. https://doi.org/10.1007/s13162-020-00161-0
Johnson, B. D. (2011a). Love and God and Robots: The Science Behind the Science Fiction Prototype “Machinery of LoveGrace”. In Workshop Proceedings of the 7th International Conference on Intelligent Environments, 10, 99–127. https://doi.org/10.3233/978-1-60750-795-6-99
Johnson, B. D. (2011b). Science Fiction Prototyping: Designing the Future with Science Fiction. Synthesis Lectures on Computer Science, 3(1), 1–190. https://doi.org/10.2200/S00336ED1V01Y201102CSL003
Jordan, P. (2002). Human factors for pleasure seekers. In Frascara, J., Design and the social science making new connections. London: Taylor & Francis.
Johnson, J. (2020). Designing with the mind in mind. Burlington: Elsevier.
Interface Studies. (2016, March 15). http://www.interfacestudies.org/
Iso, M. (Writer & Director). (2007). Dennou Coil [TV series]. Tokuma Shoten; Bandai Visual; NHK Enterprises.
Kim, H., Lee, J., Yeo, H.-S., Quigley, A., & Woo, W. (2018). SWAG Demo: Smart Watch Assisted Gesture Interaction for Mixed Reality Head-Mounted Displays. 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). https://doi.org/10.1109/ismar-adjunct.2018.00130
Kim, D.-H., Go, Y.-G., Choi, S.-M. (2020). An Aerial Mixed-Reality Environment for First-Person-View Drone Flying. Applied Sciences, 10(16). https://doi.org/10.3390/app10165436
Kirby, D. (2009). The Future is Now. Social Studies of Science, 40(1), 41–70. https://doi.org/10.1177/0306312709338325
Knoller, N., & Ben-Arie, U. (2015). The Holodeck is all Around Us – Interface Dispositifs in Interactive Digital Storytelling. In Koenitz, H., Ferri, G., Haahr, M., Sezen, D., & Sezen, T.İ. (Eds.), Interactive Digital Narrative: History, Theory and Practice. Routledge. https://doi.org/10.4324/9781315769189-5
Kymäläinen, T. (2016). Science Fiction Prototypes as a Method for Discussing Socio-Technical Issues within Emerging Technology Research and Foresight. Athens Journal of Technology and Engineering, 3(4), 333–347.
Lavie, T., & Tractinsky, N. (2004). Assessing dimensions of perceived visual aesthetics of websites. International Journal of Human-Computer Studies, 60(3), 269–298. https://doi.org/10.1016/j.ijhcs.2003.09.002
Lee, R. M., Fielding, N., & Blank, G. (2008). The Internet as a research medium: An editorial introduction to the Sage handbook of online research methods. In Fielding, N., Lee, R. M., & Blank, G. (Eds.), The Sage handbook of online research methods, 3–20. SAGE Publications. https://doi.org/10.4135/9780857020055.n1
Lee, T., & Hollerer, T. (2007). Handy AR: Markerless Inspection of Augmented Reality Objects Using Fingertip Tracking. 11th IEEE International Symposium on Wearable Computers. https://doi.org/10.1109/iswc.2007.4373785
Lim, S. G., & Kim, C. (2018). A Study on the 21st Century Digital Visuality Through Lacan’s Notion of the Real Gaze – From an Aspect of Digital Frame Expension. Journal of Korea Multimedia Society, 21(5), 638–647. https://doi.org/10.9717/KMMS.2018.21.5.638
Ngo, D., Teo, L. S., & Byrne, J. G. (2003). Modeling interface aesthetics. Information Sciences, 152(1), 25-46. https://doi.org/10.1016/S0020-0255(02)00404-8
MacInnis, D. J. (2011). A framework for conceptual contributions in marketing. Journal of Marketing, 75(4), 136–154. https://doi.org/10.1509/jmkg.75.4.136
Marcus, A., Norman, D. A., Rucker, R., Sterling, B., & Vinge, V. (1992). Sci-fi at CHI. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’92. https://doi.org/10.1145/142750.142892
Marcus, A., Soloway, E., Sterling, B., Swanwick, M., & Vinge, V. (1999). Opening plenary: sci-fi @ CHI-99. Extended Abstracts on Human Factors in Computing Systems – CHI ’99. https://doi.org/10.1145/632716.632775
Marcus, A. (2013). The history of the future. Interactions, 20(4), 64. https://doi.org/10.1145/2486227.2486240
Merino, L., Schwarzl, M., Kraus, M., Sedlmair, M., Schmalstieg D., & Weiskopf, D. (2020). Evaluating Mixed and Augmented Reality: A Systematic Literature Review (2009–2019). 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 438–451. https://doi.org/10.1109/ISMAR50242.2020.00069
Milgram, P., Takemura, H., Utsumi, A., & Kishino, F. (1995). Augmented reality: a class of displays on the reality-virtuality continuum. Telemanipulator and Telepresence Technologies. https://doi.org/10.1117/12.197321
Montfort, N. (2017). The Future. MIT Press.
Moshagen, M., & Thielsch, M. T. (2010). Facets of visual aesthetics. International Journal of Human-Computer Studies, 68(10), 689–709. https://doi.org/10.1016/j.ijhcs.2010.05.006
Mubin, O., Obaid, M., Jordan, P., Alves-Oliveria, P., Eriksson, T., Barendregt, W., Sjolle, D., Fjeld, M., Simoff, S., & Billinghurst, M. (2016). Towards an Agenda for Sci-Fi Inspired HCI Research. Proceedings of the 13th International Conference on Advances in Computer Entertainment Technology. https://doi.org/10.1145/3001773.3001786
Perez Mata, M. & Ahmed-Kristensen, S. (2015). Principles for Designing for Perception. In Proceedings of the 20th International Conference on Engineering Design (ICED 15), 9, 239–248.
Pham, B. (1999). Design for aesthetics: interactions of design variables and aesthetic properties. In Proceedings of SPIE IS&T/SPIE 11th Annual Symposium (Electronic Imaging '99), 3644, 364–371. https://doi.org/10.1117/12.348457
Pinhanez, C. (2001). Augmenting reality with projected interactive displays. In Proceedings of the International Symposium on Virtual and Augmented Architecture (VAA’01), 93–100. Springer, London. https://doi.org/10.1007/978-1-4471-0337-0_9
Raskar, R., van Baar, J., Beardsley, P., Willwacher, T., Rao, S., & Forlines, C. (2006). iLamps: geometrically aware and self-configuring projectors. ACM SIGGRAPH 2006 Courses on – SIGGRAPH’06. https://doi.org/10.1145/1185657.1185802
Rauschnabel, P. A. (2018). Virtually enhancing the real world with holograms: An exploration of expected gratifications of using augmented reality smart glasses. Psychology & Marketing, 35(8), 557–572. https://doi.org/10.1002/mar.21106
Rosenberg, L. B. (1993). Virtual fixtures: Perceptual tools for telerobotic manipulation. Proceedings of IEEE Virtual Reality Annual International Symposium, 76–82, https://doi.org/10.1109/VRAIS.1993.380795
Soegaard, M. (2015). Gestalt principles of form perception. In Papantoniou, B., Soegaard, M., Lupton, J. R., Goktürk, M., Trepess, D., Knemeyer, D., Svoboda, E., Memmel, T., Folmer, E., Gunes, H., Harrod, M., Spillers, F., Hornecker, E. (eds). The Glossary of Human Computer Interaction. The Interaction Design Foundation.
Sonderegger, A., & Sauer, J. (2010). The influence of design aesthetics in usability testing: Effects on user performance and perceived usability. Applied ergonomics, 41(3), 403–410. https://doi.org/10.1016/j.apergo.2009.09.002
Schmidt, S., Bruder, G., & Steinicke F. (2020). Depth Perception and Manipulation in Projection-Based Spatial Augmented Reality. PRESENCE: Virtual and Augmented Reality, 27(2), 242–256. https://doi.org/10.1162/pres_a_00329
Schmitz, M., Endres, C., & Butz, A. (2008). A Survey of Human-Computer Interaction Design in science fiction movies. In Proceedings of the 2nd international conference on INtelligent TEchnologies for interactive enterTAINment (INTETAIN '08). https://doi.org/10.4108/ICST.INTETAIN2008.2476
Speicher, M., Hall, B. D., & Nebeling, M. (2019). What is Mixed Reality? Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems – CHI’19. https://doi.org/10.1145/3290605.3300767
Tanenbaum, J. (2014). Design fictional interactions. Interactions, 21(5), 22–23. https://doi.org/10.1145/2648414
Thüring, M., & Mahlke, S. (2007). Usability, aesthetics and emotions in human-technology interaction. International Journal of Psychology, 42(4), 253–264. https://doi.org/10.1080/00207590701396674
Terashima, N. (2005). The definition of hyperreality. In John Tiffin & Nobuyoshi Terashima (org.), Hyperreality – Paradigm for the Third Millennium, 4–24. Routledge.
Tiffin, J. (2005). The hyperreality paradigm. In John Tiffin & Nobuyoshi Terashima (org.), Hyperreality – Paradigm for the Third Millennium, 25–42. Routledge.
Tschimmel, K. (2011). Design as a Perception-in-Action Prozess. In Taura, T., & Nagai, Y., (Eds.), Design Creativity 2010, 223–230. Springer Verlag. https://doi.org/10.1007/978-0-85729-224-7_29
Udsen, L., & Jørgensen, A-H. (2005). The aesthetic turn: Unraveling recent aesthetic approaches to human-computer interaction. Digital Creativity, 16(4), 205-216. https://doi.org/10.1080/14626260500476564
Unberath, M., Yu, K., Barmaki, R., Johnson, A., & Navab, N. (2020). Augment Yourself: Mixed Reality Self-Augmentation Using Optical See-through Head-mounted Displays and Physical Mirrors. ArXiv. https://doi.org/10.48550/arXiv.2007.02884
Wagemans, J., Elder, J. H., Kubovy, M., Palmer, S. E., Peterson, M. A., Singh, M., & von der Heydt, R. (2012). A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization. Psychological Bulletin, 138(6), 1172–1217. https://doi.org/10.1037/a0029333
Wei, L., & Cho, D.-M. (2021). Research on Design of Mixed Reality Interface Based on Spatial Perception. Journal of Korea Multimedia Society, 24(6), 815–824. https://doi.org/10.9717/KMMS.2021.24.6.815
Weiser, M. (1996). Open House. Review, Web magazine of the Interactive Telecommunications Program of New York University.
Wertheimer, M. (1923). Untersuchungen zur Lehre von der Gestalt. II. Psychol. Forsch., 4, 301–350. https://doi.org/10.1007/BF00410640
Zimmerman, J., Stolterman, E., & Forlizzi, J. (2010). An analysis and critique of research through design: Towards a formalization of a research approach. In Proceedings of the 8th ACM Conference on Designing Interactive Systems, 310–319. https://doi.org/10.1145/1858171.1858228
