Where Digital Meets Physical: A Conceptual Framework for Exploring Aesthetic Potential of Mixed Reality Interface

Keywords: aesthetic potential, interface studies, HCI, GUI, mixed reality

Abstract

This paper aims to conceptually explore the aesthetic potential phenomenon of a mixed reality interface. The research is intended to create a novel conceptual framework that includes characteristics of mixed reality interface and approaches to studying the aesthetic potential phenomenon. The framework is based on the synthesis of knowledge and integration of human-computer interaction concepts and methodologies from design, philosophy, psychology, and media culture. This paper offers a new expanded view of the aesthetic potential phenomenon. This includes seeing the aesthetic potential in the way the mixed reality interface manifests itself as a meaningful part of our lifeworld. Furthermore, the next research steps are outlined under a new perspective of studying the interface and its future.

Downloads

Download data is not yet available.

Author Biography

Daniil Igorevich Solosyatov, National Research University Higher School of Economics

Student of the Doctoral Programme “Fine and decorative arts and architecture”

References

Arnheim, R. (2004). Art and Visual Perception: A Psychology of the Creative Eye. University of California Press.

Baudrillard, J. (1994). Simulacra and Simulation. University of Michigan Press.

Berning, M., Kleinert, D., Riedel T., & Beigl M. (2014). A study of depth perception in hand-held augmented reality using autostereoscopic displays. IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 93–98. https://doi.org/10.1109/ISMAR.2014.6948413

Bertelsen, O. W. (2004). Transparency by Tertiary Artefactness. In Workshop on Aesthetic Approaches to Human-Computer Interaction, DAIMI Report Series, 33(572), 1–4. https://doi.org/10.7146/dpb.v33i572.7129

Bleecker, J. (2009) Design Fiction: A short essay on design, science, fact and fiction. Near Future Laboratory.

Blythe, M. (2014). Research through design fiction. Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems – CHI ’14. https://doi.org/10.1145/2556288.2557098

Carroll, J. M. (2010). Conceptualizing a possible discipline of human–computer interaction. Interacting with Computers, 22(1), 3–12. https://doi.org/10.1016/j.intcom.2009.11.0

Chen, S., & Duh, H. (2019). Interface of mixed reality: from the past to the future. CCF Trans. Pervasive Comp. Interact., 1, 69–87. https://doi.org/10.1007/s42486-018-0002-8

Christensen, M. S. (2004). Introducing Excitability! In Workshop on Aesthetic Approaches to Human-Computer Interaction, DAIMI Report Series, 33(572), 10–13. https://doi.org/10.7146/dpb.v33i572.7129

Chuah, S. H-W. (2019). Wearable XR-technology: literature review, conceptual framework and future research directions. International Journal of Technology Marketing, 13(3/4), 205–259. https://doi.org/10.1504/IJTMKT.2019.104586

Costanza, E., Kunz, A., & Fjeld, M. (2009). Mixed Reality: A Survey. In Lalanne, D., & Kohlas, J. (Eds.), Human Machine Interaction, 47–68. Springer https://doi.org/10.1007/978-3-642-00437-7_3

Coulton, P., Burnett, D., and Gradinar, A. (2016). Games as Speculative Design: Allowing Players to Consider Alternate Presents and Plausible Features. In Lloyd, P. & Bohemia, E. (Eds.), Future Focused Thinking – DRS International Conference 2016. https://doi.org/10.21606/drs.2016.15

Coulton, P., & Hook, A. (2017). Games design research through game design practice. In Lankoski, P., & Holopainen J. (Eds.), Game Design Research: An Introduction to Theory and Practice, 97–116. ETC Press.

Delbridge, R., & Fiss, P. C. (2013). Editors’ comments: Styles of theorizing and the social organization of knowledge. Academy of Management Review, 38(3), 325–331. https://doi.org/10.5465/amr.2013.0085

Dourish, P., & Bell, G. (2013). “Resistance is futile”: reading science fiction alongside ubiquitous computing. Personal and Ubiquitous Computing, 18(4), 769–778. https://doi.org/10.1007/s00779-013-0678-7

Dunne, A. (1999). Hertzian Tales: Electronic Products, Aesthetic Experience, and Critical Design. RCA CRD Research

El Jamiy, F., & Marsh, R. (2019). Survey on depth perception in head mounted displays: distance estimation in virtual reality, augmented reality, and mixed reality. IET Image Processing, 13, 707-712. https://doi.org/10.1049/iet-ipr.2018.5920

Ermi, L., & Mäyrä, F. (2005). Player-Centered Game Design: Experiences in Using Scenario Study to Inform Mobile Game Design. Game Stud., 5.

Figueiredo L.S., Pinheiro M., Neto E.V., Chaves T., & Teichrieb V. (2015). Sci-Fi Gestures Catalog. In Abascal J., Barbosa S., Fetter M., Gross T., Palanque P., Winckler M. (Eds.), Human-Computer Interaction – INTERACT 2015. INTERACT 2015. Lecture Notes in Computer Science, vol 9297. Springer, Cham. https://doi.org/10.1007/978-3-319-22668-2_30

Gray, B. (1950). The Social Effects of the Film. The Sociological Review, a42(1), 135–144. https://doi.org/10.1111/j.1467-954x.1950.tb02467.x

Guberman, S. (2017). Gestalt Theory Rearranged: Back to Wertheimer. Frontiers in Psychology, 8. https://doi.org/10.3389/fpsyg.2017.01782

Guerrilla Games. (2017). Horizon Zero Dawn (PC version) [Video game]. Sony Interactive Entertainment.

Hallnäs, L., & Redström, J. (2001). Slow Technology – Designing for Reflection. Personal and Ubiquitous Computing, 5(3), 201–212. https://doi.org/10.1007/pl00000019

Hallnäs, L., & Redström, J. (2002). From Use to Presence: On the Expression and Aesthetics of Everyday Computational Things. ACM Transaction on Computer-Human Interaction, 9(2), 106-124. https://doi.org/10.1145/513665.513668

Hammady, R., Strathearn, C., & Ma, M. (2019). User Experience Design for Mixed Reality: A Case Study of HoloLens in Museum. International Journal of Technology Marketing. https://doi.org/10.1504/ijtmkt.2019.10025609

Hammel, M. (2004). The aesthetics of Use. In Workshop on Aesthetic Approaches to Human-Computer Interaction, DAIMI Report Series, 33(572), 17–19. https://doi.org/10.7146/dpb.v33i572.7129

Hartmann, J., Sutcliffe, A., & De Angeli, A. (2008). Towards a theory of user judgment of aesthetics and user interface quality. ACM Transactions on Computer-Human Interaction, 15(4), 1–30. https://doi.org/10.1145/1460355.1460357

Heinrich, F., Bornemann, K., Lawonn, K., & Hansen, C. (2019). Depth Perception in Projective Augmented Reality: An Evaluation of Advanced Visualization Techniques. 25th ACM Symposium on Virtual Reality Software and Technology (VRST ’19), 26, 1–11. https://doi.org/10.1145/3359996.3364245

Hirschheim, R. (2008). Some guidelines for the critical reviewing of conceptual papers. Journal of the Association for Information Systems, 9(8), 432–441. https://aisel.aisnet.org/jais/vol9/iss8/21

Hochheiser, H., & Lazar, J. (2007). HCI and Societal Issues: A Framework for Engagement. International Journal of Human-Computer Interaction, 23(3), 339–374. https://doi.org/10.1080/10447310701702717

Jaakkola, E. (2020). Designing conceptual articles: four approaches. AMS Rev 10, 18–26. https://doi.org/10.1007/s13162-020-00161-0

Johnson, B. D. (2011a). Love and God and Robots: The Science Behind the Science Fiction Prototype “Machinery of LoveGrace”. In Workshop Proceedings of the 7th International Conference on Intelligent Environments, 10, 99–127. https://doi.org/10.3233/978-1-60750-795-6-99

Johnson, B. D. (2011b). Science Fiction Prototyping: Designing the Future with Science Fiction. Synthesis Lectures on Computer Science, 3(1), 1–190. https://doi.org/10.2200/S00336ED1V01Y201102CSL003

Jordan, P. (2002). Human factors for pleasure seekers. In Frascara, J., Design and the social science making new connections. London: Taylor & Francis.

Johnson, J. (2020). Designing with the mind in mind. Burlington: Elsevier.

Interface Studies. (2016, March 15). http://www.interfacestudies.org/

Iso, M. (Writer & Director). (2007). Dennou Coil [TV series]. Tokuma Shoten; Bandai Visual; NHK Enterprises.

Kim, H., Lee, J., Yeo, H.-S., Quigley, A., & Woo, W. (2018). SWAG Demo: Smart Watch Assisted Gesture Interaction for Mixed Reality Head-Mounted Displays. 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). https://doi.org/10.1109/ismar-adjunct.2018.00130

Kim, D.-H., Go, Y.-G., Choi, S.-M. (2020). An Aerial Mixed-Reality Environment for First-Person-View Drone Flying. Applied Sciences, 10(16). https://doi.org/10.3390/app10165436

Kirby, D. (2009). The Future is Now. Social Studies of Science, 40(1), 41–70. https://doi.org/10.1177/0306312709338325

Knoller, N., & Ben-Arie, U. (2015). The Holodeck is all Around Us – Interface Dispositifs in Interactive Digital Storytelling. In Koenitz, H., Ferri, G., Haahr, M., Sezen, D., & Sezen, T.İ. (Eds.), Interactive Digital Narrative: History, Theory and Practice. Routledge. https://doi.org/10.4324/9781315769189-5

Kymäläinen, T. (2016). Science Fiction Prototypes as a Method for Discussing Socio-Technical Issues within Emerging Technology Research and Foresight. Athens Journal of Technology and Engineering, 3(4), 333–347.

Lavie, T., & Tractinsky, N. (2004). Assessing dimensions of perceived visual aesthetics of websites. International Journal of Human-Computer Studies, 60(3), 269–298. https://doi.org/10.1016/j.ijhcs.2003.09.002

Lee, R. M., Fielding, N., & Blank, G. (2008). The Internet as a research medium: An editorial introduction to the Sage handbook of online research methods. In Fielding, N., Lee, R. M., & Blank, G. (Eds.), The Sage handbook of online research methods, 3–20. SAGE Publications. https://doi.org/10.4135/9780857020055.n1

Lee, T., & Hollerer, T. (2007). Handy AR: Markerless Inspection of Augmented Reality Objects Using Fingertip Tracking. 11th IEEE International Symposium on Wearable Computers. https://doi.org/10.1109/iswc.2007.4373785

Lim, S. G., & Kim, C. (2018). A Study on the 21st Century Digital Visuality Through Lacan’s Notion of the Real Gaze – From an Aspect of Digital Frame Expension. Journal of Korea Multimedia Society, 21(5), 638–647. https://doi.org/10.9717/KMMS.2018.21.5.638

Ngo, D., Teo, L. S., & Byrne, J. G. (2003). Modeling interface aesthetics. Information Sciences, 152(1), 25-46. https://doi.org/10.1016/S0020-0255(02)00404-8

MacInnis, D. J. (2011). A framework for conceptual contributions in marketing. Journal of Marketing, 75(4), 136–154. https://doi.org/10.1509/jmkg.75.4.136

Marcus, A., Norman, D. A., Rucker, R., Sterling, B., & Vinge, V. (1992). Sci-fi at CHI. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’92. https://doi.org/10.1145/142750.142892

Marcus, A., Soloway, E., Sterling, B., Swanwick, M., & Vinge, V. (1999). Opening plenary: sci-fi @ CHI-99. Extended Abstracts on Human Factors in Computing Systems – CHI ’99. https://doi.org/10.1145/632716.632775

Marcus, A. (2013). The history of the future. Interactions, 20(4), 64. https://doi.org/10.1145/2486227.2486240

Merino, L., Schwarzl, M., Kraus, M., Sedlmair, M., Schmalstieg D., & Weiskopf, D. (2020). Evaluating Mixed and Augmented Reality: A Systematic Literature Review (2009–2019). 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 438–451. https://doi.org/10.1109/ISMAR50242.2020.00069

Milgram, P., Takemura, H., Utsumi, A., & Kishino, F. (1995). Augmented reality: a class of displays on the reality-virtuality continuum. Telemanipulator and Telepresence Technologies. https://doi.org/10.1117/12.197321

Montfort, N. (2017). The Future. MIT Press.

Moshagen, M., & Thielsch, M. T. (2010). Facets of visual aesthetics. International Journal of Human-Computer Studies, 68(10), 689–709. https://doi.org/10.1016/j.ijhcs.2010.05.006

Mubin, O., Obaid, M., Jordan, P., Alves-Oliveria, P., Eriksson, T., Barendregt, W., Sjolle, D., Fjeld, M., Simoff, S., & Billinghurst, M. (2016). Towards an Agenda for Sci-Fi Inspired HCI Research. Proceedings of the 13th International Conference on Advances in Computer Entertainment Technology. https://doi.org/10.1145/3001773.3001786

Perez Mata, M. & Ahmed-Kristensen, S. (2015). Principles for Designing for Perception. In Proceedings of the 20th International Conference on Engineering Design (ICED 15), 9, 239–248.

Pham, B. (1999). Design for aesthetics: interactions of design variables and aesthetic properties. In Proceedings of SPIE IS&T/SPIE 11th Annual Symposium (Electronic Imaging '99), 3644, 364–371. https://doi.org/10.1117/12.348457

Pinhanez, C. (2001). Augmenting reality with projected interactive displays. In Proceedings of the International Symposium on Virtual and Augmented Architecture (VAA’01), 93–100. Springer, London. https://doi.org/10.1007/978-1-4471-0337-0_9

Raskar, R., van Baar, J., Beardsley, P., Willwacher, T., Rao, S., & Forlines, C. (2006). iLamps: geometrically aware and self-configuring projectors. ACM SIGGRAPH 2006 Courses on – SIGGRAPH’06. https://doi.org/10.1145/1185657.1185802

Rauschnabel, P. A. (2018). Virtually enhancing the real world with holograms: An exploration of expected gratifications of using augmented reality smart glasses. Psychology & Marketing, 35(8), 557–572. https://doi.org/10.1002/mar.21106

Rosenberg, L. B. (1993). Virtual fixtures: Perceptual tools for telerobotic manipulation. Proceedings of IEEE Virtual Reality Annual International Symposium, 76–82, https://doi.org/10.1109/VRAIS.1993.380795

Soegaard, M. (2015). Gestalt principles of form perception. In Papantoniou, B., Soegaard, M., Lupton, J. R., Goktürk, M., Trepess, D., Knemeyer, D., Svoboda, E., Memmel, T., Folmer, E., Gunes, H., Harrod, M., Spillers, F., Hornecker, E. (eds). The Glossary of Human Computer Interaction. The Interaction Design Foundation.

Sonderegger, A., & Sauer, J. (2010). The influence of design aesthetics in usability testing: Effects on user performance and perceived usability. Applied ergonomics, 41(3), 403–410. https://doi.org/10.1016/j.apergo.2009.09.002

Schmidt, S., Bruder, G., & Steinicke F. (2020). Depth Perception and Manipulation in Projection-Based Spatial Augmented Reality. PRESENCE: Virtual and Augmented Reality, 27(2), 242–256. https://doi.org/10.1162/pres_a_00329

Schmitz, M., Endres, C., & Butz, A. (2008). A Survey of Human-Computer Interaction Design in science fiction movies. In Proceedings of the 2nd international conference on INtelligent TEchnologies for interactive enterTAINment (INTETAIN '08). https://doi.org/10.4108/ICST.INTETAIN2008.2476

Speicher, M., Hall, B. D., & Nebeling, M. (2019). What is Mixed Reality? Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems – CHI’19. https://doi.org/10.1145/3290605.3300767

Tanenbaum, J. (2014). Design fictional interactions. Interactions, 21(5), 22–23. https://doi.org/10.1145/2648414

Thüring, M., & Mahlke, S. (2007). Usability, aesthetics and emotions in human-technology interaction. International Journal of Psychology, 42(4), 253–264. https://doi.org/10.1080/00207590701396674

Terashima, N. (2005). The definition of hyperreality. In John Tiffin & Nobuyoshi Terashima (org.), Hyperreality – Paradigm for the Third Millennium, 4–24. Routledge.

Tiffin, J. (2005). The hyperreality paradigm. In John Tiffin & Nobuyoshi Terashima (org.), Hyperreality – Paradigm for the Third Millennium, 25–42. Routledge.

Tschimmel, K. (2011). Design as a Perception-in-Action Prozess. In Taura, T., & Nagai, Y., (Eds.), Design Creativity 2010, 223–230. Springer Verlag. https://doi.org/10.1007/978-0-85729-224-7_29

Udsen, L., & Jørgensen, A-H. (2005). The aesthetic turn: Unraveling recent aesthetic approaches to human-computer interaction. Digital Creativity, 16(4), 205-216. https://doi.org/10.1080/14626260500476564

Unberath, M., Yu, K., Barmaki, R., Johnson, A., & Navab, N. (2020). Augment Yourself: Mixed Reality Self-Augmentation Using Optical See-through Head-mounted Displays and Physical Mirrors. ArXiv. https://doi.org/10.48550/arXiv.2007.02884

Wagemans, J., Elder, J. H., Kubovy, M., Palmer, S. E., Peterson, M. A., Singh, M., & von der Heydt, R. (2012). A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization. Psychological Bulletin, 138(6), 1172–1217. https://doi.org/10.1037/a0029333

Wei, L., & Cho, D.-M. (2021). Research on Design of Mixed Reality Interface Based on Spatial Perception. Journal of Korea Multimedia Society, 24(6), 815–824. https://doi.org/10.9717/KMMS.2021.24.6.815

Weiser, M. (1996). Open House. Review, Web magazine of the Interactive Telecommunications Program of New York University.

Wertheimer, M. (1923). Untersuchungen zur Lehre von der Gestalt. II. Psychol. Forsch., 4, 301–350. https://doi.org/10.1007/BF00410640

Zimmerman, J., Stolterman, E., & Forlizzi, J. (2010). An analysis and critique of research through design: Towards a formalization of a research approach. In Proceedings of the 8th ACM Conference on Designing Interactive Systems, 310–319. https://doi.org/10.1145/1858171.1858228

Published
2022-10-10
How to Cite
SolosyatovD. I. (2022). Where Digital Meets Physical: A Conceptual Framework for Exploring Aesthetic Potential of Mixed Reality Interface. Communications. Media. Design, 7(3), 159-185. Retrieved from https://cmd-journal.hse.ru/article/view/16121
Section
Scientific Articles