About Joan Mora Guiard

Dr. Mora Guiard is a lead developer at Inflight VR and researcher in the field of Virtual Reality. He holds a PhD from the Department of Information and Communication Technologies in Universitat Pompeu Fabra, Barcelona. His research focused on designing, developing and evaluating Embodied Interaction Virtual Reality Systems as intervention tools for children with Autism Spectrum Disorders.

The interview

Vanessa Zervogianni (VZ): What are the advantages of  full-body interaction media in autism treatment as opposed to more “traditional” technology platforms e.g. mere desktop applications? Can you give us an example from the “Lands of Fog” virtual reality system?

Joan Mora Guiard (JMG): I focused my research in full-body interaction, understanding it as a subset of embodied interaction, because embodied cognition theories argue that cognitive processes (e.g. learning or communication) are mediated by the body, more precisely, by the fact that our body is situated (i.e. actively embedded in its surrounding which is populated by other people and objects). Thus, I strongly believe that full-body interaction systems, as a myriad of technologies that take advantage of the mediation of the body within physical and virtual space, provide a greater potential for the development of virtual environments geared towards intervention.

My approach to using full-body interaction technology for the development of tools for intervention is by focusing on the development of play spaces that help practitioners during treatment by motivating and promoting social behaviours of the final users.

There is scientific evidence that full-body interaction is a suitable interactive paradigm to develop virtual environment systems for learning different skills, especially social and communication skills. Full-body interaction allows for the development of more natural and less physically constrained interfaces for the users. Given that this approach focuses on understanding user’s whole body behaviour, from an interface perceptual point, it is possible to develop systems which go beyond fine motor sensory activity, allowing users to actively engage with the environment with their whole body.

Let’s imagine that we want to build a co-located multi-user environment, which would be an interesting physical setting for developing a system to learn and practice social and communication skills. If we developed the system with a classic desktop approach, most probably users would be seated in chairs, maybe sharing the desk, but all steady seated in front of a computer and with their hands on the keyboard and mouse. There is always the freedom to move your hands and stand up, but then as a user you are out of reach of the physical interface, which, in the end, is the only means you have to interact with the virtual environment. In full-body interactive systems, we can develop virtual interactive environments which allow for gesticulation freedom while never leaving the physical interface. This allows for the freedom of bodily behaviour which still is perceived by the system, and can be used both as an interactive mean, but moreover, as a way for tracking user behaviour which can later by analysed.

As an example, “Lands of Fog” is a full-body interactive system based on a large circular floor projection of 6 meters in diameter. The space is big enough to accommodate different play styles by the users, such as solitary behaviours, while still allowing for the user to keep playing with the system, always being tracked by the physical interface and being a co-located subject for other users. Users interact with the system by using butterfly nets, which are tracked by the four camera based tracking system. With them, they can capture the digital fireflies that fly around the virtual environment. The interaction with the system feels natural, as through the use of cultural forms (i.e. a butterfly net) we help children understand the interactive affordances of the system.

As the interactive space is large, users have their own space and can avoid the feeling of pressure to interact with their peer(s) if they don’t want. If a user wants, they can keep playing by themselves and still discover different elements of the experience. The design of the system motivates exploratory behaviours which can lead to serendipitous encounters with other peers. These encounters feel natural, partially given there are no physical body constraints imposed by chairs, desks or small interactive surfaces, but because digital elements smartly bring children together.

One could argue that this freedom might hinder the therapeutic potential of the installation. But, with a proper interactive design, it is possible to develop intervention tools that are useful without forcing the user to always do what you planned as a designer. Our approach for the design of “Lands of Fog” was to encourage collaboration, which means to design a game that promotes collaboration, while still allowing for solitary play. Through this approach, we avoid frustrating the users, as they can keep playing with the system whether they collaborate or not. Yet, only through collaboration, they will be able to discover all the secrets of the system.

VZ: How can technology help foster social behaviours and collaboration among children with autism? Can you give us a few examples of pieces of technology that help achieve that goal?

JMG: I think that the main advantage of digital technology is that it allows for the design of engaging co-located multi-user environments which can adapt themselves so as to continuously provide a space to scaffold social behaviours and collaboration between users.

In play therapy, therapists build communication with patients through modulating play activities towards patients’ interests. Digital solutions, in the form of interactive screens, projections or even physical tools which have no digital displays, can also be the ground for a play space between therapists and children with autism. Nonetheless, as I stated previously, digitally based solutions have a really important advantage: they can modulate themselves given users input. Technology-based solutions are usually based on interactive systems, whose primary goal is to adapt to user’s behaviour and give an immediate response to their input. This means they are systems which always monitor user’s behaviour through the physical interfaces. This not only allows for this system self-modulation, but has also the advantage of making the play system itself responsible of the tracking of user behaviour, information which can later be used to analyse users’ progress throughout sessions. Finally, the tracking of users can be also used to understand who is doing what, so not only the system can adapt itself to overall users interaction, but can also adapt itself specifically to each user. As a result, a unique overall experience can behave slightly differently for each user and the therapy can therefore be better adapted to each user’s needs.

In my opinion technological solutions should always be developed with the view to developing tools that help caregivers, therapists, psychologists, practitioners in general, but not as a substitution of human-mediated intervention. With this in mind, I always focused my research on the development of multi-user play spaces that would foster behaviours such as social initiation and collaboration through the design of unique novel experiences and collaborative scenarios.

One example on how this can be achieved comes from one of the installations we did in Universitat Pompeu Fabra called “Pico’s Adventure”. In this Kinect-based game children with autism help an alien, named Pico, who is stranded in Earth, to reassemble his spaceship, so he can return to his planet and help his friends. During the whole game, children play in different partner styles; alone, with their parents and with another child they meet for the first time. During the first session with their parents, children have to help Pico find all his spaceship parts. We designed an enforced collaboration scenario where some of the pieces could not be reached by the child, but only by their parents who are physically taller. By controlling who was the shorter player and who was the taller, thanks to the Kinect technology, the game forced the children to ask their parents for help to achieve the game goal of helping Pico. With a non-digital device, this kind of scenarios would have to be heavily mediated by therapists. Nonetheless, thanks to full-body interaction, physical interfaces, we could seamlessly embed these kind of play constraints in the interactive experience.

Another example would be the aforementioned “Lands of Fog” system. In “Lands of Fog” we followed an encouraged collaboration approach. Thus, children could play alone with the system, but only through collaboration they could discover all the secrets hidden in the game. A precise example is that by playing alone, children could discover one of the companion creatures of the game. If players kept playing alone, they could discover variations of this same creature (i.e. changes on texture and small geometry details of the model). But only by sharing their creature with their peers by joining them, they could discover different creatures. Analysis of users’ behaviour indicated that the collaborative mechanics of the game fostered a significantly increasing amount of social and collaborative behaviours as sessions progressed, although children with autism played with a different partner every time.

VZ: What is the importance of participatory design with autistic people when designing a technological intervention?

JMG: As a designer, only through participatory design I can better understand the requirements specific to autistic people, which are the ultimate users of the system. By including them as informants during the design process, they contribute to the design by explaining what they like, expect, understand from it and their needs. Moreover, their unique way of experiencing the world brings a new perspective to the interactive design that a team only composed by neurotypical participants could not have. During “Pico’s Adventure” design, led by Dr. Laura Malinverni, we saw that participatory design was not only a process to give voice to the final users, but also a way to empower them. The four children we worked with involved themselves a lot in the design of “Pico’s Adventure” and were extremely willing to know which were the outcomes of their design, whether final users liked it or not and even whether it was usable.

Our design processes always involved autistic children as early as possible. Right after the therapeutic goals elicitation process was complete and an initial experimental design was planned according to the therapeutic requirements, participatory design sessions started. In some specific cases, such as “Pico’s Adventure”, the participatory design outcomes lead to a re-design of nearly half of the initial design experiment. Through different approaches, such as quickly prototyping and mapping children ideas, we could quickly confirm whether interactive scenarios would be understandable and valid to achieve our goals.

VZ: How can a multimodal approach facilitate the analysis of participants’ contributions in a participatory design context?

JMG: A multimodal approach to the analysis of contributions helps to shift the view from a more limited and classical approach of participatory design, typically based on speaking, drawing or sculpture, to a more embodied process. If embodied cognition theorises that cognitive processes are situated, i.e. based on the active embodied relation with the objects and other subjects in the environment, a multimodal approach seems more suitable for the analysis of all potential contributions of participants and their experiential relation with the design. More specifically, when working with full-body interaction, it is necessary to think of design scenarios that are attuned to the uniqueness of the interactive media in practice. By using methods to analyse a wider range of resources employed by people to construct meaning (i.e. gestures, gaze, movements, etc.), it is possible to extend the analysis beyond the intentional communication prompts, thus overcoming constraints which could hinder communication and expression of autistic people.

During participatory design we have always worked with a wide variety of tools that ranged from low-fidelity cardboard prototypes of the interactive environment, to role-playing of both the game characters and the final interactive scenarios. This allowed to cover not only the design from a content point of view, but also from an embodied standpoint, more suitable for full-body interaction experiences. Advantages of full-body interaction include greater protagonism of the body, situatedness and gesticulation during the interactive process. That is why it is essential to device a wider range of design scenarios that allow for a more complex analysis of participants’ contributions. Thus a multimodal approach does not necessarily facilitate the analysis of participants’ contributions, but it allows designers to have unique insights of these contributions.

More about Dr Mora Guiard

Personal website
LinkedIn profile
Twitter