RehabCafe

Design of a VR environment for cognitive rehabilitation

Preview

View of the final VR environment produced

Brief

Purpose:
MSc Human-Computer Interaction dissertation

Team Size:
1 person

Project Length:
July 2019 – September 2019
( 2 months )

Target Platform:
Fove (VR headset)

Role:
UX/UI Designer, Researcher, Software Engineer, Usability Testing

Techniques Involved:
Focus Group, Storyboarding, Wireframing, Prototyping, Pilot Study, User Interviews, Mixed-Methods Analysis


Summary

For my Masters dissertation, I was interested in using VR and how I can expand its use in psychology. As this was an individual project, I was in-charge of the end-to-end process from research to design to prototype development and testing. By the end of my dissertation, I delivered a prototype made in Unity and a paper of about 20,000 words for submission. My supervisor helped to guide the scope of the project and made sure I was able to produce a minimum viable product by reviewing my designs.

After submitting my project, I obtained a First Class Honours ๐Ÿ† for the project. My solution was also shared with my supervisor, lecturers involved, and the clinical psychologists who gave insights in the project. Later students can also choose to refine my work if they are interested in the field.

Features

Semi-autonomous dialogue
Psychologists can semi-monitor behaviours of their patients.

Client-facing Role
To help psychologists draw emotions from their patients and to train patients confidence for getting back to jobs.

Gamification of sessions
To motivate patients to continue their rehabilitation and monitor their own performance


Problem Statement

In the United Kingdom, at least 1 person is admitted to hospital for ABI every 90 seconds.

Headway (2017)

Acquired Brain Injuries (ABI) are known as one of the most common causes of death and disability in children and adults.

With the number of patients with ABI increasing in the UK every year, there is a need to come up with an alternative tool to current rehabilitation strategies.

What’s the problem?

60% of prison inmates have a history of traumatic brain injuries.

Williams et al. (2018)

A patient with Acquired Brain Injuries (ABI) not only suffers from disabilities and impairments to their cognition and other functions of their brain. They are also likely to display poor social behaviour and quality of life. Due to their condition, they are more likely to repeat criminal offences and engage in anti-social behaviour. This can also cause distress to their families and caregivers.

  • Issues with current rehabilitation strategies for ABI:
    • high risk for ABI patients injuring themselves and those around them
    • limited resources and high maintenance cost in terms of time, money, and personnel
    • limited assessment of executive functions
    • lack of awareness in patients about their performance (it is common for patients to think that they are not affected by their injury when they are heavily disabled)

Proposed Solution

In RehabCafe, clinical psychologists can provide a safe and controlled environment for their patients in an immersive VR environment. For patients, they are able to prepare themselves for real world situations they may encounter due to their disability. With my solution, there is no need for real world physical space and possible enhance the effectiveness of the strategy with the use of VR. It is also possible to transfer the skills they have learnt into their daily life and help them get back into the workplace.

Why VR?

There are several benefits of using a VR environment compared to current rehabilitation strategies that brings patients to real-world environments:

  • Safe – Reduces risk of damaging real world environment and injuring people
  • Cheap – Reduces staffing and time requirements
  • Customizable – Customizable for different levels of difficulty
  • Transferable – Imitates real world settings
  • Recordable – Recording of data to increase ecological validity to prove effectiveness of therapy
  • Replayable – Replayable sessions for patients to be self-aware of their performance and to practice their skills
High risk problem: Usually, a clinical psychologist will observe the patient’s behaviour in the real world settings. The patient will have one or two caregivers alongside to help them manage their emotions. Sometimes, a dispute may occur between the patient and a stranger. It’s also not fair to the stranger as they did not volunteer to be part of the assessment.
Cost efficient solution: With VR rehabilitation as a solution, it helps the patient to not hurt others even without caregivers around. The clinical psychologists can also observe better and compare their performance with previous sessions. Making it replayable also helps the patient to know their current performance better.

Design Process

In this project, I produced 4 design iterations/revisions. The prototype was meant to be evaluated by clinical psychologists to validate the efficiency of the VR environment as a viable solution. The prototype produced was a VR environment used for usability testing.

Actual process taken

1๏ธโƒฃ Establishing requirements – Literature Review

Read, read, and read! ๐Ÿ“š

My first step was to understand the topic and themes in mental health to be able to understand the current situation, and be able to hold conversations. Through literature review by reading and analysing journal articles, I tried to understand:

  • What is ABI and what are its effects?
  • What are the current methods used to treat ABI?
  • What are the limitations of using current methods?
  • How are the methods derived from? What are the foundation theories of these rehabilitation strategies?
  • Have VR technology been used in clinical psychology before?
  • What were their challenges and achievements?
  • What is the optimum duration of a VR session?
  • How can I produce the VR environment from the technical aspect?

2๏ธโƒฃ Designing alternatives – Iteration 1

Setting: Streets of a virtual city

Core features: Real-time data visualisation and social interactions

Let’s dream big! ๐Ÿ’ญ

For my first iteration, I thought to go as far as I can so that I can scope it down from there. I went for a big ambition to incorporate the Multiple Errands Task into the environment. Besides having tasks itself, there was an opportunity for a real-time data visualisation that would unleash full potential of VR technology. This would allow clinical psychologists to not just monitor their patients’ performance in real-time, but also be able to record the performance, and review it at a later time.

As there was a need to schedule a meeting with the clinical psychologists, I sketched a rough idea of the VR environment to possibly become a topic for discussion.

Rough sketch of idea ๐Ÿ“

I thought that perhaps 2 different screens would be required for the 2 different groups of users. The one for the patient would be within the VR space. The one for the clinical psychologists would be on a 2D space separate from the patient. This is so that the clinical psychologists can make sure the patients are safe from physical objects in their surroundings, and also be able to observe and manipulate the VR environment.

Sketch storming: Initial sketch of what both user groups would see
View Details

3๏ธโƒฃ Evaluating design – Iteration 1

To evaluate my designs, I approached my supervisor whose responsibility was to guide the scope of my supervisor and acts as a first point of contact for feedback. It was also part of the discussions during the focus group with the clinical psychologists in the next step.

Opportunities:

  • real-time monitoring of performance
  • full control over what avatars in the environment would say

Shortcomings:

  • scope of project too wide (in terms of scale of environment and features)
  • clinical psychologists having to oversee too many things at once

4๏ธโƒฃ Establishing requirements – Focus group

Informal discussion with the experts ๐Ÿ’ฌ

View Details

Summary of findings ๐Ÿ”

  • The VR environment will need memory assistance aids or movement aids due to disabilities caused by ABI
  • There is a need for clinical evaluation to ensure the ethicality in recording of data is ensured and making sure that itโ€™s a safe and effective environment
  • Current rehabilitative strategies are unable to assess the emotion factor of decision-making as it is unpredictable in real world settings, endangering those around the patient. Having a VR environment will reduce such risks
  • Although motion sickness is common in immersive VR settings, it was rarely reported in the test sample despite the patients’ brain injuries
  • Having distractions and unexpected social scenarios in the environment would help clinical psychologists assess the executive functions of their patients

What now?

From there, I was able to come up with a list of requirements for 2 user groups. This list will be the core backbone of what the VR environment has to include.

Clinical psychologists

  • Able to perform a series of tasks that access different skillsets of patients
  • Able to communicate rules and changes if they occur
  • Able to partially control conversations that will take place between a virtual avatar and the patient
  • Able to adjust the amount of distractions on the patient
  • Able to monitor how the client reacts to different situations
  • Able to checklist if the given tasks were completed
  • Able to report to the patients about their performance

Patients

  • Able to experience VR without feeling motion sickness
  • Able to navigate and interact in VR without hindrance from their disabilities (such as if they cannot walk)
  • Able to remember how to navigate or interact in VR (some patients have dementia and will not remember)
  • Able to understand what their tasks are
  • Able to have natural social interactions with the virtual avatar and display emotions like how they would in real life
  • Able to identify errors they made (some patients may be denial to their underperformance because of their brain injury)

In contrast to the common use of personas, I used a list of requirements as I would want to avoid bias when it is such an important tool being used by professionals and those invulnerable. Another reason I didn’t construct personas was to save time. I would later need to develop the environment under a short amount of time. Being able to incorporate these requirements into the environment quickly to generate usability test results is more important than the presentable artefacts for my dissertation.


5๏ธโƒฃ Designing alternatives – Iteration 2

Setting: Streets of virtual city

Core features: controls (interactions with VR environment), cut-scene introduction, distractions, virtual handheld device for social interactions and object interactions

Changes from previous iteration:

โ€“ real-time data visualisation was removed from the scope of the project (due to time constraints)
โ€“ consideration of motion sickness issues and immersive experience in VR
โ€“ more details on avatar interactions (how the patient will interact with an avatar and how the conversation will be carried out)

Similar with previous iteration:

โ€“ same social and cognitive tasks to be completed by patients

User flow ๐Ÿ–Š๏ธ

After the focus group with the clinical psychologists and university professors, I had a better idea of how they usually carry out a rehabilitation session. Using those details, I created a user flow for the sequence of actions that will be carried out by both the patients and the clinical psychologists.

User flow: Flow of a rehabilitation session for both the patient and clinical psychologist

Tackling challenges ๐Ÿงฉ

Two major challenges were also brought up during the discussion. The first was tackling motion sickness as it is the most common issue in VR environments in general. The second was to create a believable environment for the patients so that the patients can have natural interactions in VR.

Dealing with motion sickness

From the initial literature review, I found out motion sickness comes from the misalignment between the visual senses and the feeling of movement. Although it was not an issue from previous experiments that my professors conducted, the clinical psychologists were concerned during the focus group session. This is because motion sickness can cause discomfort and trigger spasms in the patients. To deal with motion sickness, I had the patients start in a sitting position, and have their avatar seated at the start of the VR environment as well. This will help avoid motion sickness and save them from hazards such as tripping onto cables from the setup. From the technical side, I also had to reduce the graphical needs of the VR environment to prevent “lag” in the system. If lag occurs, there is more likely to be a misalignment with what they see and their in-VR movement, and causes motion sickness.

Creating a believable environment

I wanted to create a believable VR environment so that the patients can interact naturally and take the activity seriously. To create a believable environment, I thought to give patients context of ‘why they are there’ so it gives them a motivation and reason to carry out the tasks later on. This also resembles real life situations of when they have to go out alone where they will have specific goals to reach.

Storyboarding ๐Ÿ’ญ

With the user flow and list of requirements generated, I pictured how a rehabilitation session will be carried out and what kind of scenarios will be in the VR environment.

Storyboarding: How a rehabilitation session will be carried out in a room – patient starts with a sitting position to minimize motion sickness; clinical psychologist observes what the patient sees in the first monitor and manipulates the VR environment in the second monitor

For the patient, they will be asked to carry out some tasks with some rules in place, and distractions to shift their attention from their tasks.

Due to ethical considerations, it was important to let patients know that they are free to exit the VR environment at any time. They also had to know about how their data is protected and other safety measures. I considered whether to have this ethical information within the VR environment or to be given before they enter the environment.

After considering that the clinical psychologists will be busy setting up the environment during this time, I decided to have the information within the VR environment so that the clinical psychologists will have less work and can set up the tasks in the VR environment.

Wireframes ๐Ÿ–ผ

After having a storyboard of what entails in the VR environment, I created wireframes for the screens clinical psychologists will use. These are to be placed in a second monitor.


6๏ธโƒฃ Evaluating design – Iteration 2

With this design, I asked for my supervisor’s advice again as well as asking other lecturers to provide feedback for what they think about the design. By giving them a quick run through of my study, they were able to identify the limitations of my environment and suggested to rescope my project due to the constraints of time.

Opportunities:

  • interfaces for therapists to observe and manipulate the VR environment for cognitive rehabilitation
  • having ways for both clinical psychologists and patients to interact with the VR world

Shortcomings:

  • long development time needed to build a virtual city with the tasks embedded and interactions with virtual avatars and the objects in environment
  • long navigation time
  • lack of framework for virtual avatars

7๏ธโƒฃ Establishing requirements – Literature Review

Looking at virtual avatars ๐Ÿค–

When I looked back at Iteration 2, it was not clear for how the social interactions between the patient and virtual avatars would work. As it plays a big part of assessment and recovery for the patients and would also help increase believability of the environment, I researched on how non-player characters (NPCs) are used in games and other applications based on my gaming experience and literature review on models of dialogues. After collating my research findings, I created a chart with three layers of a virtual avatar. With this framework, it will make virtual avatars seem more human and more believable.

Chart: Proposed framework of a virtual avatar containing three layers – Physical, Emotional, Social

Looking at social dialogue models ๐Ÿ’ฌ

For the content of what virtual avatars would say, I looked into models of typical conversations in people, known as social dialogues. I compared the different dialogues and used one that was easier to implement from the developer point of view. In the end, I adjusted a small talk model to incorporate impolite behaviour so that they can provoke the patients to display negative emotions. This is important for clinical psychologists to assess the patients’ behaviour. Although the model was of a high level, it was easier to comprehend and applicable for the communication between a virtual avatar and a patient.

Flow: How a small talk occurs in conversations
(Adapted from Treanor, McCoy and Sullivan 2015)

8๏ธโƒฃ Designing alternatives – Iteration 3

Setting: Cafe

Core features: client-facing role for patients, virtual avatars as customers in a cafe

Changes from previous iteration:

โ€“ change of settings from city to cafe
โ€“ change of tasks to do to fit in cafe
โ€“ addition of emotion wheel to clinical psychologist’s interface
โ€“ addition of an end-screen to clinical psychologist’s interface

Similar with previous iteration:

โ€“ same dual monitor setup and controls
โ€“ narrative cut-scene intro
โ€“ same foundations of virtual avatars

Storyboarding ๐Ÿ’ญ

The primary focus of this iteration was to reduce the size of the VR environment so that I can develop a prototype in time. Instead of using a virtual city, I decided to only develop one part of a virtual city. I decided to use a cafe because it is a common place for patients to visit, so it can be transferred to real life situations. Also, it is a place where people gather so patients can have social interactions.

View Details

9๏ธโƒฃ Evaluating design – Iteration 3

After discussions with my supervisor, the iteration was deemed to be of a good base of research as there was no research to date that puts a patient in a client-facing role. Furthermore, I was able to fit the requirements of both the clinical psychologists and patients within the new environment.

Opportunities:

  • new perspective of assessment with client-facing role of patients
  • fits cognitive and social tasks in the environment

Shortcomings:

  • lack of technical design for development

๐Ÿ”Ÿ Prototyping

With the shortcomings in Iteration 3, the technical design for development was done before producing the prototype. During the prototyping process however, a few problems came up. The whole process of prototyping took only 2 weeks for the functional environment so that I could do usability testing.

View Details

Results

To evaluate the design, participants were recruited for a pilot study that involved two parts. Part 1 involved carrying out tasks in the VR environment and Part 2 was an interview session to understand the participants’ background and opinion of the system built. Essentially, the pilot study sessions would help inform the usability of the controls, immersive experience (measured by in VR, and efficiency of the system to evoke natural

Healthy participants were recruited for this study instead of patients with ABI because patients with ABI were less accessible and more vulnerable to take the test at this stage of development. As such, the environment was adjusted to be more difficult for the participants such as not having a tutorial and were only told general rules of the environment.

By using transcriptions from interview data and observations in lab settings, the data was examined for features of interest and grouped by common themes. A mixed-methods approach was taken to interpret and present the results.

  • 50% of participants were successful at completing the tasks. Those who had experience with VR, or those who were less immersed in the VR environment, and those who did not find the controls to be difficult were more likely to be successful in the tasks.
  • 25% of participants were frustrated at the customers as they simply wanted to progress in the simulation. Other participants reported being frustrated at the environment and the controls of the system.
  • Participants found it more frustrating when a customer changed their order when they were completing a simple task.
  • Participants forget to do a task if the customer becomes annoyed.
  • Those who were more explorative in the environment was able to complete the tasks.
  • Many participants found it unnatural to talk-aloud to a virtual avatar knowing that they were physically situated in a room without anyone else beside the researcher.
  • 1 participant was showing unfriendly behaviour to the virtual customer knowing that it was not real
  • Some participants missed out on subtitles despite the subtitles being in their field of vision all the time
  • There was a lack of feedback in the environment so participants was not sure if they completed a task

Reflection

Challenges

What challenges did you face and how did you deal with them?

  • Data loss – During the usability testing stage, the screen recording did not save for half of the sessions. I found out there was not enough storage space in the disk drive which I was saving to. I did not have any other backups and the screen capture data was lost. There was nothing I could do except making sure the recordings would be saved for the later sessions.
  • Unavailable equipment and hardware problems – The initial designs were based on the HTC Vive, but the equipment was not available and I had to use FOVE. The main difference was that FOVE does not have controllers like the HTC Vive, so I had to develop workarounds that with the equipment I have. The only other interaction FOVE had was eye gazing, which I faced hardware problems during development. In the end, I resorted to using a gamepad to replace the original controllers while having a static “cursor” in the middle of the screen.
  • Limited access to patients and clinical psychologists – Due to the vulnerability of patients and nature of the job as clinical psychologists, I was not able to contact them as often as possible to clarify further questions that came up. I had to use my own judgment through my own background in psychology as well as understanding them more through academic journals.

Learnings

What did you learn?

  • Prepare for the unexpected – Things might not always turn out the way you expect, be ready to come up with alternatives and adapt to meet requirements within tight schedules.
  • Talk to people about your ideas – I learnt to talk through with others to get feedback and to strengthen the arguments for my ideas. Everyone have different perspectives and different experiences that can provide valuable opinions.
  • Everyone is a designer – From getting feedback from people around me and the getting suggestions from my usability tests, everyone was able to give a lot of good ideas I previously didn’t think of. It is good to keep an open mind and learn from others as everyone knows a bit of something more than yourself.

Different Approach

What would you do next time?

  • Take one step ahead – There were times when I was waiting for comments or feedback from my supervisor and it was delayed or postponed because of unexpected events. I felt like I should have just taken the step forward and do what I think see fit rather than waiting for the green light to continue on the work.
  • Confirm resources beforehand – Even though it was said that I would be able to get a hold of the specific VR environment at the start and also gain access to volunteers with ABI, I should have confirmed it before going ahead. Due to the different circumstances that came up that limited my resources to both the equipment and volunteers, I had to come up with a lot of last minute turnarounds, both in the design and development stages.