MSc Human-Computer Interaction dissertation
July 2019 – September 2019
( 2 months )
Fove (VR headset)
UX/UI Designer, Researcher, Software Engineer, Usability Testing
Focus Group, Storyboarding, Wireframing, Prototyping, Pilot Study, User Interviews, Mixed-Methods Analysis
For my Masters dissertation, I was interested in using VR and how I can expand its use in psychology. As this was an individual project, I was in-charge of the end-to-end process from research to design to prototype development and testing. By the end of my dissertation, I delivered a prototype made in Unity and a paper of about 20,000 words for submission. My supervisor helped to guide the scope of the project and made sure I was able to produce a minimum viable product by reviewing my designs.
After submitting my project, I obtained a First Class Honours 🏆 for the project. My solution was also shared with my supervisor, lecturers involved, and the clinical psychologists who gave insights in the project. Later students can also choose to refine my work if they are interested in the field.
Psychologists can semi-monitor behaviours of their patients.
To help psychologists draw emotions from their patients and to train patients confidence for getting back to jobs.
Gamification of sessions
To motivate patients to continue their rehabilitation and monitor their own performance
In the United Kingdom, at least 1 person is admitted to hospital for ABI every 90 seconds.Headway (2017)
Acquired Brain Injuries (ABI) are known as one of the most common causes of death and disability in children and adults.
With the number of patients with ABI increasing in the UK every year, there is a need to come up with an alternative tool to current rehabilitation strategies.
What’s the problem?
60% of prison inmates have a history of traumatic brain injuries.Williams et al. (2018)
A patient with Acquired Brain Injuries (ABI) not only suffers from disabilities and impairments to their cognition and other functions of their brain. They are also likely to display poor social behaviour and quality of life. Due to their condition, they are more likely to repeat criminal offences and engage in anti-social behaviour. This can also cause distress to their families and caregivers.
- Issues with current rehabilitation strategies for ABI:
- high risk for ABI patients injuring themselves and those around them
- limited resources and high maintenance cost in terms of time, money, and personnel
- limited assessment of executive functions
- lack of awareness in patients about their performance (it is common for patients to think that they are not affected by their injury when they are heavily disabled)
In RehabCafe, clinical psychologists can provide a safe and controlled environment for their patients in an immersive VR environment. For patients, they are able to prepare themselves for real world situations they may encounter due to their disability. With my solution, there is no need for real world physical space and possible enhance the effectiveness of the strategy with the use of VR. It is also possible to transfer the skills they have learnt into their daily life and help them get back into the workplace.
There are several benefits of using a VR environment compared to current rehabilitation strategies that brings patients to real-world environments:
- Safe – Reduces risk of damaging real world environment and injuring people
- Cheap – Reduces staffing and time requirements
- Customizable – Customizable for different levels of difficulty
- Transferable – Imitates real world settings
- Recordable – Recording of data to increase ecological validity to prove effectiveness of therapy
- Replayable – Replayable sessions for patients to be self-aware of their performance and to practice their skills
In this project, I produced 4 design iterations/revisions. The prototype was meant to be evaluated by clinical psychologists to validate the efficiency of the VR environment as a viable solution. The prototype produced was a VR environment used for usability testing.
1️⃣ Establishing requirements – Literature Review
Read, read, and read! 📚
My first step was to understand the topic and themes in mental health to be able to understand the current situation, and be able to hold conversations. Through literature review by reading and analysing journal articles, I tried to understand:
- What is ABI and what are its effects?
- What are the current methods used to treat ABI?
- What are the limitations of using current methods?
- How are the methods derived from? What are the foundation theories of these rehabilitation strategies?
- Have VR technology been used in clinical psychology before?
- What were their challenges and achievements?
- What is the optimum duration of a VR session?
- How can I produce the VR environment from the technical aspect?
2️⃣ Designing alternatives – Iteration 1
Setting: Streets of a virtual city
Core features: Real-time data visualisation and social interactions
Let’s dream big! 💭
For my first iteration, I thought to go as far as I can so that I can scope it down from there. I went for a big ambition to incorporate the Multiple Errands Task into the environment. Besides having tasks itself, there was an opportunity for a real-time data visualisation that would unleash full potential of VR technology. This would allow clinical psychologists to not just monitor their patients’ performance in real-time, but also be able to record the performance, and review it at a later time.
As there was a need to schedule a meeting with the clinical psychologists, I sketched a rough idea of the VR environment to possibly become a topic for discussion.
Rough sketch of idea 📝
I thought that perhaps 2 different screens would be required for the 2 different groups of users. The one for the patient would be within the VR space. The one for the clinical psychologists would be on a 2D space separate from the patient. This is so that the clinical psychologists can make sure the patients are safe from physical objects in their surroundings, and also be able to observe and manipulate the VR environment.View Details
For the patients, they can physically move within a small area in the real world, or use a controller to move their virtual character depending on their ability to walk or move. They will then carry out some cognitive tasks for therapists to assess their executive functions in the VR. Example of such cognitive tasks are determining when a supermarket open and finding out what day it was based on hints from the environment. Besides that, they would also be given social tasks that involves having a conversation such as finding out what time a bus is coming by asking a virtual avatar.
For the clinical psychologists, they will have a different screen to monitor the patients’ performance. This involves a display of real-time charts mapped into 3D space. This will contain information about how long the patient is looking at something and also a trail of the patient’s movement in the VR environment. A text input is also put into their screen to let patients know there is a change of rules. To let clinical psychologists have control over the social interactions taking place in the VR environment, their screen will also have a list of avatars found in the VR scene. If the clinical psychologist wants to give out a task to the patient through an avatar, they can click on an avatar and click “Press to talk” on their screen. However, when a patient actively interacts with a specific avatar, they will be able to know which avatar the patient was interacting with on their screen. Then, they will click on a dialogue option of what the avatar would say from the avatar list. If the option was not on the avatar list, they can send a voice clip via a microphone, with the voice changed to match the corresponding avatar’s voice to make the avatars seem real.
3️⃣ Evaluating design – Iteration 1
To evaluate my designs, I approached my supervisor whose responsibility was to guide the scope of my supervisor and acts as a first point of contact for feedback. It was also part of the discussions during the focus group with the clinical psychologists in the next step.
- real-time monitoring of performance
- full control over what avatars in the environment would say
- scope of project too wide (in terms of scale of environment and features)
- clinical psychologists having to oversee too many things at once
4️⃣ Establishing requirements – Focus group
Informal discussion with the experts 💬View Details
After the initial literature review, I conducted a focus group with two clinical psychologists as well as two professors from my university. During this discussion, we let the clinical psychologists try out a different environment developed by one of the professors to experience the VR environment. We then have a discussion about the issues they were currently facing and what are the possibilities that could be fulfilled by the VR environment. This allows us to get a better understanding of how clinical psychologists would run a session of Multiple Errands Task. Multiple Errands Task is a test allows them to assess the criticality of the injury in patients and is also used to help patients recover. It involves giving patients a set of rules, tasks, and distractions which are flexible that be adapted for different environments in a rehabilitation session. Most of the findings from the initial literature review overlapped with the discussions especially about the issues they were facing.
Some other questions answered during the focus group discussion includes:
- Do clinical psychologists inform their patients what they are assesssing before or after an assessment?
- Is it useful to repeat the activity? Would it be transferable from VR to real life?
Other suggestions during the discussion came up such as:
- What if we place a ball in the middle of the virtual road? This will allow clinical psychologists to assess if their patients are able to stop or limit their behaviour given a dangerous scenario. This would often not be done in real life as it imposes real life dangers.
- What if someone in VR behaves rudely towards the patient? How will they react? Will they be able to stop or limit their impulsive behaviour?
Summary of findings 🔍
- The VR environment will need memory assistance aids or movement aids due to disabilities caused by ABI
- There is a need for clinical evaluation to ensure the ethicality in recording of data is ensured and making sure that it’s a safe and effective environment
- Current rehabilitative strategies are unable to assess the emotion factor of decision-making as it is unpredictable in real world settings, endangering those around the patient. Having a VR environment will reduce such risks
- Although motion sickness is common in immersive VR settings, it was rarely reported in the test sample despite the patients’ brain injuries
- Having distractions and unexpected social scenarios in the environment would help clinical psychologists assess the executive functions of their patients
From there, I was able to come up with a list of requirements for 2 user groups. This list will be the core backbone of what the VR environment has to include.
- Able to perform a series of tasks that access different skillsets of patients
- Able to communicate rules and changes if they occur
- Able to partially control conversations that will take place between a virtual avatar and the patient
- Able to adjust the amount of distractions on the patient
- Able to monitor how the client reacts to different situations
- Able to checklist if the given tasks were completed
- Able to report to the patients about their performance
- Able to experience VR without feeling motion sickness
- Able to navigate and interact in VR without hindrance from their disabilities (such as if they cannot walk)
- Able to remember how to navigate or interact in VR (some patients have dementia and will not remember)
- Able to understand what their tasks are
- Able to have natural social interactions with the virtual avatar and display emotions like how they would in real life
- Able to identify errors they made (some patients may be denial to their underperformance because of their brain injury)
In contrast to the common use of personas, I used a list of requirements as I would want to avoid bias when it is such an important tool being used by professionals and those invulnerable. Another reason I didn’t construct personas was to save time. I would later need to develop the environment under a short amount of time. Being able to incorporate these requirements into the environment quickly to generate usability test results is more important than the presentable artefacts for my dissertation.
5️⃣ Designing alternatives – Iteration 2
Setting: Streets of virtual city
Core features: controls (interactions with VR environment), cut-scene introduction, distractions, virtual handheld device for social interactions and object interactions
Changes from previous iteration:
– real-time data visualisation was removed from the scope of the project (due to time constraints)
– consideration of motion sickness issues and immersive experience in VR
– more details on avatar interactions (how the patient will interact with an avatar and how the conversation will be carried out)
Similar with previous iteration:
– same social and cognitive tasks to be completed by patients
User flow 🖊️
After the focus group with the clinical psychologists and university professors, I had a better idea of how they usually carry out a rehabilitation session. Using those details, I created a user flow for the sequence of actions that will be carried out by both the patients and the clinical psychologists.
Tackling challenges 🧩
Two major challenges were also brought up during the discussion. The first was tackling motion sickness as it is the most common issue in VR environments in general. The second was to create a believable environment for the patients so that the patients can have natural interactions in VR.
Dealing with motion sickness
From the initial literature review, I found out motion sickness comes from the misalignment between the visual senses and the feeling of movement. Although it was not an issue from previous experiments that my professors conducted, the clinical psychologists were concerned during the focus group session. This is because motion sickness can cause discomfort and trigger spasms in the patients. To deal with motion sickness, I had the patients start in a sitting position, and have their avatar seated at the start of the VR environment as well. This will help avoid motion sickness and save them from hazards such as tripping onto cables from the setup. From the technical side, I also had to reduce the graphical needs of the VR environment to prevent “lag” in the system. If lag occurs, there is more likely to be a misalignment with what they see and their in-VR movement, and causes motion sickness.
Creating a believable environment
I wanted to create a believable VR environment so that the patients can interact naturally and take the activity seriously. To create a believable environment, I thought to give patients context of ‘why they are there’ so it gives them a motivation and reason to carry out the tasks later on. This also resembles real life situations of when they have to go out alone where they will have specific goals to reach.
With the user flow and list of requirements generated, I pictured how a rehabilitation session will be carried out and what kind of scenarios will be in the VR environment.
For the patient, they will be asked to carry out some tasks with some rules in place, and distractions to shift their attention from their tasks.
Due to ethical considerations, it was important to let patients know that they are free to exit the VR environment at any time. They also had to know about how their data is protected and other safety measures. I considered whether to have this ethical information within the VR environment or to be given before they enter the environment.
After considering that the clinical psychologists will be busy setting up the environment during this time, I decided to have the information within the VR environment so that the clinical psychologists will have less work and can set up the tasks in the VR environment.
After having a storyboard of what entails in the VR environment, I created wireframes for the screens clinical psychologists will use. These are to be placed in a second monitor.
6️⃣ Evaluating design – Iteration 2
With this design, I asked for my supervisor’s advice again as well as asking other lecturers to provide feedback for what they think about the design. By giving them a quick run through of my study, they were able to identify the limitations of my environment and suggested to rescope my project due to the constraints of time.
- interfaces for therapists to observe and manipulate the VR environment for cognitive rehabilitation
- having ways for both clinical psychologists and patients to interact with the VR world
- long development time needed to build a virtual city with the tasks embedded and interactions with virtual avatars and the objects in environment
- long navigation time
- lack of framework for virtual avatars
7️⃣ Establishing requirements – Literature Review
Looking at virtual avatars 🤖
When I looked back at Iteration 2, it was not clear for how the social interactions between the patient and virtual avatars would work. As it plays a big part of assessment and recovery for the patients and would also help increase believability of the environment, I researched on how non-player characters (NPCs) are used in games and other applications based on my gaming experience and literature review on models of dialogues. After collating my research findings, I created a chart with three layers of a virtual avatar. With this framework, it will make virtual avatars seem more human and more believable.
Looking at social dialogue models 💬
For the content of what virtual avatars would say, I looked into models of typical conversations in people, known as social dialogues. I compared the different dialogues and used one that was easier to implement from the developer point of view. In the end, I adjusted a small talk model to incorporate impolite behaviour so that they can provoke the patients to display negative emotions. This is important for clinical psychologists to assess the patients’ behaviour. Although the model was of a high level, it was easier to comprehend and applicable for the communication between a virtual avatar and a patient.
8️⃣ Designing alternatives – Iteration 3
Core features: client-facing role for patients, virtual avatars as customers in a cafe
Changes from previous iteration:
– change of settings from city to cafe
– change of tasks to do to fit in cafe
– addition of emotion wheel to clinical psychologist’s interface
– addition of an end-screen to clinical psychologist’s interface
Similar with previous iteration:
– same dual monitor setup and controls
– narrative cut-scene intro
– same foundations of virtual avatars
The primary focus of this iteration was to reduce the size of the VR environment so that I can develop a prototype in time. Instead of using a virtual city, I decided to only develop one part of a virtual city. I decided to use a cafe because it is a common place for patients to visit, so it can be transferred to real life situations. Also, it is a place where people gather so patients can have social interactions.View Details
In traditional rehabilitative methods, patients go into a shop buy items and have minimum social interactions with people. This makes it difficult for clinical psychologists to assess some parts of their behaviour.
I turned this around by having the patients to be on a client-facing role instead by letting them be a barista. This pushes them to face more people and interact as them, as well as doing cognitive tasks. This also trains their ability to manage real life situations and also boost their confidence to get back into society.
From the clinical psychologists’ point of view, by keeping the tasks within the cafe, it makes the tasks more focused. This helps them to find subtle differences or variations of the patients’ abilities.
9️⃣ Evaluating design – Iteration 3
After discussions with my supervisor, the iteration was deemed to be of a good base of research as there was no research to date that puts a patient in a client-facing role. Furthermore, I was able to fit the requirements of both the clinical psychologists and patients within the new environment.
- new perspective of assessment with client-facing role of patients
- fits cognitive and social tasks in the environment
- lack of technical design for development
With the shortcomings in Iteration 3, the technical design for development was done before producing the prototype. During the prototyping process however, a few problems came up. The whole process of prototyping took only 2 weeks for the functional environment so that I could do usability testing.View Details
Challenges during prototyping
First, the design was based on the controls for the HTC Vive which came with hand controllers. However, when I was building the environment, I was notified the equipment was no longer available despite initial agreements .
I was asked to choose between the Vive Focus and Fove headsets. While the Vive Focus had controllers for tactile interactions with the environment, Fove had eye tracking capabilities for recording data. In the end, I chose the Fove mainly because it was compatible to a dual monitor setup and more convenient development testing.
Second, during development, there was a problem with the software for eye tracking on the Fove . After calibration, the eye tracking was not giving the correct data. As such, I did a workaround by having a gamepad for interaction with the environment.
Third, I was not skilled in creating assets (audio, 3D models, environment, etc.) and there was a short time for creating the assets for the prototype . To get some 3D models of the environment, I asked the supervisor for funds on the assets and added some Unity Asset packs into my environment. They were very limited so I also had to create models of my own for the customers with Autodesk Character Generator and Mixamo. For the audio, I also had to scout the internet for some royalty free audio assets which I can plug them into my environment. I also had to edit them as I was using noise as a distractor in the tasks.
To evaluate the design, participants were recruited for a pilot study that involved two parts. Part 1 involved carrying out tasks in the VR environment and Part 2 was an interview session to understand the participants’ background and opinion of the system built. Essentially, the pilot study sessions would help inform the usability of the controls, immersive experience (measured by in VR, and efficiency of the system to evoke natural
Healthy participants were recruited for this study instead of patients with ABI because patients with ABI were less accessible and more vulnerable to take the test at this stage of development. As such, the environment was adjusted to be more difficult for the participants such as not having a tutorial and were only told general rules of the environment.
By using transcriptions from interview data and observations in lab settings, the data was examined for features of interest and grouped by common themes. A mixed-methods approach was taken to interpret and present the results.
- 50% of participants were successful at completing the tasks. Those who had experience with VR, or those who were less immersed in the VR environment, and those who did not find the controls to be difficult were more likely to be successful in the tasks.
- 25% of participants were frustrated at the customers as they simply wanted to progress in the simulation. Other participants reported being frustrated at the environment and the controls of the system.
- Participants found it more frustrating when a customer changed their order when they were completing a simple task.
- Participants forget to do a task if the customer becomes annoyed.
- Those who were more explorative in the environment was able to complete the tasks.
- Many participants found it unnatural to talk-aloud to a virtual avatar knowing that they were physically situated in a room without anyone else beside the researcher.
- 1 participant was showing unfriendly behaviour to the virtual customer knowing that it was not real
- Some participants missed out on subtitles despite the subtitles being in their field of vision all the time
- There was a lack of feedback in the environment so participants was not sure if they completed a task
What challenges did you face and how did you deal with them?
- Data loss – During the usability testing stage, the screen recording did not save for half of the sessions. I found out there was not enough storage space in the disk drive which I was saving to. I did not have any other backups and the screen capture data was lost. There was nothing I could do except making sure the recordings would be saved for the later sessions.
- Unavailable equipment and hardware problems – The initial designs were based on the HTC Vive, but the equipment was not available and I had to use FOVE. The main difference was that FOVE does not have controllers like the HTC Vive, so I had to develop workarounds that with the equipment I have. The only other interaction FOVE had was eye gazing, which I faced hardware problems during development. In the end, I resorted to using a gamepad to replace the original controllers while having a static “cursor” in the middle of the screen.
- Limited access to patients and clinical psychologists – Due to the vulnerability of patients and nature of the job as clinical psychologists, I was not able to contact them as often as possible to clarify further questions that came up. I had to use my own judgment through my own background in psychology as well as understanding them more through academic journals.
What did you learn?
- Prepare for the unexpected – Things might not always turn out the way you expect, be ready to come up with alternatives and adapt to meet requirements within tight schedules.
- Talk to people about your ideas – I learnt to talk through with others to get feedback and to strengthen the arguments for my ideas. Everyone have different perspectives and different experiences that can provide valuable opinions.
- Everyone is a designer – From getting feedback from people around me and the getting suggestions from my usability tests, everyone was able to give a lot of good ideas I previously didn’t think of. It is good to keep an open mind and learn from others as everyone knows a bit of something more than yourself.
What would you do next time?
- Take one step ahead – There were times when I was waiting for comments or feedback from my supervisor and it was delayed or postponed because of unexpected events. I felt like I should have just taken the step forward and do what I think see fit rather than waiting for the green light to continue on the work.
- Confirm resources beforehand – Even though it was said that I would be able to get a hold of the specific VR environment at the start and also gain access to volunteers with ABI, I should have confirmed it before going ahead. Due to the different circumstances that came up that limited my resources to both the equipment and volunteers, I had to come up with a lot of last minute turnarounds, both in the design and development stages.