Petsona AR

Petsona AR| The Isolation Solution

The Isolation Solution

Social distancing policies enacted the world over during the COVID-19 pandemic have left many people socially isolated. Your challenge is to develop innovative solutions to combat social isolation.

Summary

Social Isolation measures have been enacted around the globe to reduce the risk of contracting and spreading COVID-19. Some restaurants have placed stuffed toys on seats to help remind people to keep physical distancing from other people. We demonstrate the potential use of Augmented Reality (AR) in shared mealtime scenarios with lessons learned from long-term Isolated-Confined Environment (ICE) experiments conducted by NASA.

How We Addressed This Challenge

ICE experiments highlighted the importance of shared mealtimes in maintaining feelings of social-connectedness. Petsona AR is a proof-of-concept that aims to improve the social-isolation experience for those living alone by creating an AR dining companion. The social disconnection felt despite using videoconferencing apps can be attributed to the fact that video calling can cause stress for people because having delays can cause people to be perceived as less attentive, extraverted, and conscientious (Schoenenberg et al., 2014). Thus, this project wanted to serve as a countermeasure to feelings of social isolation by using AR since it can counteract sensory monotony by creating a presence similar to that of a human presence (Bailenson et al., 2019).

How We Developed This Project

We were inspired by photos on social media where restaurants placed teddy bears on seats as guides for physical distancing. Since food was an essential component of social connectedness, we came up with a project involving food. We decided that we wanted to build an app that we would personally use as we saw ourselves as some of the more vulnerable in times of social isolation.

We were also able to chat with one of the NASA Subject Matter Experts (Romain) who shared with us his experiences after being in isolation for 512 days and he explained how important shared mealtimes were for him.

We conducted a lot of research using NASA open data and technology research about astronauts in isolated, confined, and extreme (ICE) environment simulation or on duty in space (NASA, 2020). We used these to inform our design and nature of our application; hence AR instead of VR and the focus on food.

We have no prior experience building an AR application so we decided to use Unity as it seemed the easiest to learn for AR applications and used Vuforia because of it’s free and available nature.
We used online collaboration tools for communications, project management, and creative collaboration such as Jitsi, Discord, Facebook Messenger, Aggie.io, and Trello.
We used drawing tablets, smartphones, cameras, 3D Scanning software.
We created our assets using Adobe Illustrator and all our 3D Models came from Poly Google.
We created our demo using OBS, Shotcut, and Google Slides.

Problems: We struggled exporting our Unity Project to Android and we had problems with the 3D tracking of the teddy bear.

Achievements: We were able to create AR demonstrations in less than 24 hours of learning Unity. Our Facebook page passed 100 likes.

Project Demo

Youtube link: https://youtu.be/rVCBHeJMpq0

Given the opportunity and resources, we will:

Create a self-contained Android/iOS application
Add speech input and NLP for user feedback
Use machine learning for the AR Petsona to dynamically respond to user behavior

Add more features such as

Relationship development with pet
Facial recognition for user recognition and facial emotion detection
Ability to import own stuffed toy for a more personalized experience
More food choices for the pet, particularly a wider cuisine scope (as we learned from NASA Romain)

Project Code

https://pastebin.com/W7BeedrP?fbclid=IwAR09V4KKfLYDm7ulxRwz2EWKLSqSrc7j1vlidIbKZlfhIZcInRtBK9JfDKQ

Data & Resources

3D Model Assets used:

Glass of Milk - Poly by Google (Public/Remixable (CC-BY)) https://poly.google.com/view/f5Uw-npR6kG
Cloud - Poly by Google (Public/Remixable (CC-BY))https://poly.google.com/view/44cGXp6_8WD
Takoyaki by Sugamo, uploaded in poly.google.com (Public/Remixable (CC-BY)) https://poly.google.com/view/dKZ84zp5T6O

NASA research and article links:

1. Individualized Behavioral Health Monitoring Tool

https://data.nasa.gov/dataset/Individualized-Behavioral-Health-Monitoring-Tool-P/5ghh-gt3j

https://data.nasa.gov/dataset/Individualized-Behavioral-Health-Monitoring-Tool-P/q2x7-9u78

2. Automated Behavior and Cohesion Assessment Tools

https://data.nasa.gov/dataset/Automated-Behavior-and-Cohesion-Assessment-Tools-P/t56y-mg4q

3. ANSIBLE: A Network of Social Interactions for Bilateral Life Enhancement

https://data.nasa.gov/dataset/ANSIBLE-A-Network-of-Social-Interactions-for-Bilat/x3ic-v679

https://data.nasa.gov/dataset/ANSIBLE-A-Network-of-Social-Interactions-for-Bilat/mjw8-cp3j

https://data.nasa.gov/dataset/ANSIBLE-A-Network-of-Social-Interactions-for-Bilat/x3ic-v679

4. Siri-like Smart Crew Assistant

https://data.nasa.gov/dataset/Siri-Like-Smart-Crew-Assistant/u2xk-2uqr