Legend:

1.What inspired your team to choose this challenge?

2. What was your approach to developing this project?

3. How did you use space agency data in your project?

4. What tools, coding languages, hardware, software did you use to develop your project?

5. What problems and achievements did your team have?

1. Our team was inspired by an opportunity to participate in contributing to the space industry by designing a remote crew support system for situations, such as the current pandemic and when the HQ cannot be reached. We hoped it would help reduce mission failures and enable a task force to be self-sustaining. We also wanted to express our gratitude to the valiant members of the space force and researchers who risk their lives and well-being in the name of science

2.   First of all, we brainstormed ideas to develop a plan for this project, and having discussed and consolidated the final version of the plan, we proceeded to understand each others’ skill sets to see how we can best create deliverables and bring this project together. The skillset included data analysis, data engineering, ML, solution architecture, and organizing and maintaining an organized schedule of the activities via daily meetings.

Our first approach was to identify which audience we wanted to focus on which we identified using the “Persona technique”. We also were keen to find an accurate but non-intrusive way of capturing data as well as what kind of data we would need to capture.

Our next task was to come up with a solution to autonomously monitor crews without the need for human labor. Conversing on the subject, we concluded we can use small embedded devices such as Raspberry pi's to analyze crewmembers' facial emotions. In parallel with the other tasks, we developed the entire hardware system.

We then identified the next phase which was how to implement privacy-friendly processing of that data and how it can be presented in a user-friendly way. We also acknowledged the importance of those who handled sensitive medical data and implemented that in our plan.

As J.A.R.V.I.S. is a tool, we put in a lot of dedication in developing a user-friendly way to interact with the complex AI system. One of our teammates had experience in creating data pipelines and data dashboards, so we spent time connecting our datasets and camera feeds to the dashboard. This allowed us to design a user-friendly visual that concisely summed up J.A.R.V.I.S’ analysis.

3. We have used psychological data on the Performance Readiness Evaluation Tool (PRET), made available by the Canadian Space Agency. The reason why we picked this data set is because it provides us with the psychological perspective of potential end-users of our project and helped us to visualize how our dashboard would work.

Through our research, we did not find other raw data sets (so .csv files, spreadsheets, etc) to assist us in our project. However, the following datasets were used:

We have used biometric data from wearable devices which came from the WESAD dataset, available in the University of California Machine Learning Repository. We have chosen this dataset because this is the data we would be collecting from end-users. Due to our time constraint and the lack of essential hardware (we did not have access to a heart rate sensor) we analyzed what kind of response AI could potentially recommend to the crew captain.

Finally, we have used:

• The segmentation model (drawing of the person contour)which was was trained on MSCOCO dataset which was fundamental to our project development
• An emotional analysis model that was trained on the FER13 dataset. This dataset was used to analyze human emotions which is fundamental to the functioning of J.A.R.V.I.S. in this project

4. We have used draw.io to draw architecture diagrams. We also used Google docs to exchange information, edit documentation, organize meetings, and share agendas. For meetings, we have used rocket chat, zoom, and google meet.

For the project, we have used:

Infogram: used for dataset illustration;

Python: Language used for the back end

• Libraries used:
  • TensorFlow: Used for deep learning (using Keras);
  • OpenCV: Used for image manipulation and preparing our video demo;
  • Pandas: Used for data operations, tabular data analysis and manipulation;

Hardware:

Raspberry Pi: Used to collect, pre-process and transmit video data (prototyping edge computing); In the future, this will be replaced by an application-specific embedded device

AWS: Used as a computation platform;

Spark: Used as a data processing framework.

5. Outline of issues faced while making J.A.R.V.I.S:

1. Remote communication and idea sharing in a multi-timezone environment - we have agreed on the length of the meeting to make sure we plan ahead and don’t spend too much time on the issues or exhaust team members in other time zones;
2. Language barrier - where there was an issue of a language barrier, team members would help translate the idea or clarify that to the team;
3. Finding data sets that had to match our project- we spread out our resources to try and find any relevant datasets which turned out to be a success, however, we have the potential to find more datasets next time;
4. Discussing the issue of privacy and how it will be embedded in our system - we designated a special call for this and prepared a document with all of our ideas. This was all in an effort to ensure that we can see where we agreed and disagreed on those matters. After doing so, we then debated and agreed on the common elements of the privacy infrastructure in our project;
5. Integrating the subsystems together: As we were limited to only two days of working, we developed the hardware, software, and visuals all in parallel. This meant that we had to dedicate a serious amount of time to fully integrating every part together; and
6. Writing documentation: With a limited amount of word space, we chose every single character with care. This was definitely a tedious task.

We had achieved the following:

1. We have created a prototype of a significant part of our system which includes data ingestion and processing, video recognition, information dashboards, risk detection, and an alerting engine. We also had time to implement high-level design for this project and were able to fully validate the concept of J.A.R.V.I.S.
2. We successfully resolved emerging issues by maintaining a successful and on-going communication and agenda-keeping to avoid wasting time and focusing on the core of the project. We continued to stay on task despite feeling the urge to celebrate our small victories, which greatly benefited the project.

https://drive.google.com/open?id=1xDbKU7APTcmyEHJKxMvtkjVd8W9mYmIj

1) Biometric data from wearable devices: from WESAD dataset, available in the University of California Machine Learning Repository

https://archive.ics.uci.edu/ml/datasets/WESAD+(Wearable+Stress+and+Affect+Detection)

2) Psychological data: data from Performance Readiness Evaluation Tool (PRET), made available by the Canadian Space Agency

https://www.asc-csa.gc.ca/eng/open-data/access-the-data.asp

3) Video analysis: the segmentation model (drawing of the person contour) was trained on MSCOCO dataset, and the emotion analysis model was trained on the FER13 dataset

https://datarepository.wolframcloud.com/resources/FER-2013