The Masked Scales| The Art of It All
Awards & Nominations
The Masked Scales has received the following awards and nominations. Way to go!
Most Inspirational
The Art of It All
The Lockdown Musical: Sonifying Impacts of COVID19 using Data from Homemade Instruments and NASA
Summary
We built an instrument using Arduino, Sensors, and Camera to measure changes in Street Noise, Vehicular Traffic, Emissions and Light Intensity during COVID19 lockdown. We used Machine Learning (to count vehicles from video feed) and Python to assess changes happening during and after the COVID19 lockdown. We converted our Analysis into a Musical using 4 musical instruments (each representing a change in different variables): Marimba (Light), Vibraphone (Emission), Piano (Street Noise), Flute (COVID19 Infection Cases in Toronto). The tempo of Music was determined by changes in City Night Lights (using NASA VIIRS Data) and Vehicular traffic count before and after the lockdown.
How We Addressed This Challenge
COVID19 impacted kids in a big way. It took us away from school, school friends, after school clubs, sports, and music. While COVID19 lockdown closed doors of our schools we wanted to make sure windows to learning and fun remain open in our homes.
We decided to learn through experimenting and building things during COVID19 lockdown. Our project brings together our interests in robotics, coding, machine learning, and playing music (piano and guitar) in a fun way.
We built a home-made instrument using Arduino with sensors: to document changes happening in our immediate environment. For instance changes in street noises (during day and night), changes in pollution, and changes in city light levels.
Analysing the data gathered by our sensors, we were able to better appreciate the gradual changes happening around us because of the lockdown. For instance, the background city noise was less, and the air was cleaner.
We later added a camera to our instrument to get real-time traffic data. Artash created a Machine Learning algorithm to do live identification and counting of cars, buses, and streetcars. We now had more complete information on changes happening during Covid19 Lockdown and were able to compare it with data gathered after the lockdown was over.
We did not want to lose our coding skills so we ended up writing Python programs to analyze and create graphs of data being gathered.
The most fun part was definitely bringing out our Piano and Guitar and doing some jamming to find the best way to transform our Data Analysis into Sound. We experimented with different software including Reaper, Audacity but finally choose Musical Algorithm software to merge different data streams as different musical instruments.
The end result was Sonification: a COVID19 Lockdown Musical to portray the changes in form of art and music!
How We Developed This Project
We (Artash and Arushi) are sibling makers! We both love space, robotics, and music.
Our Approach
Our approach to the project was learning through experimentation and bring together all our interests in one project.
We wanted to document changes (in Environment and Traffic Patterns) happening around us because of COVID19 lockdown at all times by building a homemade scientific instrument. We then wanted to develop our own Python program to do real-time data analysis. And use a Sonification software to transform the Impact Data into a Musical of COVID19 Lockdown.
How NASA Space Agency Data was Used
We love Astronomy and it occurred to us to find out if the Night Skies have become darker during COVID19. After watching the NASA SpaceApps Bootcamp videos we came to know about NASA Visible Infrared Imaging Radiometer Suite (VIIRS) data which can observe city lights from space. The Data was also available on NASA Worldview portal in real-time. We used this VIIRS Data and compare tool available on the NASA website to assess change in city lights of Toronto during the lockdown period we were interested in.
This data was used to determine the Tempo of the Musical generated. An increase in city lights was mapped to lower music tempo and vice versa.
Hardware Used
Arduino, Light Sensor, PM 2.5 Sensor, Intel Real Sense Camera
Coding Languages Used
Python (TensorFlow, Keras)
Sonification Software
Musical Algorithm: http://musicalgorithms.org/4.1/
Achievements
1. Our homemade instrument successfully collected data and street traffic video in real-time.
2. We were able to transmit information from our Sensors and Camera to our Python program for analysis.
3. We were able to assess the changes happening in different ways: Light, Emissions, Noise, City Lights and Vehicular Counts.
4. We saw a noticeable impact of Lockdown in terms of a decrease in traffic, street noise, emissions, and even city lights.
5. We were able to identify the most appropriate musical instruments (Piano, Flute, Marimba and Vibraphone) to sonify impact measured.
Problems Faced
Our homemade instruments had a very fast sampling rate and we ended up collecting millions of data points. We had to create a write program midway to average the readings so that we are able to complete analyzing the data.
Project Demo
30 Seconds Video:https://www.youtube.com/watch?v=l2WqWKDlq9c&feature=youtu.be
Data & Resources
Data Collected from our Home-Made Instrument
1.Street Noise (Microphone)
2.Vehicular Emissions (PM 2.5 Sensor)
3.Vehicular Count on Street (Intel RealSense Camera)
4.Light Intensity Data (Light Sensor to analyze day and night data)
External Data Sources
1.NASA Night Light Data (Suomi / VIIRS Data). https://worldview.earthdata.nasa.gov
2. Toronto COVID19 Infection Data (City of Toronto). https://www.toronto.ca/home/covid-19/covid-19-latest-city-of-toronto-news/covid-19-status-of-cases-in-toronto/