LungAI - Space Apps Challenge

LungAI| Human Factors

Human Factors

The emergence and spread of infectious diseases, like COVID-19, are on the rise. Can you identify patterns between population density and COVID-19 cases and identify factors that could help predict hotspots of disease spread?

A Deep Learning Based Web App to Detect and Predict COVID19,Lung Cancer and Other Lung Abnormalities

Summary

LungAI is a web application that aims to make Covid-19 detection accessible to all as well as reliable. It also aims to make Lung Cancer detection as well as detection of 16 other common and dangerous Lung Abnormalities like Tuberculosis, Pneumonia, Edema etc. accessible so that users may be able to not only diagnose whether they are positive for Covid-19 but also detect what other medical issues they have using digital images of radiographic chest images in the form of CT Scans or X-Rays.

How I Addressed This Challenge

AIM & INTRODUCTION TO LungAI

LungAI is a web application that aims to predominantly make Covid-19 detection accessible to all as well as reliable. It also aims to make Lung Cancer detection as well as the detection of other Lung Abnormalities accessible so that users may be able to not only diagnose whether they are positive for Covid-19 but also detect what other medical issues they have in the off chance that they test negative for Covid-19.

All the web application requires is a Photograph of the affected patient’s lung CT Scan or X-Ray to be uploaded. It then tests this image against the Covid-19 detector and gives a probabilistic answer as to whether the user is positive for the coronavirus or not. In case the user tests negative for coronavirus he/she may then go further and test their lung CT scan or X-Ray for lung cancer as well as other lung diseases such as Tuberculosis, Pneumonia etc. Apart from this the website has an accuracy of over 98% when it comes to its ability to make quick detection's within 3-5 seconds.

WORKING OF LungAI

The website contains a total of 6 deep learning based models running as the backend of the website. The website contains 3 basic models, one for detection of Covid-19, one for the detection of Lung Cancer and finally one more model for the detection of other Lung Abnormalities like Pneumonia, Emphysema etc. Each of the above 3 models further contain two sub types, i.e., one for the detection of all of the above medical issues in case the user has a CT Scan of his/her lungs while the other one is in case the user has an X-Ray of his/her lungs.

Apart from this the website also has another predictor function that gives a detection based on the users current location and as to whether the location could increase his/her chances of being/getting infected by the novel coronavirus. In this manner LungAI can not only detect whether a person has COVID19 but can also predict whether the user may get coronavirus based on a net detection given using his lung's current state/chest radio-graphic image and his current location.

The way the website is to be used is that the user must first choose whether the input image will be a CT Scan or an X-Ray of his/her lungs. Then based on this the user must choose either the COVID-19 Detection(CT Scan) or COVID-19 Detection(X-Ray) model. Next the website asks the user for his/her current location from the list of areas curated based on the available datasets provided during the COVID19 Space Apps Challenge by various agencies. These areas have further been divided as per the current WHO guidelines,i.e, based on the number of active cases, rate of spread etc. Based on the input the website will give a prediction as to whether the given lung CT Scan or X-Ray indicates the presence of Coronavirus or not.

In case the website gives a prediction of the patient being positive for Covid-19 the website offers contact information as well as the location of nearby hospitals along with the all the information regarding treatment of Covid-19. On the other hand in case the website gives a prediction stating that the given lung CT Scan or X-Ray is negative for Covid-19 the user then must further test the given lung image against the Other Lung Abnormalities model(CT Scan or X-Ray).

This model tests the given radiographic lung images against 16 of the most dangerous and common lung abnormalities like Tuberculosis, Emphysema, Pneumonia etc. It also contains a category of No Finding which states that the given lung image indicates a healthy patient with no lung issues. On testing the lung image against this model the website will give a prediction as to whether the given lung image is healthy or not and in case not healthy then which lung abnormality the patient is afflicted with.

Apart from this there is also the option of Lung Cancer testing using radiographic lung images on the website. The model again gives the user the option to submit either a CT Scan or X-Ray image of the users lungs to get a correct diagnosis. The website predicts as to whether the user has cancer (malignant) or not (benign).

The website also provides users the option to get a second opinion in case they wish to get their medical case reviewed further or in case new users wish to get information regarding hospitals that are giving treatment or testing for Covid-19. In both cases the website provides the relevant contact details and information to further help users get the details they require. Apart from this the website has a system to send their cases to doctors listed on the website to further review their cases by using an automatic messaging system on the website.

LungAI also gives users live updates regarding Covid-19 cases and other relevant information not just pertaining to India but from around the world and also provides graphs and map diagrams to further supplement the data on the website. The data is updated on a daily basis so that users can remain informed with the most current information from around the world.It also aims to prevent the spread of fake news by functioning as a platform which also provides relevant and accurate information from around the world regarding the current effects of the Covid-19 pandemic

Lastly LungAI if considered a viable solution for detection of Covid-19 could potentially solve the problem of lack of testing kits in various parts of the world as it would be accessible all users and hospitals around the world provided they either have a recent X-Ray or CT Scan thus in turn also speeding up the detection process and potentially saving millions in terms of money for governments around the world to deploy in other methods to combat Covid-19.

The final aim while developing LungAI as a deep learning based website was not only to detect the presence of Covid-19 among patients in an accessible and reliable manner but was also to detect all kinds of prevalent lung issues be it Covid-19, Lung Cancer or other lung based issues like Pneumonia among patients in such a manner so as to narrow down the possibilities for doctors in case the user decided to opt for a second opinion apart from LungAI’s detection.

SYSTEM ARCHITECTURE & CAPABILITIES

The system architecture at its most basic level involves the use of a web application that it will take an input in the form of an image from the user and then give a prediction based on the application using various deep learning models in the backend.

The web application will also have the ability to give the required information about various doctors that users can send their lung imaging predictions as well as other data to get predictions from real doctors as well. In this manner the website not only provides correct diagnoses to patients but also helps reduce the load of doctors at hospitals due to them already having the websites predictions which will further help them in narrowing down the lung issue the patient is facing.

INNOVATION/OUT OF THE BOX THINKING

Wider Range of Diseases. While there are few websites that give users the option of diagnosing their issues online LungAI provides users the opportunity to detect Covid-19, Lung Cancer as well as 16 other Lung Abnormalities online using only a single radiographic lung image.

CT Scan & X-Ray Support. LungAI helps further the detection of various lung issues irrespective of whether the user has an X-Ray or CT Scan as it provides users with support for both forms of medical imaging.

High Accuracy. Despite covering over 16 types of lung conditions, Covid-19 detection and Lung Cancer detection the application still maintains an accuracy of over 98%which can only improve once the datasets of disease images increases. Apart from this the website gives detection in three levels by first classifying the lung images as Covid-19 positive or negative. In case the test comes out as negative the website allows users to then further diagnose themselves by giving lung cancer as well as other lung abnormality testing options.

Real Time Doctors and Hospital Information Available on App. As it may be difficult for certain users to completely trust an App for correct diagnosis, the application has also been provided the ability to send the relevant image along with the application’s diagnosis to select doctors who can then give the user a final judgment based on all the data. Analysis carried out by LungAI assists doctors in correct diagnosis. The website contains an automatic messaging system which automatically sends a message to the listed doctors in case the user wishes to. The website also helps users locate Covid-19 test sites in their vicinity.

Informative Platform. In times like these the spread of misinformation is inevitable and thus LungAI also acts a platform which updates users with the the most relevant and accurate information and news from around the world regarding the current Covid-19 pandemic. The website also provides users with interactive charts to further help users interpret information regarding Covid-19.

Cost Effective. While certain web applications may charge for each diagnosis or may start charging users after the first few free trials, LungAI is completely free for all its users and can be used any number of times. Apart from this the doctors communicate with users through email and other free communication services making the entire process cost free. The website also does not take any consultation charges from users.

Solution to lack of Testing Kits & Detection Methods. The shortage of testing kits around the world makes detection of Covid-19 almost impossible and this poses a major problem for hospitals as manufacturers around the world are unable to keep up with the demand. LungAI solves this issue as well; being a digital solution it only requires an X-Ray or CT Scan and thus is able to give a prediction within 3-5 seconds. The added advantage of this detection process is the fact that LungAI does not require any additional resources and accessible to all users and hospitals on the internet.

Non-Contact/Contactless Detection of COVID19. Due to its highly contagious nature spread of COVID19 occurs extremely rapidly leading to increased vulnerability to people in close contact with affected patients or while testing. LungAI solves this issue as it does not require any contact between medical staff and patients due to the process being completely digital and fast. Added to this is the benefit that the application gives instant detection's thus saving time as well.

LungAI aims to bring easy and accessible healthcare to all places around be it bustling areas or the most remote corners around the world. We aim to save lives all over the world irrespective of their region or financial status because being healthy is a universal right which all humans must share.

Thus in the COVID19 Space Apps Challenge, LungAI makes use of external internationally verified image datasets of COVID19 and Other Lung Abnormalities as well as NASA and other space agencies verified mapping and geospatial resources and data to give accurate and reliable predictions and analysis of currently affected patients as well as giving predictions on whether they are likely to be affected by the coronavirus based on their chest radiograph image and current location.

How I Developed This Project

The main aim while developing LungAI for me was always to help people, especially health workers. On one hand we as people are worried of even stepping out of our own houses due to fear of getting infected and yet there are also people who leave their homes each day and go help coronavirus affected patients even though they know that they are putting their health and in some cases their lives on the line. Thus I wished to develop a solution that could reduce patient contact and yet not cause a lack of reliable detection.

METHODOLOGY/PROCEDURE FOLLOWED IN DEVELOPING LungAI

Before we begin with the procedure I followed in making this website, a basic introduction to its functioning is necessary. LungAI is a Deep Learning website which takes input in the form of an image clicked using a camera from any device, compares it to a set of factors on which it has been trained and returns a probability of the respective disease ranging from 100% to 0%. It then gives the users an option to send the image along with the diagnosis to doctors registered on the application in case they wish to get a second opinion or in some cases, a confirmation.

Neural Networks. Neural networks are computer programs that are modeled on the functioning of the human brain.Like our brains, neural networks consist of a large number of neurons that when connected together can be used to perform various challenging tasks using large amounts of data.

In a neural network, the job of a neuron is to take in a couple of inputs and then give an output. Each of the inputs has a certain weight coefficient associated with them. These weights affect the output of the neuron and they are constantly changed to get an improved output (that is more accurate).

Neural networks learn how to make predictions by recognizing patterns in data. The more data we give it, the better it will be at making predictions, thus in turn improving the accuracy of the output. To find such patterns neural networks use a technique called back propagation where the networks learns from its errors and then corrects its pattern accordingly.

Convolutional Neural Networks (CNN). For the purpose of this project, I made use of Convolutional Neural Networks, a type of Deep Neural Network, most commonly applied to analyse visual imagery.Convolutional Neural Networks are different from normal neural networks because they contain a special type of layer called a Convolutional Layer, which contains a filterthat is able to understand certain types of patterns in the image.

This layer can almost be understood as a number of sieves where at each layer the pores of the sieve get smaller and smaller. The layers that are right at the start would be used for something very simple like edge detection while layers added later on can detect more complex features thus increasing complexity of the patterns it forms. All of these feature maps are put into a long list of features at the end of the network, which is used to finally classify the image.

Apart from convolutional layers, CNNs contain a couple of other layers, namely Pooling and Classification Layers.Pooling layers are immensely important in reducing the training time of your CNN. It does so by reducing dimensions of the image. It works quite similarly to a convolutional layer, where a filter passes over the image, except now, the filter passes over the data, extracts the most important information, and puts it into a smaller sized matrix. This makes it much easier for their computer to work with by reducing the amount of data to be analysed. All these layers are joined together at the end, into a Classification Layer which produces the final classification

Building the Image Classifier. To build the CNN I used native Keraswhich is an open-source neural-network library written in Python. I used Transfer Learningto create this model, which means that I took a pre-trained network called MobileNet which is trained with ImageNet, a dataset of over 14 million images, and added more layers to it so that it could classify various lung conditions including Covid-19, lung cancer and other lung based diseases. The idea is that we take a model that was trained with a huge amount of data and use that for a new task where we do not have as much data. We basically take a pre-trained model that can detect images and add a few more restrictions to the images being detected so that it can detect factors we need and ignore the ones we do not need.

The reason I chose MobileNet was mainly due to Speed, as speed in a real time usage web application is a major factor. A problem I encountered in most other pre-trained models (like VGG16, ResNet50 etc) was bad performance in not only training of the model but also in real-time prediction due to their large size in terms of memory as well as loading times.

In terms of finding a balance between a model that trains well with a decent accuracy as well as one which gives predictions fast enough in real-timeMobileNet worked the best for this project. It had the right number of training layers in terms of training with reasonable accuracy while also preventing over-fitting of data which is a common problem faced during machine learning.

Model Name Size of Input Size of Model Prediction Time

InceptionV3 8.64 KB 91 MB 3 minutes

VGG16 8.64 KB 512.2 MB 6 minutes

MobileNet 8.64 KB 27 MB 13 seconds

Note. Prediction Time includes time taken by the model to load onto the website being the backend of the web application.

To build the model I started off by importing all the libraries that I needed. These libraries included a large number of Keras imports required to construct the different parts of the CNN. I also importedNumpy, itertools and Scikit-Learn which are some of the other libraries I needed.

To start, we set some variables that have the location of the training data and evaluation data. The numbers of samples are how many images there are in the dataset we are using. The Batch Size is how many images you process before the weights of the network are updated. Size of the image that we input to the neural network is 224x224. Lastly we declare how many times the network needs to do a forward/backward pass until it has analysed every single image which we do by dividing the total number of images in our data by the batch size, and then rounding them to the nearest ten.

Another really important step we need to do is Data Pre-Processing because we are using transfer learning with MobileNet. Thus, we will need to retrain a couple of the layers in this network so that it works with radiographic lung images. As MobileNet takes input images of dimensions 224 x 224 we will need to pre-process our data to match this criterion.

The MobileNet model is modified by adding a Dropout Layer and a Dense Output Layer. We use Dropout to prevent over fitting to the data. This can happen when the network gets very good at classifying the data that you give it, but not really much else. After adding this layer, we add a final Dense Layer which has an Activation Function attached to it. This takes all of the feature maps that it has collected, and then gives us the Prediction. After we make these changes to the structure of the network, we want to freeze some of the layers to make the training times a lot faster. In this case, we freeze everything except for the last 23 layers, so our image data is trained on only the last 23 layers of MobileNet.

Now that we have fine-tuned our model, we need to create a couple of metrics to determine how accurate our model is. Then we compile all the information for our model so that it is ready to be trained. For training, we use the Adam Optimizer and the Categorical Cross Entropy Loss Function. Essentially, a Loss Function measures how wrong the model’s predictions are and the Optimizer changes the Weights, trying to make the model more accurate.

Finally, we train the model, piecing together all of the components we created above. The program saves this to a .H5 File, which is crucial if we want to implement this model.To run this on the web, I converted the Keras model to a TensorflowJS and then implemented the model by creating a Local Server from my computer using Microsoft Visual Studio Code. Embedding of the model onto the website has been done using JavaScript through online tutorials. Images of the website while working have also been uploaded along with this file.

This is also the part where we make use of the user’s current location. Based on the data provided by the COVID19 Space Apps Challenge I particularly concentrated on the mapping and geospatial resources which showed various cluster maps and population densities of affected patientsall over the world. Further I added a function to the predict javascript (predict.js) file which added a particular weight to the COVID-19 deep learning model. In this manner users that had coronavirus were predicted as positive but even those whose lungs showed certain patterns of fluid buildup as well as consolidation were predicted as having a high chance of getting affected by COVID-19.

In this manner a person who has been affected by an acute case of Pneumonia or SARS (as predicted by the Other Lung Abnormalities model on the LungAI website) and lives in an area which has very high number of COVID19 cases will be predicted as having a high probability of contracting COVID19 and thus the website will prevent this by informing the user while he has still not been affected by the virus. In this manner the web application not only detects COVID19 cases but also acts as a prevention technique which could potentially save millions of lives globally.

Space Agency's Geospatial/Mapping Resources. These resources were used for analysing data and mapping patterns for spread and rate of spread of COVID-19 among various areas as well in designating which areas should have what values of weights applied to them.

a)Census Bureau Data COVID19.

b)European Space Agency Satellites and COVID19

c)Census Bureau Data COVID19, American Community Survey

d)NASA Sedac Global COVID19 Viewer

e)NASA Black Marble

f)Geohealth Data Access and Mapping US Dept. of Health & Human Services

g)Canadian Space Agency(CSA) Open Data Portal

h)Government of Canada Open Data Portal

i)CSA COVID19 COVID19 Data

j)Government of Canada Open Information about Coronavirus and COVID19

k)Government of Canada Open Maps Portal

l)FEMA Covid-19 Geospatial Resource Centre

These mapping resources were all taken from NASA and other partner space agenices like CSA, ESA and other sample resources provided for the COVID-19 Space Apps Challenge

Image Datasets/Training Data. The above model was trained multiple time with different variations each time and various methodologies to improve the accuracy as well as compensate for the lack of enough data.

a)NIH Chest Dataset. This NIH Chest X-ray Dataset is comprised of 112,120 X-ray images with disease labels from 30,805 unique patients.There are 16 classes (15 diseases, and one for "No findings"). Images can be classified as "No findings" or one or more disease classes: Atelectasis, Consolidation, Infiltration, Pneumothorax, Edema, Emphysema, Fibrosis, Effusion, Pneumonia, Pleural Thickening, Cardiomegaly, Nodule Mass, Hernia and Tuberculosis.

b)China Set - The Shenzhen set - Chest X-ray Database. The standard digital image database for Tuberculosis is created by the National Library of Medicine, Maryland, USA in collaboration with Shenzhen No.3 People’s Hospital, Guangdong Medical College, Shenzhen, China. The Chest X-rays are from out-patient clinics, and were captured as part of the daily routine using Philips DR Digital Diagnose systems. The dataset contains 336 cases with manifestation of tuberculosis, and326 normal cases.

c)Montgomery County X-ray Set. X-ray images in this data set have been acquired from the tuberculosis control program of the Department of Health and Human Services of Montgomery County, MD, USA. This set contains 138 posterior-anterior x-rays, of which 80 x-rays are normal and 58 x-rays are abnormal with manifestations of tuberculosis. All images are de-identified and available in DICOM format. The set covers a wide range of abnormalities, including effusions and miliary patterns. The data set includes radiology readings available as a text file.

d)LIDC-IDRI Lung Cancer Dataset. The Lung Image Database Consortium image collection (LIDC-IDRI) consists of diagnostic and lung cancer screening thoracic computed tomography (CT) scans with marked-up annotated lesions. It is a web-accessible international resource for development, training, and evaluation of computer-assisted diagnostic (CAD) methods for lung cancer detection and diagnosis.

e)CORD19 and Covid-19 Image Dataset (GitHub). In response to the COVID-19 pandemic, the White House and a coalition of leading research groups have prepared the COVID-19 Open Research Dataset (CORD-19). CORD-19 is a resource of over 51,000 scholarly articles, including over 40,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. This freely available dataset is provided to the global research community to apply recent advances in natural language processing and other AI techniques to generate new insights in support of the on-going fight against this infectious disease. Apart from this I also made use of an open dataset available on github which contained images of CT Scans and X-Rays of Covid-19 affected patients as well as patients that tested negative for the Covid-19 virus.

f)SIRM Covid-19 Image Dataset/Database. Italian Society of Medical and Interventional Radiology(SIRM) consists of an openly available Covid-19 database consisting of live and confirmed cases of Italy along with relevant medical images of patients having tested positive for coronavirus. The database consists of a mix of CT Scans as well as X-Rays of the affected patients.

g)RSNA Covid-19 Image Database. RSNA stands for the Radiological Society of North America. RSNA is committed to connecting radiologists and the radiology community to the most timely and useful COVID-19 information and resources.RSNA has established two COVID-19 volunteer task forces to equip radiologists around the world with the tools they need to navigate the COVID-19 outbreak.Since early in the COVID-19 pandemic, RSNA has been a global leader in providing resources and imaging references to inform radiologists throughout the world on COVID-19.

Data Augmentation. To further improve accuracy of the model I decided to augment the data to increase the training dataset. Image Data Augmentation is a technique used to artificially expand size of a training dataset by creating modified versions of images in the dataset. For the above model, I augmented the data by cropping images, rotating images about their axis, adjusted the brightness of images etc. I augmented the data depending on the results provided by the various models when the datasets were unbalanced. Thus in case of the Covid-19 models due to lack of balanced datasets I augmented the given dataset to produce over 6000 images before retraining the model to get the desired models.

In terms of the Lung cancer model the only image set that was augmented in the Lung Cancer dataset was that of the benign image class which Iwas augmented so as to make all the image classes have equal number of images. Thus the Benign lung image class was augmented to form 8000 images in both the CT Scan and X-Ray model before training.

The lung abnormalities model had generally equal balanced image classes and thus required little change before final training. On final training of the 6 models covering the detection of Covid-19, Lung Cancer and other lung abnormalities the confusion matrix yielded a finally overall accuracy of 98% across all classes irrespective of the image input being a CT Scan or X-Ray.

Adding Weights to Increase Model Sensitivity of Lung Cancer Types.

(a) Finally, I also made use of Weights so that the model would become more sensitive to malignant cancer detection as was the case in the Lung Cancer detection models in both the CT Scan and X-Ray model. Specifically weights were added to the Malignant Primary Lung Cancer Type as well as the Malignant Metastatic Lung Cancer category to increase the overall accuracy of the model.

(b) There was also a requirement of weights that needed to be implemented due to the highly uneven distribution of images available for the lung cancer datasets. To reduce the loss of accuracy due to sheer size of certain lung cancer types, weights were imperative.

(c) To find the optimal weights to be added to each of the lung cancer subtypes, the model was trained again and again and the model which gave the optimum result was chosen to form the backend of the web application.

Adding Weights to Increase Model Sensitivity of Covid-19 Detection.

(a) In terms of adding weights to the Covid-19 model it was seen that experimentally on adding weights to particular subtypes of the image the bias was further reduced and was thus leading to the model gaining an overall increase inaccuracy of both the CT Scan and X-Ray models.

(b) After various experiments and training rounds it was found that adding a weight of 2 to the Covid-19 image types led to the highest accuracy to the model yielding an increase of over 4% in terms of accuracy in Covid-19 detection up from 95% to 99% and also led to an increase in Covid-19 negative detection in terms of the accuracy going up from 92% to 99% up by 7% overall.

Adding Weights to Increase Model Sensitivity of Lung Abnormalities.

(a) The Lung abnormalities model involves the detection and prediction of over 16 different types of lung diseases giving the user an option to give a detect it using either an X-Ray or CT Scan. But due to the large number of disease to be checked for and the dataset being unbalanced the model required the use of weights to increase accuracy.

(b) On training the model initially without use of weights there were certain disease types that showed a very high accuracy. But there were also certain diseases that seemed to share a lot of features with others leading to a drop in overall accuracy. To increase the accuracy based on various trials and experiments it was found that adding a weight of 3 to the categories of Pneumonia, Cardiomegaly & Pneumothorax led to an increase in accuracy & reliability of the model.