Responding to health crises requires the deployment of accurate and timely situation awareness. Understanding the location of geographical risk factors could assist in preventing the spread of contagious diseases and the system developed, Covid ID, is an attempt to solve this problem through the crowd sourcing of machine learning sensor-based health related detection reports. Specifically, Covid ID uses mobile-based Computer Vision and Machine Learning with a multi-faceted approach to understanding potential risks related to Mask Detection, Crowd Density Estimation, Social Distancing Analysis, and IR Fever Detection. Both visible-spectrum and LWIR images are used. Real results for all modules are presented along with the developed Android Application and supporting backend.
The initial development of two First-Person Perspective Video Activity Recognition Systems is discussed. The first system, the First Person Fall Detection or UFall, can be used to recognize when a person wearing or holding the mobile vision system has fallen. The problem of fall detection is tackled from the unique first-person perspective. The second system, the directed CrossWalk System (UCross), involves detection of the user movement across a crosswalk and is intended for use in helping a low vision person navigate. In both cases, the user is wearing or holding the camera device for purposes of monitoring or inspection of the environment. This first-person perspective yields unusual fall data and this is captured and used for the creation of a fall detection system. For both systems Machine Learning is employed using video input to trained Long-Term Short-Term (LSTM) Networks. These first-perspective video activity recognition systems use the Tensorflow framework  and is deployed using mobile phones for proof of concept. These applications could be useful for low vision people and in the case of fall detection for senior citizens, police, construction and other inspection-oriented jobs to help users who have fallen. The success and challenges faced with this unique first-person perspective data are presented along with future avenues of work.
ULearn is a system that uses deep learning, computer vision and NLP to assist students with the task of web-based learning. ULearn’s goal is to detect when the student is experiencing higher levels of frustration and then present alternative meaningful alternative content. The ULearn app features a web-brower, though the intention is to have ULearn assist students in learning scenarios it is equally applicable to other web-based tasks. While the user is browsing, ULearn monitors them using the front-facing camera and when negative emotions are detected the user is presented with a set of “tips”. The first step in ULearn is to perform face detection which returns an ROI that is fed into an emotion detection system. A deep-learning CNN is used to perform the emotion detection yielding one of anger, fear, disgust, surprise, neutral, and happy. If a significate negative emotion is detected ULearn generates a set of alternative content called “tips” which are a set of links to similar content web pages to the current one being viewed. These links can be found through scraping the currently viewed web page for content that is used directly in a search or first passing this information to an NLP stage. The NLP stage gives the saliency of the most prominent entities in the current web page content. Real test results are given, and the success and challenges faced by ULearn are presented along with future avenues of work.