ULearn: understanding and reacting to student frustration using deep learning, mobile vision and NLP

Lynne Grewe; Chengzhi Hu

doi:10.1117/12.2518262

7 May 2019 ULearn: understanding and reacting to student frustration using deep learning, mobile vision and NLP

Lynne Grewe, Chengzhi Hu

Proceedings Volume 11018, Signal Processing, Sensor/Information Fusion, and Target Recognition XXVIII; 110180W (2019) https://doi.org/10.1117/12.2518262
Event: SPIE Defense + Commercial Sensing, 2019, Baltimore, MD, United States

Abstract

ULearn is a system that uses deep learning, computer vision and NLP to assist students with the task of web-based learning. ULearn’s goal is to detect when the student is experiencing higher levels of frustration and then present alternative meaningful alternative content. The ULearn app features a web-brower, though the intention is to have ULearn assist students in learning scenarios it is equally applicable to other web-based tasks. While the user is browsing, ULearn monitors them using the front-facing camera and when negative emotions are detected the user is presented with a set of “tips”. The first step in ULearn is to perform face detection which returns an ROI that is fed into an emotion detection system. A deep-learning CNN is used to perform the emotion detection yielding one of anger, fear, disgust, surprise, neutral, and happy. If a significate negative emotion is detected ULearn generates a set of alternative content called “tips” which are a set of links to similar content web pages to the current one being viewed. These links can be found through scraping the currently viewed web page for content that is used directly in a search or first passing this information to an NLP stage. The NLP stage gives the saliency of the most prominent entities in the current web page content. Real test results are given, and the success and challenges faced by ULearn are presented along with future avenues of work.

Conference Presentation

Citation Download Citation

Lynne Grewe and Chengzhi Hu "ULearn: understanding and reacting to student frustration using deep learning, mobile vision and NLP", Proc. SPIE 11018, Signal Processing, Sensor/Information Fusion, and Target Recognition XXVIII, 110180W (7 May 2019); https://doi.org/10.1117/12.2518262

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available