Prof. Emily Mower Provost receives NSF grant for research in personalized emotion recognition

While much work has been done in the field of emotion recognition, the field is almost exclusively focused on estimating how an outside audience would interpret speech. As researchers further develop this technology and move into new territory tracking mental health symptom severity, the focus is now being turned to self-reported experiences. In a new project recently awarded an NSF grant, Professor Emily Mower Provost is working to create new and personalized speech recognition techniques that monitor changes in subjects’ emotions.

The project, titled “RI: Small: Advancing the Science of Generalizable and Personalizable Speech-Centered Self-Report Emotion Classifiers,” aims to advance the state-of-the-art in robust and generalizable personalized speech self-report emotion recognition classifiers and to investigate how measures created using these classifiers will allow researchers to intuit changes in symptom severity in individuals with suicidality.

Mower Provost said, “We are very excited to have the opportunity to pursue this line of research.  Many mental health conditions are associated with changes in emotion.  Personalized emotion recognition methods offer new approaches to investigate these phenomena.”

In transitioning from a focus on outsider perspectives to individuals’ interpretations of their own emotional experiences, Mower Provost and her team hope to identify changes in depression severity without requiring the element of active participation, which requires individuals to describe their symptom severity multiple times per day, often causing increased cost and participant burden. Additionally, the project aims to tackle the challenges associated with accurately estimating self-reported emotion, including cognitive bias, context, and the difference between self-report and emotional experiences, by investigating the use of passively collected audio data.

The team outlines their objectives as “creating classifiers that are robust and generalizable using new metrics that encourage models to attend to the same acoustic and language cues as human observers; personalizing classifiers to users longitudinally and evaluating the effectiveness of self-report emotion classifiers by predicting changes in mental health symptom severity using an existing real-world dataset annotated with mental health symptom severity.” The project’s goal of refocusing automatic emotion classification on users themselves will hopefully lead to further investigations in identifying health risk factors using speech recognition.

Prior to joining the faculty at Michigan, Mower Provost received her PhD in Electrical Engineering from the University of Southern California (USC), Los Angeles, CA in 2010. She was awarded a National Science Foundation Graduate Research Fellowship, the Herbert Kunzel Engineering Fellowship from USC, an Intel Research Fellowship, and the Achievement Rewards For College Scientists (ARCS) Award. She is currently a member of Tau-Beta-Pi, Eta-Kappa-Nu, and a member of ACM, IEEE, and ISCA. She was named Toyota Faculty Scholar in 2020 for her accomplishments in the field.