Project Team
Students
Ishita Sinha
Computer Science
Penn State Harrisburg,Penn State University Park
Faculty Mentors
Dr. Truong Xuan Tran
Penn State Harrisburg
Department of Computer Science
Project
Project Video
Project Abstract
Teletherapy or online therapy, has been becoming increasingly popular over the last decade, and especially post Covid-19 pandemic, due to its accessibility as well as convenience. However, teletherapy also presents its own unique challenges. One significant challenge faced by therapists across the world is the difficulty of perceiving non-verbal cues remotely or through a screen. The limited view of the client, and the absence of physical presence can make it harder for therapists to pick up on subtle non-verbal signals, which includes facial expressions, gestures or body language, tone of voice etc. These provide valuable insights into patient’s emotions and state of mind and associated psychological disorders and forms the base of accurate assessment and intervention plan by therapists.
The research project aims at interpreting one of the key Non-Verbal cue, ‘FACIAL EXPRESSIONS’ in teletherapy by utilizing deep learning technology. It analyzes video and images inputs during therapy sessions to detect and interpret non-verbal cues and provides real time feedback to therapists during live session. As part of the research project, an emotion recognition model was developed using deep learning techniques like transfer learning with the FER2013 and CK+ datasets. Transfer learning was employed with MobileNetV2 and ResNet50V2 as base models using pre-trained ImageNet weights. The outputs of these models were concatenated and fine-tuned with fully connected layers using batch normalization and dropout. The combined model, trained with the Adam optimizer and a learning rate of 0.0001, effectively detected emotions in real-time using a webcam, providing live feedback on detected emotions. For example, the model is able to pick up cues from facial expressions, and indicate emotional states or psychological disorders like stress, anxiety and depression.
Evaluate this Project
Use this form link to provide feedback to the presenters, and add your project evaluation for award(s) consideration.