Project Team
Students
Khoa Nguyen
Computer Science
Abington
Xiang Liu
Computer Science
Abington
Faculty Mentors
Yi Yang
Abington
Electrical Engineering
Project
https://sites.psu.edu/mcreu/files/formidable/2/2024-07-22/poster.pdf
Project Video
Project Abstract
This project introduces a novel application leveraging Retrieval Augmented Generation (RAG) technology to enhance accessibility for individuals with visual impairments, specifically in museum environments. The project develops a Q&A chatbot designed to assist blind visitors in querying specific artworks.
The chatbot utilizes advanced technologies including Langchain, VLLM, Mistral, and Activation-aware Weight Quantization (AWQ) for efficient model inference and operation on low-end hardware. Key components of the system include FAISS for document retrieval and Docker for versatile deployment.
The client-side implementation, designed for Raspberry Pi, integrates Whisper for robust speech-to-text conversion and Piper for natural text-to-speech synthesis, facilitating seamless user interaction via voice commands and responses.
Preliminary testing demonstrates the chatbot’s ability to provide contextually relevant responses based on user queries, sourced from curated documents within the FAISS database. Future work will focus on optimizing performance, completing client-side integration, and conducting extensive user testing to validate usability and functionality across diverse museum settings.
Evaluate this Project
Use this form link to provide feedback to the presenters, and add your project evaluation for award(s) consideration.