The objective of this project was to use computer vision and AI to automate the process of detecting when parts need to be refilled on Mack Trucks’ assembly line.
Sponsor
Mack Trucks
Team Members
Nataniel Kisten Nicholas Calabrese Andrew Heckler Rachel Brooks
Project Poster
Click on any image to enlarge.
Project Video
Project Summary
Overview
Our project will be used in Mack Truck’s assembly line to identify objects and their position on the assembly line. It will detect parts needed for a truck to continue onto the next phase of the assembly line. If the device does not detect certain parts, it will alert Mack Trucks’ IoT network and tell them that the part needs to be refilled. We are continuing the work of the Capstone team that worked on this project in Spring 2021.
Objectives
-Develop an application and GUI that auto launches upon start-up of Raspberry Pi with the required features:
– Start and stop image capture and object detection on device and display results
– Adjustable settings (resolution, brightness, contrast, resolution, framerate and rotation, object detection thresholds and boxes)
– Overall user-friendly design
– Reduce overall time for object detection results to be returned
– Device must capture and process pictures using Azure Custom Vision
– Device must have a trained AI module that performs with repeatability and accuracy
Approach
Software:
– Python3, Bash – Primary coding languages used for development
– PyQt5, Qt Designer, Figma – Tools and library used to mockup and develop updated GUI
-Azure Custom Vision, Docker – Used for training and exporting AI module used in our object detection code.
Hardware:
– Raspberry Pi 4 – single-board computer that will contain the entire program
– Raspberry Pi Camera: Connected to Raspberry Pi 4 and used to capture pictures for object detection and AI training
– Pi Foundation Display Touchscreen: 7-inch touchscreen that is used to interact with the device
– SmartiPi Touch 2 – Stand that can contain the Raspberry Pi, camera, touchscreen display and a fan to cool the CPU on the Raspberry Pi
Improving Performance:
– Original Object Detection Code:
– Total time with the original codebase to get a result was 10 seconds
– Most time is taken up with image capture and file IO operations
– 5.5 seconds for image capture and file IO operations
– Our Design:
– Switched from saving images as PNG to JPEG
– Smaller file size
– Rather than saving to a folder saved on our main disk storage, we used one saved in memory for a ~30x speed up
Outcomes
– Reduced overall time for object detection from 10 seconds to 5 seconds
– Created an application and GUI that auto launches upon start-up of Raspberry Pi with the required features
– Device can capture and process pictures using Azure Custom Vision with adjustable camera settings and improved speed
– Has a trained AI module that performs with repeatability and accuracy