Eyes-Free Vision-Based Scanning of Aligned Barcodes and Information Extraction from Aligned Nutrition Tables

Abstract

Independent grocery shopping is one of the biggest challenges faced by visually impaired (VI) individuals. VI individuals may be able to get to a store on their own by using public transportation or by walking but are unable to shop there independently. Some of the problems that they face after getting to the store include long wait times to get an employee to assist them or getting a store employee who is not familiar with the store layout, gets irritated with long searches, or does not possess the required English skills. These problems ultimately result in VI shoppers having to abandon independent shopping altogether and instead rely on friends and family for their shopping needs. Assisitive shopping systems can help VI individuals shop independently by helping them in areas such as store navigation, product retrieval, etc. Many assistive shopping systems have been developed but they usually rely on dedicated hardware or instrumenting the store with RFID tags, etc. Unlike these systems, ShopMobile 2 is a software-only solution that only uses fast computer vision algorithms running on a smartphone. ShopMobile 2 consists of three modules - an eyes free barcode scanner, an optical character recognition (OCR) module, and a tele-assistance module. The eyes-free barcode scanner allows VI shoppers to locate and retrieve products by scanning barcodes on shelves and on products. The OCR module allows shoppers to read nutrition facts on products and the tele-assistance module allows them to obtain help from sighted individuals at remote locations. This dissertation discusses, provides implementations of, and presents laboratory and real-world experiments related to all three modules