VIsION
Wearable AI for the Blind  

  OCT 2019 WINNER
Awesome Foundation's Awesome Disability Chapter  
( click photo to see this project in Awesome Foundation Disability Chapter's page )

  PROJECT SUMMARY

  This open source, do-it-yourself project aims to
  provide blind and visually impaired users with   
  a sustainable, fully customizable solution for their indoor and outdoor day-to-day activities ...  

  PROJECT STATUS

      Click This Photo to Watch Live Video Demo of Our 1st 3D Printed Prototype  

  I'm Marx, a completely blind developer. And I created this preliminary prototype.  
  This is a pair of 3D printed eyeglasses with electronic components. And it's powered by ML and DL models.  
  I call it VIsION AI Glasses for the Blind ... 

      Watch Live 90-Second Video Demo of Raw VIsION AI Glasses for the Blind

  This video is for San Francisco Lighthouse for the Blind's annual Holman Prize 2019 competition.  
  And this project was still in its infancy at that time ...  

      Watch Concept Video of VIsION AI Glasses for the Blind

  This video shows the earliest target objectives of this project.  

  PROJECT REQUIREMENTS

  

• SBC:  Tested on Intel Compute Stick (model with CORE M3 CPU), Liva Q, Raspberry Pi Zero WH, 3B+ and 4 model B
  
• PLATFORM:  Tested on Microsoft Windows 10, Raspbian Stretch and Buster, Ubuntu 1809 & Android X86
  
• PYTHON:  Tested on version 3.5 and 3.6
  
• EXTERNAL HARDWARE:  Tested with USB 2.0 and 3.0 UVC 8Mp camera board with autoFocus Lens and built-in mic, Devantech SRF10 ultrasonic ranger, USB-ISS and GPIO-to-USB board, generic GPS module, bone-conduction speaker pads and stereo amplifier board, mini vibrating disk and DRV2605 microcontroller, USB soundcard, 4-port USB hub, generic programmable Bluetooth button and LiFePo4 batteries

  QUICK START

  

1.  Install FFMPEG and Screen-Capture-Recorder (used for video and audio recording modes), ImageMagick for Linux or Microsoft Windows 10 (used for OCR image post processing), Skype v8.53.0.85 for Microsoft Windows 10 (used for manual visual assistance mode, currently working on video calling for Linux), Mozilla Firefox v68.2.0 and a few other tools for Linux (follow comments in vision.py file);
  
2.  Open Firefox. Go to about:config, search and set media > navigator > disabled to True, and toolkit > startup > max_resumed_crashes to a fairly large integer. Afterwards, install the Auto Fullscreen extension;
  
3.  Make sure the tools above are in the system path;
  
4.  Clone this repository ( git clone https://github.com/MXGray/VIsION.git );
  
5.  Go to cloned repo ( cd /path-where-you-cloned-this-repo/VIsION );
  
6.  Run python -m pip install -r requirements.txt;
  
7.  Inside your cloned repo's root directory, you'll see the freeware SeeingWithSound© vOICe offline Web app as a complete HTM directory with an HTML file. The vOICe Java applet / application is written by Peter B.L. Meijer. You can open Firefox and go here to install this freeware as an offline Web app. And when you want to use this freeware with this 3D printed eyeglasses, then don't forget to update the necessary directory and HTML filename in the vision.py file;
  
8.  Download a pre-trained Caffe model here, store it in multiobj/ade20k/ and rename the caffemodel and prototxt files ( should be dn.caffemodel and dn.prototxt );
  
9.  Download a pre-trained Mask-rCNN model here and store it inside multiobj, along with pre-trained Caffe models for face detection, age and gender estimation (folders inside multiobj are labeled);
  
10.  OR, just use re-trained models (via transfer learning) bundled with this repo, except for Ade20K and Mask-rCNN ( download here and here, as they're too big for Github, put dn.caffemodel inside multiobj/ade20k and navnet.pb in multiobj );
  
11.  NOTE: You can download here both re-trained models, along with FFMPEG 4.1.1 and Skype 8.53.0.85 ( already included in this repo are other required Windows tools ); and
  
12.  Finally, run python vision.py ... 

  NOTE:  3D printing source files are included in this repo ...  

  HOW IT WORKS

  

1. 3D SOUNDSCAPE:  Freeware SeeingWithSound© vOICe using bone conduction speaker pads) for "visual" interpretation of scenes and objects through trained neural auditory synthesis for those who became blind from the age of 7 and above;
  
2. PRE-TRAINED ML & DL MODELS:  Local and server-hosted machine learning and deep learning models For describing visual scenes and objects with its clockface locations, for converting text in physical objects like books and signboards, for detecting persons and their current actions along with estimating their age and gender, and 10x much faster processing when online;
  
3. DISTANCE SENSING:  Audible & Haptic feedback when detecting distance of objects within user's central view (up to 20 feet);
  
4. MANUAL VISUAL ASSISTANCE:  On-demand access to a remote crowdsourced team of work-at-home sighted persons with disabilities like mobility-impaired individuals who as pro agents will provide manual visual assistance services for a reasonable per-minute fee (helpful during complex situations);
  
5. WEARABLE COMPUTER:  Use it for your productivity or recreational needs. Install any compatible software for your preferred platform, including a screenreader. Tested on full Windows 10, Linux & Android X86;
  
6. TURN-by-TURN GPS NAVIGATION (to-do):  Offline audible GPS location data;
  
7. DO-IT-YOURSELF:  Anybody can build, customize and extend the features of this tech; and
  
8. OPEN SOURCE (GNU GPLv3 license):  With the exception of packaging any work that derives or is adapted from this project as proprietary or closed source, anybody can modify, distribute, sell, freely give away or adapt from this tech for any other use case or for creating new solutions for other groups, i.e. For deaf users (convert camera input of hand gestures to text, convert voice input and camera input of lip movement of others around the deaf person into text and display by micro projector, manual remote interpreting assistance, etc.), or for non-disabled users (wearable computer with micro projector display or oLED lens, camera glasses, Bluetooth glasses for calls and audio playback, etc.) ...

  PROJECT PHOTOS

  

RAW PROTOTYPE	3D PRINTED PROTOTYPE
Front (Worn by User	Front & Center (Worn by User
Front Right (Worn by User)	Right View (Worn by User)
Top (RPi Version)	Back View (Worn by User)
Top (ICS Version)
Outer Front
Inner Front Side
Outer Left Arm
Inner Left Arm
Outer Right Arm
Inner Right Arm
Bone-Conducting Speaker Placement

  VALIDATION PROCESS

  

1. CONCEPT VALIDATION:  The Department of Science & Technology of the Philippines gave us a 2-year R&D software development grant for our project. This was after we presented our proof of concept prototype to their panel of scientists, engineers and business professionals in 2017;
  
2. TECH VALIDATION:  Entered the earliest version of our prototype to the 2019 Microsoft HackerEarth Azure AI Hackathon. Our project was hand-picked for the finals round, out of more than 760+ professional dev't teams in New Zealand, South Korea and the rest of South East Asia. This was after their panel of software engineers inspected our prototype; and
  
3. NICHE MARKET VALIDATION:  We entered our project to San Francisco Lighthouse for the Blind's 2019 Holman Prize competition. Our project was inspected by a panel of esteemed blind and visually impaired judges. They hand-picked our project for the semi-finals round, out of over 100 entries from across 6 continents ...

  Sept 15 2019 Update:  Mainly focused on hardware assembly and 3D printing for the past couple of months. Will update this repo with latest software code, hardware source files and circuitry schematics once hardware prototype design has been finalized for both plastic injection molding and 3D printing ...