VisionMate is an innovative screen-reader friendly web application designed to help individuals with visual impairments explore and navigate new places with confidence. Through natural conversation and real-time location awareness, VisionMate provides detailed descriptions of surroundings and helps users discover new destinations.
- Natural Conversation Interface: Engage in natural dialogue to discover new places and get recommendations
- Live Location Awareness: Receive real-time descriptions of notable places as you walk by
- Voice-First Design: Fully accessible through voice commands and screen readers
- Smart Recommendations: Context-aware suggestions based on user preferences and location
- Voice Memo System: Record and share location-specific hazards or points of interest
- Frontend: HTML5, CSS3, JavaScript
- Backend: Express
- Speech Recognition: Web Speech API
- Location Services: Geolocation API
- APIs:
- Yelp Fusion API for business data
- OpenAI Whisper for speech-to-text
- Custom LLM for natural language understanding
- Clone the repository
git clone https://github.com/yourusername/visionmate.git
cd visionmate- Install dependencies
npm install- Install Whisper (Required for voice processing)
pip install -U openai-whisper- Install ffmpeg For Ubuntu/Debian:
sudo apt update && sudo apt install ffmpegFor other platforms, see the ffmpeg installation guide.
- Set up environment variables
Create a
.envfile in the root directory:
YELP_API_KEY=your_yelp_api_key
OPENAI_API_KEY=your_openai_api_key- Start the server
npm startVisit http://localhost:3000 in your browser.
To make VisionMate accessible from other devices:
- Install ngrok:
npm install -g ngrok- Create a tunnel:
ngrok http 3000- Offline mode support
- Multi-language support
- Community-driven hazard reporting system
- Integration with public transit APIs
- Customizable voice profiles
- OpenAI for Whisper speech recognition
- Yelp for their comprehensive business data API
- The accessibility community for valuable feedback and guidance