Mem - Temporal Video Intelligence System

Capture everything. Store efficiently. Process intelligently.

Mem is a production-ready video processing system that captures frames and transcriptions from video files and live streams, storing them with absolute UTC timestamps. Built for continuous recording scenarios (security cameras, screen recording, life logging), Mem achieves 95% storage reduction through intelligent frame deduplication while maintaining instant temporal access to any moment.

🎯 Why Mem?

Traditional video storage is expensive and inefficient. A 24/7 security camera generates ~2.4GB per day, but 95% of frames are redundant. Mem solves this by:

Deduplicating identical frames using perceptual hashing (120MB vs 2.4GB per day)
Anchoring everything to UTC time for precise temporal queries
Separating capture from analysis - store once, process many times
Supporting both files and live streams from OBS Studio or IP cameras
GPU-accelerated transcription with local Whisper models (no cloud dependencies)

✨ Key Features

📊 95% Storage Reduction - Perceptual hash deduplication for static scenes
⏰ Temporal Architecture - All data anchored to absolute UTC timestamps
🔴 Live Streaming - RTMP server for OBS Studio and IP cameras
🎯 GPU Acceleration - CUDA-enabled Whisper transcription
🗄️ Self-Contained - Frames stored as BLOBs in DuckDB, no filesystem dependencies
🔒 Privacy-First - All processing happens locally, no cloud APIs
🚀 Production-Ready - Docker deployment with health checks and monitoring

🚀 Quick Start

Option 1: Local Development

# Clone repository
git clone <repository-url>
cd mem

# Start full stack (backend + frontend)
./start-mem.sh

# Access the application
# Frontend: http://localhost:3000
# Backend API: http://localhost:8000
# API Docs: http://localhost:8000/docs

Option 2: Docker Deployment

# Start all services with Docker Compose
docker-compose up -d

# Services will be available at:
# Frontend: http://localhost
# Backend API: http://localhost:8000
# RTMP Stream: rtmp://localhost:1935/live

Option 3: Manual Setup

# Backend
cd mem
uv sync
uv run uvicorn src.api.app:app --reload --port 8000

# Frontend (new terminal)
cd mem-ui
npm install
npm run dev

# Access at http://localhost:5173

📹 Processing Videos

Important: Filename Format

Videos MUST be named YYYY-MM-DD_HH-MM-SS.mp4 (UTC timestamp when recording started)

# Process a video via API
curl -X POST http://localhost:8000/api/capture \
  -H "Content-Type: application/json" \
  -d '{"filepath": "/path/to/2025-08-22_14-30-45.mp4"}'

# Or use the web UI
# 1. Navigate to http://localhost:3000
# 2. Click the Upload button
# 3. Select your video file

🔴 Live Streaming with OBS

1. Create Stream Session (Web UI)

Open http://localhost:3000
Click "Streams" button in header
Click "Create New Stream"
Copy the stream key

2. Configure OBS Studio

Settings → Stream
Service: Custom
Server: rtmp://localhost:1935/live
Stream Key: (paste from UI)

3. Start Streaming

Click Play button on stream card in UI
Start streaming in OBS
Watch real-time stats and deduplication

📊 Architecture Overview

┌──────────────────────────────────────────────────────┐
│                   Input Sources                       │
│  ┌────────────┐        ┌─────────────────────────┐  │
│  │Video Files │        │ RTMP Streams (OBS/IP)   │  │
│  └─────┬──────┘        └───────────┬─────────────┘  │
└────────┼───────────────────────────┼─────────────────┘
         │                           │
         ▼                           ▼
┌──────────────────────────────────────────────────────┐
│               Capture Pipeline                        │
│  ┌─────────────┐    ┌──────────────────────────┐   │
│  │Frame Extract│───▶│ Perceptual Hashing (95%  │   │
│  │  (5 sec)    │    │ deduplication)           │   │
│  └─────────────┘    └──────────────────────────┘   │
│  ┌─────────────┐    ┌──────────────────────────┐   │
│  │Audio Extract│───▶│ Whisper Transcription    │   │
│  │             │    │ (GPU Accelerated)        │   │
│  └─────────────┘    └──────────────────────────┘   │
└────────────────────────┬─────────────────────────────┘
                         ▼
┌──────────────────────────────────────────────────────┐
│              DuckDB Storage Layer                     │
│  ┌─────────────────────────────────────────────┐   │
│  │ Tables:                                      │   │
│  │ • unique_frames (BLOB storage)              │   │
│  │ • timeline (temporal index)                 │   │
│  │ • transcriptions (Whisper output)           │   │
│  │ • sources (videos/streams metadata)         │   │
│  └─────────────────────────────────────────────┘   │
└────────────────────┬─────────────────────────────────┘
                     ▼
┌──────────────────────────────────────────────────────┐
│                  Access Layer                         │
│  ┌──────────┐   ┌──────────┐   ┌──────────────┐   │
│  │ REST API │   │  Web UI  │   │ Future: AI    │   │
│  │  (8000)  │   │  (3000)  │   │  Processing   │   │
│  └──────────┘   └──────────┘   └──────────────┘   │
└──────────────────────────────────────────────────────┘

🛠️ Development

Prerequisites

Python 3.9 (exact version required)
Node.js 18+ and npm
FFmpeg for audio/video processing
CUDA toolkit (optional, for GPU acceleration)

Development Commands

# Backend
cd mem
uv sync                     # Install dependencies
make format                 # Auto-format code
make lint                   # Run linters
uv run pytest tests/        # Run tests

# Frontend
cd mem-ui
npm install                 # Install dependencies
npm run dev                 # Start dev server
npm run test               # Run tests
npm run build              # Build for production

# Full Stack
./start-mem.sh             # Start everything
./stop-mem.sh              # Stop everything

Using Nix (Recommended)

# Enter development environment with all dependencies
nix develop

# Run commands with guaranteed dependencies
nix develop -c uv run uvicorn src.api.app:app --reload

📈 Performance Metrics

Metric	Value	Notes
Storage Reduction	95%	Via perceptual hashing
Daily Storage	~120MB	For 24/7 recording
Query Latency	<100ms	Time-range searches
Frame Processing	30 fps	On modern hardware
Stream Latency	<1 second	RTMP to database
Max Concurrent Streams	10	Configurable

🔧 Configuration

Backend (`mem/config.yaml`)

capture:
  frame_interval_seconds: 5    # Extract frame every N seconds
  jpeg_quality: 85             # JPEG compression (1-100)

whisper:
  model: "base"                # tiny/base/small/medium/large
  device: "cuda"               # cuda or cpu
  compute_type: "float16"      # float16 (GPU) or int8 (CPU)

streaming:
  rtmp:
    port: 1935
    max_concurrent_streams: 10

🐳 Docker Deployment

Production Deployment with GPU

# Build and push images
./deploy/build-and-push.sh

# Deploy to server
./deploy/deploy-to-wise18.sh

# Or manually with docker-compose
docker-compose -f docker-compose.yml up -d

GPU Requirements

NVIDIA GPU with CUDA support
nvidia-docker runtime installed
4GB+ VRAM for optimal performance

📚 API Documentation

Interactive API documentation available at http://localhost:8000/docs

Core Endpoints

# Process video
POST /api/capture
{"filepath": "/path/to/video.mp4"}

# Search timeline
GET /api/search?type=timeline&start=2025-01-01&end=2025-01-02

# Get system status
GET /api/status

# Stream management
POST /api/streams/create
GET /api/streams
POST /api/streams/{stream_key}/start

🧪 Testing

# Run all tests
cd mem && uv run pytest tests/

# Run with coverage
uv run pytest tests/ --cov=src --cov-report=html

# Frontend tests
cd mem-ui && npm run test

📦 Project Structure

mem/
├── mem/                    # Backend (Python/FastAPI)
│   ├── src/
│   │   ├── api/           # REST API endpoints
│   │   ├── capture/       # Frame/audio extraction
│   │   └── storage/       # DuckDB operations
│   └── tests/             # Unit tests
├── mem-ui/                 # Frontend (React/TypeScript)
│   ├── src/
│   │   ├── components/    # React components
│   │   ├── hooks/         # Custom hooks
│   │   └── utils/         # API client
│   └── tests/             # Frontend tests
├── rtmp/                   # RTMP streaming server
├── deploy/                 # Deployment scripts
└── docker-compose.yml      # Container orchestration

🚧 Current Limitations

Videos must follow YYYY-MM-DD_HH-MM-SS.mp4 naming convention
No built-in authentication (use reverse proxy)
Synchronous video processing (async coming soon)
Maximum 10 concurrent streams (configurable)

🗺️ Roadmap

Authentication and multi-user support
Async video processing with job queue
Cloud storage backends (S3, GCS)
AI-powered scene analysis
Mobile app for viewing
Export to standard formats
Kubernetes deployment

🤝 Contributing

Contributions are welcome! Please:

Fork the repository
Create a feature branch
Make your changes
Ensure tests pass (make test)
Format code (make format)
Submit a pull request

📄 License

MIT License - See LICENSE file for details

🙏 Acknowledgments

OpenAI Whisper for transcription
DuckDB for time-series storage
FastAPI for the backend framework
React for the frontend framework

Built with ❤️ for continuous recording and temporal intelligence

Name		Name	Last commit message	Last commit date
Latest commit History 7 Commits
deploy		deploy
mem-ui		mem-ui
mem		mem
rtmp		rtmp
.env.production		.env.production
CLAUDE.md		CLAUDE.md
README.md		README.md
docker-compose.yml		docker-compose.yml
start-mem.sh		start-mem.sh
stop-mem.sh		stop-mem.sh

sdelcore/mem

Folders and files

Latest commit

History

Repository files navigation