Redis Migration Toolkit

A pair of Python CLI scripts to migrate all keys (including RedisJSON) from a remote Upstash Redis instance into a local Redis server. Includes both a simple, straightforward version and a high-speed, pipelined version to suit different needs.

---

Prerequisites

• Python 3.7+
• A local Redis server running (default: localhost:6379, DB 0)
• An Upstash Redis database (host, port, and password)
• pip

---

Installation

1. Clone this repo:

   git clone https://github.com/technance-foundation/redis-migration-toolkit.git
   cd redis-migration-toolkit

2. Install dependencies:

   pip install redis python-dotenv

3. Copy and edit the example environment file:

   cp .env.example .env
   # then edit .env to set UPSTASH_HOST, UPSTASH_PORT, UPSTASH_PASSWORD

4. Run Redis Stack & RedisInsight via Docker

   # Pull & run Redis Stack server
   docker pull redis/redis-stack:latest
   docker run -d \
     --name redis-stack-server \
     -p 6379:6379 \
     redis/redis-stack:latest
   
   # Pull & run RedisInsight UI
   docker pull redis/redisinsight:latest
   docker run -d \
     --name redisinsight \
     -p 5540:5540 \
     redis/redisinsight:latest
   
   # Open the GUI:
   http://localhost:5540

---

Configuration

Your .env file should look like this (no quotes):

UPSTASH_HOST=your_upstash_host_here
UPSTASH_PORT=6379
UPSTASH_PASSWORD=your_upstash_password_here

---

Scripts

1. Basic Migration

File: migrate_redis.py Use when: You need a quick, minimal-overhead migration without special optimizations.

python migrate_redis.py

What it does:

• Uses SCAN to iterate keys in batches.
• For non-JSON keys, runs DUMP + RESTORE.
• For JSON keys, runs JSON.GET + JSON.SET.
• Transfers TTLs.

2. High-Speed Migration

File: migrate_redis_batch.py Use when: You want maximum throughput via pipelining.

python migrate_redis_batch.py

Key differences:

• Batch size is configurable via the BATCH_SIZE constant.
• Pipelines all TYPE calls → splits keys into JSON vs. other → pipelines GET/DUMP + TTL → pipelines RESTORE/SET back into local Redis.
• Progress indicator per batch + total keys migrated.

---

Usage Examples

1. Run basic migration

   python migrate_redis.py

2. Run high-speed migration (with custom batch size)

   # edit BATCH_SIZE in migrate_redis_batch.py, then:
   python migrate_redis_batch.py

3. Migrate a different logical DB

   • By default both scripts target DB 0.
   • To migrate DB 1, modify the LOCAL_DB = 1 (and pass db=1 to migrate_db) in the script.

---

Troubleshooting

• Connection errors:

  • Verify your .env values.
  • Ensure your local Redis is running (redis-cli ping returns PONG).

• Missing RedisJSON support:

  • Make sure your local Redis has the RedisJSON module installed (e.g. via Redis Stack).

---

Tip: If you only need occasional, small-scale migrations, stick with the basic script. For large databases or frequent runs, the high-speed variant can cut migration time significantly.