Redis

Faktory 1.0 will switch from RocksDB storage to an "embedded" version of Redis. Redis gives us several advantages over RocksDB:

• Copious documentation - RocksDB is a black box and hugely painful to configure correctly. No one outside of Facebook knows how to properly administer it.
• Go-native - RocksDB is C++ and requires CGo to compile, making binaries larger and platform-specific. Builds will be much faster and easier to debug with Redis.
• Real-time replication - RocksDB supports point-in-time backups but does not provide real-time stream of changes to a remote replica. This makes it easy to lose data if running in an unreliable environment (e.g. most cloud systems).

It is possible replication will be a commercial feature of Faktory Pro. My thinking: if you need high availability and reliability, you should have a budget for good tools and support.

Drawbacks:

• The Redis command set is not specifically tuned for Faktory so some current Faktory features are harder than necessary or will need to be removed. A reliable pop operation from multiple queues is overly difficult, job prioritization is a O(log N) operation not O(1), etc.
• Performance is harder to ensure if using a non-localhost Redis.

Implementation

• Faktory will require a named socket connection to Redis, ensuring that Redis is running on localhost and therefore guaranteed to be fast.
• Redis 3.2+ is required.

Configuration

• faktory_redis_dir - directory which holds the RDB, defaults to /var/lib/faktory/default/
• faktory_redis_sock - location of named socket, defaults to /var/run/faktory/default.sock
• faktory_redis_bin - location of Redis binary, defaults to searching PATH

Faktory will start the Redis binary itself with a custom redis.conf. Faktory will not use an existing, running Redis instance; this is so Faktory controls its own Redis and access to the data. I want to avoid shared tenancy, misconfigurations and any associated support issues.

You will still be able to connect redis-cli to the named socket to muck about but beware that Faktory is running too so data can change in real-time.