Add Redis Cluster support

[ptaoussanis]

@bpoweski Hi Ben, just pinging you to reopen this - didn't realise GitHub had automatically closed the last PR (I wish they wouldn't close issues automatically).

Salvatore's just posted on multi-key Cluster support: https://groups.google.com/forum/#!msg/redis-db/R7hKYR5iBuo/8owvFFhm84sJ

Shouldn't require much (anything?) from our end.

If you're still interested in finishing up the implementation, just let me know if there's anything I can assist with.

Cheers! :-)

[bpoweski]

Definitely interested, just fighting the time constraints.

I'm tracking and rebasing against dev on my fork. So don't feel like you need to wait for me if want to make progress on it.

[ptaoussanis]

Absolutely no rush or stress - please take your time!

I'm tracking and rebasing against dev on my fork.

Sure. You'll notice the old cluster branch is gone and merged into dev. It should be stable from here - all the protocol stuff that needed reworking for Cluster is done. The taoensso.carmine.cluster namespace is basically a skeleton with some notes. A lot of your code should be transplantable there.

So don't feel like you need to wait for me if want to make progress on it.

No, no - not at all. My attention's elsewhere atm and I'm not using Cluster myself so this isn't something I'll be hacking on. The initial burst in activity was just to get the protocol stuff in the right state for supporting Cluster pipelines.

[ptaoussanis]

Quick update: discovered a minor bug with the new protocol, just pushed a fix to dev branch. The change is purely an implementation detail: won't interfere with the Cluster API at all.

[ptaoussanis]

Update: going to be cutting a new stable release soon with all the changes in the dev branch so far.

[bpoweski]

Sounds great. I'll pull in the changes.

[shmish111]

@bpoweski did you get anywhere with this?

[bpoweski]

@shmish111 I was able to get a proof of concept working but then we went a different route for the project where I planned to use it.

Coincidently, we're currently building out a twemproxy + docker + consul environment as it fits some of our goals better.

[shmish111]

@bpoweski how are you doing the sharding, your own consistent hashing?

[bpoweski]

@shmish111 We plan to use the hashing mechanism provided in twemproxy along with pre-sharding our Redis instances. One of the biggest disadvantages to this approach is the management cost of the numerous shards and how to manage the migration to different hosts as you need more RAM. We think docker, with its ecosystem, along with consul provide much of what's needed to manage it without too much pain. Granted, this is all conjecture at this point.

[shmish111]

@bpoweski I have started to look into this again recently, how did it go with twemproxy and consul? I am investigating how we can get some sort of redis cluster going that is fast and has some failover capabilities. Using Redis Cluster is one option and I would be grateful if you could release your proof of concept somewhere? Twemproxy is the other major contender I think, could you tell me why you chose it over redis cluster and if you feel you made the right decision?

[bbsbb]

Is there any plan on shipping this or is it looking for external help at this point?

[ptaoussanis]

Fully-baked PRs welcome, otherwise happy to implement if someone wants to fund my time on this. Otherwise don't need Cluster myself so unfortunately no other plans atm, sorry.

[lemonteaa]

Hey, just to let people know that I'm working on this in my forked repo. Since it's been so long (last updates on the order of year), with some non-trivially intertwined developments, I have first merged from the main branch into the dev branch for cluster, as well as combine codes from two different attempts. (Using rebase/cherrypick) (to take advantage of stuffs in main branch since then)

Things are actually already 70-80% completed before I started, from my first glance of the codes. I will mostly follow the plans included in the source code except for the keyslot cache data structure.

@ptaoussanis I've written a sketch of my planned design of said keyslot cache data structure in cluster.clj, could you take a look and see if it's okay? Thanks!

[ptaoussanis]

Closing to focus on #274