Redis Memory Optimization

WORK IN PROGRESS

Redis keeps all its data in memory, and so it is important to optimize memory usage. This document explains several strategies to reduce memory overheads. This document builds up on the official memory optimization notes on redis.io, so you should read that document first.

Special Encoding for Small Aggregate Objects

Redis uses a special, memory-efficient encoding if a list, sorted set or hash contains a) small number of elements, or b) if the size of each element is less than a threshold.

The following directives control when this special encoding, memory efficient encoding is used :

hash-max-zipmap-entries 64 (hash-max-ziplist-entries for Redis >= 2.6)
hash-max-zipmap-value 512  (hash-max-ziplist-value for Redis >= 2.6)
list-max-ziplist-entries 512
list-max-ziplist-value 64
zset-max-ziplist-entries 128
zset-max-ziplist-value 64

The default settings are a good start, but it is possible to achieve greater memory savings by adjusting these settings according to your data. The memory analyzer helps you find the best value for these parameters based on your data. Run the analyzer on your dump.rdb file, and then open the generated csv file.

Start by filtering on type=list and encoding=linkedlist. If there are no matching rows, or the number of filtered rows is very small, you already are using memory efficiently.

Otherwise, look at the columns num_elements and len_largest_element, and compare them to the settings list-max-ziplist-entries and list-max-ziplist-value. Try to find a value for these two settings such that most linkedlists are converted to ziplists.

Similarly, perform the same steps for hashmaps and sorted sets, and find the optimum configuration values.

Use Integer Sets instead of Hashtable based Sets

Avoid Strings smaller than 200 bytes

String data type has an overhead of about about 90 bytes on a 64 bit machine. In other words, calling set foo bar uses about 96 bytes, of which 90 bytes is overhead.

Use Integer IDs

Always use integer ids for objects. See http://stackoverflow.com/questions/10109921/username-or-id-for-keys-in-redis/10110407#10110407

Use lists instead of dictionaries for small, consistent objects

Lets say you want to store user details in Redis. The logical data structure is a Hash. For example, you would do this - hmset user:123 id 123 firstname Sripathi lastname Krishnan location Mumbai twitter srithedabbler.

Now, Redis 2.6 will store this internally as a Zip List; you can confirm by running debug object user:123 and look at the encoding field. In this encoding, key value pairs are stored sequentially, so the user object we created above would roughly look like this ["firstname", "Sripathi", "lastname", "Krishnan", "location", "Mumbai", "twitter", "srithedabbler"]

Now, if you create a second user, the keys will be duplicated. If you have a million users, well, its a big waste repeating the keys again.

To get around this, we can borrow a concept from Python - NamedTuples. A NamedTuple is simply a read-only list, but with some magic to make that list look like a dictionary.

Your application needs to maintain a mapping from field names to indexes. So, "firstname" => 0, "lastname" => 1 and so on. Then, you simply create a list instead of a hash, like this - lpush user:123 Sripathi Krishnan Mumbai srithedabbler. With the right abstractions in your application, you can save significant memory.

Don't use this technique if -

1. You have less than 50,000 objects
2. Your objects are not regular i.e. some users have lots of information, others very little.

The memory analyzer output has a column "bytes_saved_if_converted_to_list" against every hash. Sum up this column and see how much memory you'd save if you make that change. If it is significant and worth the added complexity, go ahead and make that change.

Use Bitmaps to encode data

Create your own data types