Skip to content

hibari-labs/ubf-redis

BranchesTags
 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
doc
doc
 
 
ebin
ebin
 
 
include
include
 
 
src
src
 
 
test
test
 
 
.gitignore
.gitignore
 
 
.travis.yml
.travis.yml
 
 
LICENSE
LICENSE
 
 
Makefile
Makefile
 
 
README.md
README.md
 
 
rebar
rebar
 
 
rebar.config
rebar.config
 
 
rebar.config.doc
rebar.config.doc
 
 

Repository files navigation

#Universal Binary Format and Redis#

Copyright (c) 2012 by Joseph Wayne Norton

Authors: Joseph Wayne Norton (norton@alum.mit.edu).

This is UBF-REDIS, a framework for integrating UBF and the Redis Unified Format (RUF) protocol. This repository depends on the ubf open source repository.

This repository is experimental in nature - use at your own risk and please contribute if you find UBF-REDIS useful.

Quick Start Recipe

To download, build, and test the ubf_redis application in one shot, please follow this recipe:

$ mkdir working-directory-name
$ cd working-directory-name
$ git clone https://github.com/ubf/ubf-redis.git ubf_redis
$ cd ubf_redis
$ make deps clean compile test

For an alternative recipe with other "features" albeit more complex, please read further.

Documentation

Where should I start?

This README is a good first step.

The UBF User's Guide is the best next step. Check out http://ubf.github.com/ubf/ubf-user-guide.en.html for further detailed information.

This repository has building blocks for constructing your own Redis server clone in Erlang.

  • src/ruf.erl Redis network protocol (a.k.a. RUF) encoder/decoder

  • src/ruf_term.erl Redis raw Erlang encoder/decoder

  • src/ubf_redis_types_plugin.erl Erlang types for Redis RPC

  • src/ubf_redis_plugin.erl Erlang Redis RPC request/response pairs

The QC (a.k.a. QuickCheck, PropEr, etc.) tests underneath the "tests/qc" directory should be helpful for understanding the specification and behavior of these building blocks.

  • test/qc/ruf_tests.erl prop_ruf_requests() tests encoding and decoding of RUF requests. Test inputs are automagically generated using the above Redis RPC types.This QC test has been completed.

  • test/qc/ruf_tests.erl prop_ruf_responses() tests encoding and decoding of RUF responses. Test inputs are automagically generated using the above Redis RPC types.This QC test has been completed.

  • test/qc/redis_proxy_tests.erl tests an Erlang Redis client talking with an Erlang Redis "Proxy" server. The Redis "Proxy" server forwards all requests transparently to a "real" Redis server (using another internal Erlang Redis client). All requests and responses sent to the Redis "Proxy" server are automagically generated and checked using UBF's contract manager with the above Redis RPC types and request/response pairs.This QC test is stil in-progress.

A few "hello world" Eunit tests can also be found in the test/eunit directory.

See below for further details on how to run these tests.

What is UBF?

UBF is the "Universal Binary Format", designed and implemented by Joe Armstrong. UBF is a language for transporting and describing complex data structures across a network. It has three components:

  • UBF(a) is a "language neutral" data transport format, roughly equivalent to well-formed XML.

  • UBF(b) is a programming language for describing types in UBF(a) and protocols between clients and servers. This layer is typically called the "protocol contract". UBF(b) is roughly equivalent to Verified XML, XML-schemas, SOAP and WDSL.

  • UBF(c) is a meta-level protocol used between a UBF client and a UBF server.

See http://ubf.github.com/ubf for further details.

What is Redis?

Redis is an open source, advanced key-value store. See http://redis.io for full details. In particular, see http://redis.io/topics/protocol for a description of the Redis protocol.

To download

  1. Configure your e-mail and name for Git 

    $ git config \--global user.email "you@example.com"
$ git config \--global user.name "Your Name"

  2. Install Repo 

    $ mkdir -p ~/bin
$ wget -O - https://dl-ssl.google.com/dl/googlesource/git-repo/repo > ~/bin/repo
$ chmod a+x ~/bin/repo

  3. Create working directory 

    $ mkdir working-directory-name
$ cd working-directory-name
$ repo init -u https://github.com/ubf/manifests.git -m ubf-redis-default.xml
    Note Your "Git" identity is needed during the init step. Please enter the name and email of your GitHub account if you have one. Team members having read-write access are recommended to use "repo init -u git@github.com:ubf/manifests.git -m ubf-redis-default-rw.xml".
    Tip If you want to checkout the latest development version, please append " -b dev" to the repo init command.

  4. Download Git repositories 

    $ cd working-directory-name
$ repo sync

For further information and help for related tools, please refer to the following links:

To build - basic recipe

  1. Get and install an erlang system http://www.erlang.org

  2. Build 

    $ cd working-directory-name
$ make compile

  3. Run the unit tests 

    $ cd working-directory-name
$ make eunit

To build - optional features

  1. Dialyzer Testing basic recipe

    1. Build Dialyzer's PLT (required once) 

      $ cd working-directory-name
$ make build-plt
      Tip Check Makefile and dialyzer's documentation for further information.

    2. Dialyze with specs 

      $ cd working-directory-name
$ make dialyze
      Caution If you manually run dialyzer with the "-r" option, execute "make clean compile" first to avoid finding duplicate beam files underneath rebar's .eunit directory. Check Makefile for further information.

    3. Dialyze without specs 

      $ cd working-directory-name
$ make dialyze-nospec

To test - QuickCheck

  1. Make sure QuickCheck is in your Erlang code path. One simple way to accomplish this is by adding the code path to your ~/.erlang resource file. 

    true = code:add_pathz(os:getenv("HOME")++"/.erlang.d/deps/quviq/eqc-X.Y.Z/ebin").

  2. Compile for QuickCheck 

    $ cd working-directory-name
$ make clean eqc

  3. Run 5,000 Redis Encoder/Decoder QuickCheck tests 

    $ cd working-directory-name/deps/ubf_redis/.eqc
$ erl -smp +A 5 -pz ../../ubf/ebin -pz ../../qc/ebin

1> ruf_tests:qc_run(5000).
prop_ruf_requests: ...
...
Ok, passed 5000 tests
prop_ruf_responses: ...
...
Ok, passed 5000 tests
[]
.......

  4. Run 500 Redis Proxy QuickCheck tests 

    $ cd working-directory-name/deps/ubf_redis/.eqc
$ erl -smp +A 5 -pz ../../ubf/ebin -pz ../../qc/ebin

1> redis_proxy_tests:qc_run(500).
#Port<0.646>: {data,{eol,<<"[9204] 28 Jul 23:29:49 # Opening port 6379: bind: Address already in use">>}}
#Port<0.663>: {data,{eol,<<"[9209] 28 Jul 23:29:49 * Server started, Redis version 2.4.15">>}}
#Port<0.663>: {data,{eol,<<"[9209] 28 Jul 23:29:49 * The server is now ready to accept connections on port 6379">>}}
....
OK, passed 500 tests

100.0% {1,attempts}

1.01% {decr_req,'->','integer()'}
1.01% {bitop_and_req,'->',{error,'ERR unknown command \'BITOP\''}}
0.98% {bitop_xor_req,'->',{error,'ERR unknown command \'BITOP\''}}
0.96% {srandmember_req,'->',undefined}
0.95% {hmset_req,'->',ok}
0.95% {bitop_not_req,'->',{error,'ERR unknown command \'BITOP\''}}
0.93% {zcount_req,'->','integer()'}
0.93% {zadd_req,'->','integer()'}
0.93% {msetnx_req,'->','integer()'}
0.92% {zscore_req,'->',undefined}
0.92% {zremrangebyrank_req,'->','integer()'}
0.92% {pexpire_req,'->',{error,'ERR unknown command \'PEXPIRE\''}}
0.92% {incr_req,'->','integer()'}
0.91% {scard_req,'->','integer()'}
0.91% {object_encoding_req,'->',undefined}
0.90% {bitop_or_req,'->',{error,'ERR unknown command \'BITOP\''}}
0.90% {append_req,'->','integer()'}
0.89% {time_req,'->',{error,'ERR unknown command \'TIME\''}}
0.89% {sinterstore_req,'->','integer()'}
0.89% {pttl_req,'->',{error,'ERR unknown command \'PTTL\''}}
0.88% {zrem_req,'->','integer()'}
0.88% {smembers_req,'->','list()'}
0.88% {move_req,'->',{error,'ERR index out of range'}}
0.88% {monitor_req,'->',ok}
0.88% {mget_req,'->','list()'}
0.88% {lindex_req,'->',undefined}
0.88% {info_req,'->','list()'}
0.88% {flushall_req,'->',ok}
0.88% {config_resetstat_req,'->',ok}
0.87% {zrevrank_req,'->',undefined}
0.87% {setex_req,'->',ok}
0.87% {rpushx_req,'->','integer()'}
0.87% {hincrbyfloat_req,'->',{error,'ERR unknown command \'HINCRBYFLOAT\''}}
0.87% {expireat_req,'->','integer()'}
0.87% {exists_req,'->','integer()'}
0.86% {sunionstore_req,'->','integer()'}
0.86% {strlen_req,'->','integer()'}
0.86% {renamenx_req,'->',{error,'ERR no such key'}}
0.86% {rename_req,'->',{error,'ERR no such key'}}
0.86% {keys_req,'->','list()'}
0.86% {bitcount_req,'->',{error,'ERR unknown command \'BITCOUNT\''}}
0.86% {sadd_req,'->','integer()'}
0.86% {mset_req,'->',ok}
0.86% {hlen_req,'->','integer()'}
0.86% {bgrewriteaof_req,'->',ok}
0.85% {sdiff_req,'->','list()'}
0.85% {getrange_req,'->','binary()'}
0.84% {slaveof_req,'->',ok}
0.84% {incrbyfloat_req,'->',{error,'ERR unknown command \'INCRBYFLOAT\''}}
0.84% {hsetnx_req,'->','integer()'}
0.84% {decrby_req,'->','integer()'}
0.83% {zcard_req,'->','integer()'}
0.83% {select_req,'->',{error,'ERR invalid DB index'}}
0.83% {getset_req,'->',undefined}
0.82% {zremrangebyscore_req,'->','integer()'}
0.82% {srem_req,'->','integer()'}
0.82% {sismember_req,'->','integer()'}
0.82% {restore_req,'->',{error,'ERR unknown command \'RESTORE\''}}
0.82% {pexpireat_req,'->',{error,'ERR unknown command \'PEXPIREAT\''}}
0.82% {incrby_req,'->','integer()'}
0.82% {sdiffstore_req,'->','integer()'}
0.82% {ltrim_req,'->',ok}
0.82% {lset_req,'->',{error,'ERR no such key'}}
0.82% {lpush_req,'->','integer()'}
0.82% {del_req,'->','integer()'}
0.81% {type_req,'->',none}
0.81% {multi_req,'->',ok}
0.81% {lrem_req,'->','integer()'}
0.81% {hexists_req,'->','integer()'}
0.81% {debug_object_req,'->',{error,'ERR no such key'}}
0.80% {smove_req,'->','integer()'}
0.80% {object_idletime_req,'->',undefined}
0.80% {linsert_req,'->','integer()'}
0.80% {get_req,'->',undefined}
0.80% {flushdb_req,'->',ok}
0.79% {zrangebyscore_req,'->','list()'}
0.79% {watch_req,'->',ok}
0.79% {migrate_req,'->',{error,'ERR unknown command \'MIGRATE\''}}
0.78% {zrevrangebyscore_req,'->','list()'}
0.78% {zrank_req,'->',undefined}
0.78% {set_req,'->',ok}
0.78% {lpushx_req,'->','integer()'}
0.78% {config_get_req,'->','list()'}
0.78% {lpop_req,'->',undefined}
0.78% {hvals_req,'->','list()'}
0.78% {hgetall_req,'->','list()'}
0.77% {rpush_req,'->','integer()'}
0.77% {psetex_req,'->',{error,'ERR unknown command \'PSETEX\''}}
0.77% {dump_req,'->',{error,'ERR unknown command \'DUMP\''}}
0.76% {spop_req,'->',undefined}
0.76% {hincrby_req,'->','integer()'}
0.76% {hget_req,'->',undefined}
0.76% {discard_req,'->',{error,'ERR DISCARD without MULTI'}}
0.75% {llen_req,'->','integer()'}
0.75% {expire_req,'->','integer()'}
0.74% {zrevrange_req,'->',{error,'ERR syntax error'}}
0.74% {lrange_req,'->','list()'}
0.74% {hset_req,'->','integer()'}
0.74% {getbit_req,'->','integer()'}
0.74% {hdel_req,'->','integer()'}
0.73% {sunion_req,'->','list()'}
0.73% {hmget_req,'->','list()'}
0.73% {hkeys_req,'->','list()'}
0.73% {config_set_req,'->','error()'}
0.72% {setnx_req,'->','integer()'}
0.72% {rpop_req,'->',undefined}
0.72% {persist_req,'->','integer()'}
0.72% {lastsave_req,'->','integer()'}
0.72% {dbsize_req,'->','integer()'}
0.71% {setbit_req,'->',{error,'ERR bit is not an integer or out of range'}}
0.71% {auth_req,'->',{error,'ERR Client sent AUTH, but no password is set'}}
0.70% {sinter_req,'->','list()'}
0.70% {ping_req,'->',ok}
0.70% {object_refcount_req,'->',undefined}
0.69% {zrange_req,'->',{error,'ERR syntax error'}}
0.68% {sort_req,'->','list()'}
0.65% {ttl_req,'->','integer()'}
0.63% {rpoplpush_req,'->',undefined}
0.63% {echo_req,'->','binary()'}
0.61% {setrange_req,'->',{error,'ERR string exceeds maximum allowed size (512MB)'}}
0.55% {randomkey_req,'->','binary()'}
0.44% {zincrby_req,'->','integer()'}
0.23% {zincrby_req,'->','float()'}
0.15% {randomkey_req,'->',undefined}
0.14% {setrange_req,'->','integer()'}
0.10% {zrevrange_req,'->','list()'}
0.10% {sort_req,'->','integer()'}
0.09% {zrange_req,'->','list()'}
0.06% {setbit_req,'->','integer()'}
0.02% {sdiff_req,'->',{error,'ERR Operation against a key holding the wrong kind of value'}}
0.02% {bgrewriteaof_req,'->',{error,'ERR Background append only file rewriting already in progress'}}
0.01% {sunion_req,'->',{error,'ERR Operation against a key holding the wrong kind of value'}}
0.01% {sdiffstore_req,'->',{error,'ERR Operation against a key holding the wrong kind of value'}}
.......
    Tip This test requires redis-server to be installed under the /usr/local/ directory.
    Caution This test removes all data under the /usr/local/var/db/redis/ directory.

To test - PropEr

Under Construction - To Be Added

Roadmap

  • Documentation

    • Explain overall implementation and test strategy.

    • Explain relevant UBF features.

    • Explain how one can start to build their own Redis server in Erlang.

  • Testing - Black Box

    • Implement real "statem" test model for the Redis Proxy.

##Modules##

ruf
ruf_driver
ruf_term
ubf_redis_plugin
ubf_redis_types_plugin

About

Universal Binary Format 2.0 and Redis

Resources

Readme

License

MIT license
Activity
Custom properties

Stars

0 stars

Watchers

2 watching

Forks

0 forks
Report repository

Releases

6 tags

Packages

No packages published