kterl

A Kyoto Tycoon client for Erlang.

Features:

• 100% feature support of Kyoto Tycoon's HTTP and binary protocols.
• Proven non-blocking protocol handler, derived from eredis.
• Easy to use and fully documented, with plenty of examples.
• Clean, consistent, idiomatic interface.
• OTP compliant.

Examples

Retrieve and build kterl:

    git clone git://github.com/rcb/kterl.git
    cd kterl
    make && make docs
    erl -pa ebin/

Download and install kyoto tycoon onto your machine, and start it with a cache tree (in-memory b+tree) database:

    ktserver %

With ktserver now running on localhost, it's easy to start experimenting:

    Eshell V5.9.1  (abort with ^G)
    1> {ok, Pid} = kterl:start_link().

    =INFO REPORT==== 3-Apr-2012::14:08:25 ===
    Attempting connection to 127.0.0.1:1978

    =INFO REPORT==== 3-Apr-2012::14:08:25 ===
    Connected to Kyoto Tycoon server at 127.0.0.1:1978
    {ok,<0.35.0>}
    2>

[Set | add | replace | append | remove] records from Kyoto Server

Kyoto Tycoon supports the standard set of key value set/add/replace/append/remove operations. The application can pass to kterl lists or binaries as key/values.

    1> ok = kterl:add(Pid, <<"hello">>, <<"world">>).
    ok
    2> {ok, Res} = kterl:get(Pid, "hello").
    {ok, ...}
    3> kterl_result:get_value(Res).
    <<"world">>
    4> ok = kterl:replace(Pid, "hello", <<"github">>).
    ok
    5> kterl_result:get_value(kterl:get(Pid, "hello")).
    <<"github">>
    6> ok = kterl:append(Pid, <<"hello">>, <<"...">>).
    ok
    7> kterl_result:get_value(kterl:get(Pid, "hello")).
    <<"github...">>
    8> ok = kterl:remove(Pid, "hello").
    ok
    9> kterl:get(Pid, "hello").
    {error,no_record}
    10> e(7).
    {error,no_record}

Kyoto Tycoon supports atomic record compare-and-swap and compare-and-remove operations. See the documentation for kterl:cas/3 and kterl:cas/4 for further information.

Automatic value conversions

As a convenience, kterl automatically converts integer, float, or atom values to binary before storing the record.

    kterl:add(Pid, "int_key", 12345).
    kterl_result:get_value(kterl:get(Pid, "int_key")).
    <<"12345">>

    kterl:add(Pid, "float_key", 12345.123).
    kterl_result:get_value(kterl:get(Pid, "float_key")).
    <<"12345.123000">>

    kterl:add(Pid, "atom_key", hello_world).
    kterl_result:get_value(kterl:get(Pid, "atom_key")).
    <<"hello_world">>

    kterl:append(Pid, "atom_key", 123456).
    kterl_result:get_value(kterl:get(Pid, "atom_key")).
    <<"hello_world123456">>

    kterl:add(Pid, "erlang_term", term_to_binary([hello,{1,2,3,4},<<"testing">>])).
    binary_to_term(kterl_result:get_value(kterl:get(Pid, "erlang_term"))).
    [hello,{1,2,3,4},<<"testing">>]

Bulk record operations, using the HTTP or binary protocol

Kyoto Tycoon supports bulk record set/get/remove operations through HTTP or a binary protocol. kterl supports both access methods. In this snippet, 1000 key/value pairs are added, and kterl_result:get_num() is then called on the set_bulk result to extract the number of records added by the server:

    11> Recs = [{"key_" ++ integer_to_list(N), N} || N <- lists:seq(1,1000)].
    ...
    12> {ok, Res1} = kterl:set_bulk(Pid, Recs).
    {ok, ...}
    13> kterl_result:get_num(Res1).
    1000

Here's the binary protocol bulk set call:

    14> kterl:bin_set_bulk(Pid, Recs).
    {ok, 1000}

Bulk retrieval is easy using the HTTP protocol...

    15> Keys = [Key || {Key, _} <- Recs].
    ...
    16> {ok, Res2} = kterl:get_bulk(Pid, Keys).
    {ok,...}
    17> kterl_result:get_num(Res2).
    1000
    18> length(kterl_result:get_records(Res2)).
    1000
    19> hd(kterl_result:get_records(Res2)).
    {<<"key_1">>,<<"1">>}

... or the binary protocol:

    20> {ok, BinRes} = kterl:bin_get_bulk(Pid, Keys).
    {ok, ...}
    21> length(BinRes).
    1000
    22> Fbr = hd(BinRes).
    {kt_bin_rec,0,1099511627775,<<"key_1000">>,<<"1000">>}
    23> kterl_binrec:get_key(Fbr).
    <<"key_1000">>
    24> kterl_binrec:get_value(Fbr).
    <<"1000">>
    25> kterl:bin_remove_bulk(Pid, Keys).
    {ok, 1000}

As a convenience, the application can also pass an erlang dict() to the bulk set calls:

    D = dict:from_list([{"key1","val1"}, {"key2","val2"}]).
    kterl:set_bulk(Pid, D).
    kterl:bin_set_bulk(Pid, D).

Bulk record operations - Binary vs HTTP. Which is better?

The HTTP protocol has a richer feature set than the binary protocol. It allows the application to specify target databases with a filename or a numeric identifier, and can also interact with Kyoto Tycoon's signaling mechanisms. The primary reason to use the binary calls is performance, especially for large bulk get operations:

    D = dict:from_list([{"k_" ++ integer_to_list(N), N} || N <- lists:seq(1,10000)]).
    {ok, 10000} = kterl:bin_set_bulk(Pid, D).
    timer:tc(kterl,get_bulk,[Pid, dict:fetch_keys(D)]).
    {90587, ...}
    timer:tc(kterl,bin_get_bulk,[Pid, dict:fetch_keys(D)]).
    {35183, ...}

Automatic Record expiration:

Kyoto Tycoon supports automatic record expiration:

    kterl:add(Pid, "exprec", "5..4..3..", [{xt, 5}]).
    kterl_result:get_value(kterl:get(Pid, "exprec")).
    <<"5..4..3..">>
    timer:sleep(timer:seconds(5)).
    kterl:get(Pid,"exprec").
    {error,no_record}

As a convenience, kterl converts calendar:datetime() types when setting an expiration time:

    calendar:gregorian_seconds_to_datetime(
           20 + calendar:datetime_to_gregorian_seconds(calendar:local_time())). 
    {{2012,4,3},{22,30,35}}

    XT = calendar:gregorian_seconds_to_datetime(
           20 + calendar:datetime_to_gregorian_seconds(calendar:local_time())).
    kterl:add(Pid, "exprec", "20..19..18..", [{xt, XT}]).
    kterl_result:get_value(kterl:get(Pid,"exprec")).
    <<"20..19..18..">>
    timer:sleep(timer:seconds(20)).
    kterl_result:get_value(kterl:get(Pid,"exprec")).
    {error,no_record}

Many functions in kterl allow the application to set a record's expiration time. Please check the documentation for additional information.

Counters

Kyoto Tycoon records can contain a value representing a counter:

    1> kterl:get(Pid, "inc_test").
    {error,no_record}
    2> {ok,Res} = kterl:increment(Pid, "inc_test", 1).
    ...
    3> kterl_result:get_num(Res).
    1
    4> kterl_result:get_num(kterl:increment(Pid, "inc_test", -2)).
    -1

Regex matching

    1> {ok, Res} = kterl:match_regex(Pid, "(^c..$|^(a|z).$)").
    {ok, ...}
    2> kterl_result:get_keys(Res).
    [<<"ad">>,<<"ah">>,<<"am">>,<<"an">>,<<"as">>,<<"at">>,
     <<"ax">>,<<"ay">>,<<"cab">>,<<"cad">>,<<"cam">>,<<"can">>,
     <<"cap">>,<<"car">>,<<"cat">>,<<"caw">>,<<"chi">>,<<"cob">>,
     <<"cod">>,<<"cog">>,<<"con">>,<<"coo">>,<<"cop">>,<<"cot">>,
     <<"cow">>,<<"cox">>,<<"coy">>,<<"cry">>,<<...>>|...]

Cursors

Kyoto Tycoon supports database cursors. With these, an application can traverse, access, and mutate records.

    1> {ok, Pid} = kterl:start_link().
    ...
    2> Cursor = kterl:cursor(Pid).
    {kterl_cursor,1,<0.35.0>}
    3> kterl:cur_jump(Cursor, [{key, "key_name"}]).
    ok
    4> {ok, Cres} = kterl:cur_get(Cursor).
    ...
    5> kterl_result:get_key(Cres).
    <<"key_name">>
    6> kterl_result:get_value(Cres).
    <<"record_value">>

Iterating over records in a database is simple. In this example, the print_record function instructs the Kyoto Tycoon server to retrieve a record's key and value, and then automatically step the cursor to an adjacent record.

    print_database(Cursor) ->
        case kterl:cur_get(Cursor, true) of
            {ok, Res} ->
                Key = kterl_result:get_key(Res),
                Value = kterl_result:get_value(Res),
                io:format("~p ~p~n",[Key, Value]),
                print_database(Cursor);
            {error, invalid_cursor} ->
                ok
        end.

    print_forward(Cursor) ->
        ok = kterl:cur_jump(Cursor),
        print_database(Cursor).

    print_reverse(Cursor) ->
        ok = kterl:cur_jump_back(Cursor),
        print_database(Cursor).

There are many cursor calls available: seize (get and remove), setting a record's value, removing records, and so on. Please refer to the documentation.

Much more...

Kyoto Tycoon also supports server-side scripting, client-side signaling, and much more. Please review the documentation for additional information.