Library for working with ClickHouse's RowBinary format.
Currently, the library only supports encoding to RowBinary format. A decoder might be implemented later. This library is meant to be used for ingestion data into ClickHouse with Elixir efficiently.
The package can be installed by adding rowbinary to your list of dependencies in mix.exs:
def deps do
[
{:rowbinary, "~> 0.1.0"}
]
endRowBinary.encode/2 can be used to encode Elixir data into ClickHouse RowBinary-encoded binaries. Additionally, a protocol spec RowBinary.RowBinaryEncoding is included to help with implementing encoders for structs or maps.
iex> RowBinary.encode(17, [:int8])
<<17>>
iex> RowBinary.encode("hello", [:enum8, %{"hello" => 0, "world" => 1}])
<<0>>
iex> RowBinary.encode(["foo", nil, "barbazbarbaz"], [:array, :nullable, :fixedstring, 8])
<<3, 0, 102, 111, 111, 0, 0, 0, 0, 0, 1, 0, 98, 97, 114, 98, 97, 122, 98, 97>>
iex> RowBinary.encode(1337, [:int8]) # 1137 is out of range for 8 bit integers
** (ArgumentError) value=1337 with wrong types=[:int8]RowBinary.encode/2 takes a value that should be encoded and a type definition. See RowBinary.encode/2 for information about supported types.
- Decimal and AggregateFunction types are not supported
- Type definitions are not checked before encoding. Types like Nullable(Array(Int8)) are not allowed in ClickHouse, but
RowBinary.encode/2will generate a result for now. Keep that in mind.
- Not all unit test are done yet, but will be gradually updated.
- You can run
mix generate_test_casesto generate ExUnit test cases with data directly pulled from a local ClickHouse server.
Name ips average deviation median 99th %
float32 13.44 M 74.38 ns ±31997.44% 46 ns 97 ns
float64 13.42 M 74.54 ns ±31437.63% 39 ns 173 ns
int16 12.83 M 77.92 ns ±34046.24% 42 ns 93 ns
uint16 12.23 M 81.78 ns ±32837.36% 43 ns 102 ns
uint8 11.68 M 85.63 ns ±31902.38% 47 ns 115 ns
int32 11.47 M 87.18 ns ±30597.14% 47 ns 107 ns
int8 11.40 M 87.73 ns ±31150.32% 48 ns 118 ns
uint32 10.93 M 91.48 ns ±30263.29% 52 ns 110 ns
int64 8.97 M 111.50 ns ±21277.66% 82 ns 159 ns
uint64 8.09 M 123.63 ns ±25362.26% 83 ns 264 ns
fixedstring_medium 7.99 M 125.18 ns ±29659.83% 61 ns 103 ns
fixedstring_short 7.49 M 133.54 ns ±29224.88% 61 ns 96 ns
enum16 7.06 M 141.70 ns ±12062.72% 107 ns 185 ns
enum8 6.81 M 146.92 ns ±12132.97% 109 ns 241 ns
int16_nullable 6.26 M 159.70 ns ±19120.97% 101 ns 237 ns
float32_nullable 6.25 M 159.91 ns ±19210.34% 101 ns 232 ns
uint8_nullable 6.14 M 162.81 ns ±19420.34% 100 ns 350 ns
uint16_nullable 6.04 M 165.56 ns ±18865.28% 108 ns 234 ns
int32_nullable 6.01 M 166.30 ns ±18483.14% 106 ns 349 ns
int8_nullable 5.85 M 170.89 ns ±19343.78% 105 ns 413 ns
float64_nullable 5.64 M 177.17 ns ±19409.67% 99 ns 226 ns
fixedstring_nullable_short 5.61 M 178.26 ns ±18491.88% 125 ns 246 ns
ipv4 5.48 M 182.52 ns ±9926.33% 145 ns 211 ns
uint32_nullable 5.46 M 183.14 ns ±17913.89% 109 ns 445 ns
uint64_nullable 5.12 M 195.19 ns ±12095.99% 145 ns 344 ns
enum8_nullable 4.37 M 228.63 ns ±13658.96% 169 ns 314 ns
enum16_nullable 4.31 M 232.22 ns ±13503.33% 173 ns 329 ns
int64_nullable 4.28 M 233.47 ns ±15233.64% 151 ns 323 ns
ipv6 4.11 M 243.57 ns ±5280.01% 216 ns 348 ns
ipv4_nullable 3.67 M 272.79 ns ±11722.87% 207 ns 414 ns
string_medium 3.61 M 276.79 ns ±214.24% 251 ns 542 ns
string_short 3.23 M 309.88 ns ±5660.74% 186 ns 1738 ns
date 2.79 M 358.03 ns ±7836.94% 288 ns 611 ns
ipv6_nullable 2.78 M 360.14 ns ±9366.41% 279 ns 646 ns
string_large 2.64 M 378.87 ns ±53.17% 303 ns 615 ns
fixedstring_nullable_medium 2.59 M 385.93 ns ±11247.24% 199 ns 469 ns
date_nullable 2.21 M 452.19 ns ±8436.48% 350 ns 705 ns
string_nullable_short 2.19 M 456.62 ns ±8237.23% 242 ns 774 ns
fixedstring_nullable_large 2.02 M 494.99 ns ±6464.26% 247 ns 1475 ns
fixedstring_large 1.96 M 509.81 ns ±6193.13% 248 ns 1659 ns
datetime 1.48 M 674.67 ns ±4135.51% 585 ns 1000 ns
datetime_nullable 1.33 M 752.78 ns ±3806.41% 656 ns 1118 ns
string_nullable_medium 1.26 M 795.26 ns ±4828.47% 384 ns 1191 ns
string_nullable_large 0.88 M 1136.22 ns ±2881.13% 644 ns 1392 ns
array_short 0.75 M 1331.87 ns ±2054.50% 1044 ns 2544 ns
uuid_hex 0.26 M 3827.59 ns ±504.17% 3411 ns 6774 ns
uuid_hex_nullable 0.26 M 3904.76 ns ±537.69% 3472 ns 6772 ns
uuid_default 0.25 M 3979.64 ns ±359.59% 3551 ns 9099 ns
uuid_default_nullable 0.24 M 4089.80 ns ±378.71% 3629 ns 9390.95 ns
array_long 0.112 M 8946.26 ns ±158.33% 8180 ns 18199 ns
array_large 0.0121 M 82707.29 ns ±29.05% 83042 ns 97103.83 ns