Skip to content

Commit

Permalink
Migration to Trino (#70)
Browse files Browse the repository at this point in the history
  • Loading branch information
takezoe authored Jul 9, 2021
1 parent 697d20f commit 933db57
Show file tree
Hide file tree
Showing 34 changed files with 3,055 additions and 272 deletions.
1 change: 1 addition & 0 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,5 @@ gemspec

group :development, :test do
gem 'tiny-presto', '~> 0.0.7'
gem 'presto-client'
end
38 changes: 19 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
# Presto client library for Ruby
# Trino client library for Ruby

[![Build Status](https://travis-ci.org/treasure-data/presto-client-ruby.svg?branch=master)](https://travis-ci.org/treasure-data/presto-client-ruby) [![Gem](https://img.shields.io/gem/v/presto-client)](https://rubygems.org/gems/presto-client) [![Gem](https://img.shields.io/gem/dt/presto-client)](https://rubygems.org/gems/presto-client) [![GitHub](https://img.shields.io/github/license/treasure-data/presto-client-ruby)]()
[![Build Status](https://travis-ci.org/treasure-data/trino-client-ruby.svg?branch=master)](https://travis-ci.org/treasure-data/trino-client-ruby) [![Gem](https://img.shields.io/gem/v/trino-client)](https://rubygems.org/gems/trino-client) [![Gem](https://img.shields.io/gem/dt/trino-client)](https://rubygems.org/gems/trino-client) [![GitHub](https://img.shields.io/github/license/treasure-data/trino-client-ruby)]()

Presto is a distributed SQL query engine for big data:
https://prestosql.io/
Trino is a distributed SQL query engine for big data:
https://trino.io/

This is a client library for Ruby to run queries on Presto.
This is a client library for Ruby to run queries on Trino.

## Example

```ruby
require 'presto-client'
require 'trino-client'

# create a client object:
client = Presto::Client.new(
client = Trino::Client.new(
server: "localhost:8880", # required option
ssl: {verify: false},
catalog: "native",
Expand Down Expand Up @@ -74,7 +74,7 @@ $ bundle exec rake modelgen:latest

## Options

* **server** sets address (and port) of a Presto coordinator server.
* **server** sets address (and port) of a Trino coordinator server.
* **ssl** enables https.
* Setting `true` enables SSL and verifies server certificate using system's built-in certificates.
* Setting `{verify: false}` enables SSL but doesn't verify server certificate.
Expand All @@ -84,34 +84,34 @@ $ bundle exec rake modelgen:latest
* **cert_store**: a `OpenSSL::X509::Store` object used for verification
* **client_cert**: a `OpenSSL::X509::Certificate` object as client certificate
* **client_key**: a `OpenSSL::PKey::RSA` or `OpenSSL::PKey::DSA` object used for client certificate
* **catalog** sets catalog (connector) name of Presto such as `hive-cdh4`, `hive-hadoop1`, etc.
* **schema** sets default schema name of Presto. You need to use qualified name like `FROM myschema.table1` to use non-default schemas.
* **source** sets source name to connect to a Presto. This name is shown on Presto web interface.
* **catalog** sets catalog (connector) name of Trino such as `hive-cdh4`, `hive-hadoop1`, etc.
* **schema** sets default schema name of Trino. You need to use qualified name like `FROM myschema.table1` to use non-default schemas.
* **source** sets source name to connect to a Trino. This name is shown on Trino web interface.
* **client_info** sets client info to queries. It can be a string to pass a raw string, or an object that can be encoded to JSON.
* **client_tags** sets client tags to queries. It needs to be an array of strings. The tags are shown on web interface.
* **user** sets user name to connect to a Presto.
* **password** sets a password to connect to Presto using basic auth.
* **user** sets user name to connect to a Trino.
* **password** sets a password to connect to Trino using basic auth.
* **time_zone** sets time zone of queries. Time zone affects some functions such as `format_datetime`.
* **language** sets language of queries. Language affects some functions such as `format_datetime`.
* **properties** set session properties. Session properties affect internal behavior such as `hive.force_local_scheduling: true`, `raptor.reader_stream_buffer_size: "32MB"`, etc.
* **query_timeout** sets timeout in seconds for the entire query execution (from the first API call until there're no more output data). If timeout happens, client raises PrestoQueryTimeoutError. Default is nil (disabled).
* **plan_timeout** sets timeout in seconds for query planning execution (from the first API call until result columns become available). If timeout happens, client raises PrestoQueryTimeoutError. Default is nil (disabled).
* **query_timeout** sets timeout in seconds for the entire query execution (from the first API call until there're no more output data). If timeout happens, client raises TrinoQueryTimeoutError. Default is nil (disabled).
* **plan_timeout** sets timeout in seconds for query planning execution (from the first API call until result columns become available). If timeout happens, client raises TrinoQueryTimeoutError. Default is nil (disabled).
* **http_headers** sets custom HTTP headers. It must be a Hash of string to string.
* **http_proxy** sets host:port of a HTTP proxy server.
* **http_debug** enables debug message to STDOUT for each HTTP requests.
* **http_open_timeout** sets timeout in seconds to open new HTTP connection.
* **http_timeout** sets timeout in seconds to read data from a server.
* **gzip** enables gzip compression.
* **follow_redirect** enables HTTP redirection support.
* **model_version** set the presto version to which a job is submitted. Supported versions are 316, 303, 0.205, 0.178, 0.173, 0.153 and 0.149. Default is 316.
* **model_version** set the Trino version to which a job is submitted. Supported versions are 351, 316, 303, 0.205, 0.178, 0.173, 0.153 and 0.149. Default is 351.

See [RDoc](http://www.rubydoc.info/gems/presto-client/) for the full documentation.

## Development

### Releasing a new version

1. First update `lib/presto/client/version.rb` to the next version.
1. First update `lib/trino/client/version.rb` to the next version.
2. Run the following command which will update `ChangeLog.md` file automatically.
```
$ ruby release.rb
Expand All @@ -126,6 +126,6 @@ $ git tag "vX.Y.Z"

4. Push package
```
$ gem build presto-client.gemspec
$ gem push presto-client-X.Y.Z.gem
$ gem build trino-client.gemspec
$ gem push trino-client-X.Y.Z.gem
```
26 changes: 10 additions & 16 deletions Rakefile
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,7 @@ RSpec::Core::RakeTask.new(:spec)
task :default => [:spec, :build]

GEN_MODEL_VERSIONS = %w[
0.149
0.153
0.173
0.178
0.205
303
316
351
]

namespace "modelgen" do
Expand All @@ -27,25 +21,25 @@ namespace "modelgen" do
@versions = GEN_MODEL_VERSIONS
@latest_version = GEN_MODEL_VERSIONS.last
data = erb.result
File.write("lib/presto/client/models.rb", data)
File.write("lib/trino/client/models.rb", data)
end

task :all => GEN_MODEL_VERSIONS

GEN_MODEL_VERSIONS.each do |ver|
file "build/presto-#{ver}.tar.gz" do
file "build/trino-#{ver}.tar.gz" do
mkdir_p "build"
sh "curl -L -o build/presto-#{ver}.tar.gz https://github.com/prestosql/presto/archive/#{ver}.tar.gz"
sh "curl -L -o build/trino-#{ver}.tar.gz https://github.com/trinodb/trino/archive/#{ver}.tar.gz"
end

file "lib/presto/client/model_versions/#{ver}.rb" => "build/presto-#{ver}.tar.gz" do
sh "tar zxf build/presto-#{ver}.tar.gz -C build"
mkdir_p "lib/presto/client/model_versions"
sh "#{RbConfig.ruby} modelgen/modelgen.rb #{ver} build/presto-#{ver} modelgen/model_versions.rb lib/presto/client/model_versions/#{ver}.rb"
puts "Generated lib/presto/client/model_versions/#{ver}.rb."
file "lib/trino/client/model_versions/#{ver}.rb" => "build/trino-#{ver}.tar.gz" do
sh "tar zxf build/trino-#{ver}.tar.gz -C build"
mkdir_p "lib/trino/client/model_versions"
sh "#{RbConfig.ruby} modelgen/modelgen.rb #{ver} build/trino-#{ver} modelgen/model_versions.rb lib/trino/client/model_versions/#{ver}.rb"
puts "Generated lib/trino/client/model_versions/#{ver}.rb."
end

task ver => "lib/presto/client/model_versions/#{ver}.rb"
task ver => "lib/trino/client/model_versions/#{ver}.rb"
end
end

1 change: 0 additions & 1 deletion lib/presto-client.rb

This file was deleted.

1 change: 1 addition & 0 deletions lib/trino-client.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
require 'trino/client'
8 changes: 4 additions & 4 deletions lib/presto/client.rb → lib/trino/client.rb
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# Presto client for Ruby
# Trino client for Ruby
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -13,11 +13,11 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
module Presto
module Trino
module Client

require 'presto/client/version'
require 'presto/client/client'
require 'trino/client/version'
require 'trino/client/client'

end
end
10 changes: 5 additions & 5 deletions lib/presto/client/client.rb → lib/trino/client/client.rb
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# Presto client for Ruby
# Trino client for Ruby
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -13,10 +13,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
module Presto::Client
module Trino::Client

require 'presto/client/models'
require 'presto/client/query'
require 'trino/client/models'
require 'trino/client/query'

class Client
def initialize(options)
Expand Down Expand Up @@ -57,7 +57,7 @@ def run(query)
end
end

# Accepts the raw response from the Presto Client and returns an
# Accepts the raw response from the Trino Client and returns an
# array of hashes where you can access the data in each row using the
# output name specified in the query with AS:
# SELECT expression AS output_name
Expand Down
14 changes: 7 additions & 7 deletions lib/presto/client/errors.rb → lib/trino/client/errors.rb
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# Presto client for Ruby
# Trino client for Ruby
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -13,11 +13,11 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
module Presto::Client
class PrestoError < StandardError
module Trino::Client
class TrinoError < StandardError
end

class PrestoHttpError < PrestoError
class TrinoHttpError < TrinoError
def initialize(status, message)
super(message)
@status = status
Expand All @@ -26,10 +26,10 @@ def initialize(status, message)
attr_reader :status
end

class PrestoClientError < PrestoError
class TrinoClientError < TrinoError
end

class PrestoQueryError < PrestoError
class TrinoQueryError < TrinoError
def initialize(message, query_id, error_code, error_name, failure_info)
super(message)
@query_id = query_id
Expand All @@ -41,6 +41,6 @@ def initialize(message, query_id, error_code, error_name, failure_info)
attr_reader :error_code, :error_name, :failure_info
end

class PrestoQueryTimeoutError < PrestoError
class TrinoQueryTimeoutError < TrinoError
end
end
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# Presto client for Ruby
# Trino client for Ruby
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -13,10 +13,27 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
module Presto::Client
module Trino::Client

require 'cgi'

module TrinoHeaders
TRINO_USER = "X-Trino-User"
TRINO_SOURCE = "X-Trino-Source"
TRINO_CATALOG = "X-Trino-Catalog"
TRINO_SCHEMA = "X-Trino-Schema"
TRINO_TIME_ZONE = "X-Trino-Time-Zone"
TRINO_LANGUAGE = "X-Trino-Language"
TRINO_SESSION = "X-Trino-Session"
TRINO_CLIENT_INFO = "X-Trino-Client-Info";
TRINO_CLIENT_TAGS = "X-Trino-Client-Tags";

TRINO_CURRENT_STATE = "X-Trino-Current-State"
TRINO_MAX_WAIT = "X-Trino-Max-Wait"
TRINO_MAX_SIZE = "X-Trino-Max-Size"
TRINO_PAGE_SEQUENCE_ID = "X-Trino-Page-Sequence-Id"
end

module PrestoHeaders
PRESTO_USER = "X-Presto-User"
PRESTO_SOURCE = "X-Presto-Source"
Expand All @@ -35,7 +52,7 @@ module PrestoHeaders
end

HEADERS = {
"User-Agent" => "presto-ruby/#{VERSION}",
"User-Agent" => "trino-ruby/#{VERSION}",
}

def self.faraday_client(options)
Expand Down Expand Up @@ -105,38 +122,79 @@ def self.faraday_ssl_options(options)
end

def self.optional_headers(options)
usePrestoHeader = false
if v = options[:model_version] && v < 351
usePrestoHeader = true
end

headers = {}
if v = options[:user]
headers[PrestoHeaders::PRESTO_USER] = v
if usePrestoHeader
headers[PrestoHeaders::PRESTO_USER] = v
else
headers[TrinoHeaders::TRINO_USER] = v
end
end
if v = options[:source]
headers[PrestoHeaders::PRESTO_SOURCE] = v
if usePrestoHeader
headers[PrestoHeaders::PRESTO_SOURCE] = v
else
headers[TrinoHeaders::TRINO_SOURCE] = v
end
end
if v = options[:catalog]
headers[PrestoHeaders::PRESTO_CATALOG] = v
if usePrestoHeader
headers[PrestoHeaders::PRESTO_CATALOG] = v
else
headers[TrinoHeaders::TRINO_CATALOG] = v
end
end
if v = options[:schema]
headers[PrestoHeaders::PRESTO_SCHEMA] = v
if usePrestoHeader
headers[PrestoHeaders::PRESTO_SCHEMA] = v
else
headers[TrinoHeaders::TRINO_SCHEMA] = v
end
end
if v = options[:time_zone]
headers[PrestoHeaders::PRESTO_TIME_ZONE] = v
if usePrestoHeader
headers[PrestoHeaders::PRESTO_TIME_ZONE] = v
else
headers[TrinoHeaders::TRINO_TIME_ZONE] = v
end
end
if v = options[:language]
headers[PrestoHeaders::PRESTO_LANGUAGE] = v
if usePrestoHeader
headers[PrestoHeaders::PRESTO_LANGUAGE] = v
else
headers[TrinoHeaders::TRINO_LANGUAGE] = v
end
end
if v = options[:properties]
headers[PrestoHeaders::PRESTO_SESSION] = encode_properties(v)
if usePrestoHeader
headers[PrestoHeaders::PRESTO_SESSION] = encode_properties(v)
else
headers[TrinoHeaders::TRINO_SESSION] = encode_properties(v)
end
end
if v = options[:client_info]
headers[PrestoHeaders::PRESTO_CLIENT_INFO] = encode_client_info(v)
if usePrestoHeader
headers[PrestoHeaders::PRESTO_CLIENT_INFO] = encode_client_info(v)
else
headers[TrinoHeaders::TRINO_CLIENT_INFO] = encode_client_info(v)
end
end
if v = options[:client_tags]
headers[PrestoHeaders::PRESTO_CLIENT_TAGS] = encode_client_tags(v)
if usePrestoHeader
headers[PrestoHeaders::PRESTO_CLIENT_TAGS] = encode_client_tags(v)
else
headers[TrinoHeaders::TRINO_CLIENT_TAGS] = encode_client_tags(v)
end
end
if options[:enable_x_msgpack]
# option name is enable_"x"_msgpack because "Accept: application/x-msgpack" header is
# not officially supported by Presto. We can use this option only if a proxy server
# decodes & encodes response body. Once this option is supported by Presto, option
# not officially supported by Trino. We can use this option only if a proxy server
# decodes & encodes response body. Once this option is supported by Trino, option
# name should be enable_msgpack, which might be slightly different behavior.
headers['Accept'] = 'application/x-msgpack,application/json'
end
Expand Down
Loading

0 comments on commit 933db57

Please sign in to comment.