This github repository describes a simple way to integrate Hive Metastore
with Trino
.
- This integration uses
Google Storage
as the storage backend for Hive Metastore. - Used latest stable version of
Trino
which is433
. - Used hive metastore version
2.3.1
and corresponding supported version of Hadoop2.10.1
- Used latest stable version of Superset which is
2.1.3
. (3.x.y version showing incorrect version number on UI)
- Used
mysql
event listener to capture the query events from Trino. Read Docs - Fault Tolerant execution of queries: using
TASK
retry policy. Read Docs - Exchange Manager configuration for Fault Tolerant execution. Read Docs
- JMX setup for monitoring Trino. Read Docs
Google Cloud Storage
as the storage backend for Hive Metastore. Read Docs
- Used
postgres
as the backend database for Superset. Read Docs - Customize superset image for adding
trino
driver. Read Docs - Adding Trino connection to Superset. Read Docs
- Superset Initialization in Docker Container. Dockerhub Guide
- Used
mysql
as the metadata database for Hive Metastore. Youtube Guide - Used
Google File System
as the Hadoop File System for Hive Metastore. Github Guide - Used custom Hive Image for running
hiveserver2
andhive metastore
together in a single container. Read more abouthiverserver2
andhive thrift server
here
- Make sure you have a service account with access to a GCS bucket.
<!-- hive/core-site.xml -->
<property>
<name>fs.gs.project.id</name>
<value>{{YOUR GCP PROJECT}}</value>
</property>
- Get your HMAC keys from GCP console and add them to
exchange-manager.properties
file.
# trino/common/exchange-manager.properties
exchange.base-directories={{YOUR BUCKET NAME}}
exchange.s3.region={{YOUR BUCKET REGION}}
exchange.s3.aws-access-key={{HMAC ACCESS KEY}}
exchange.s3.aws-secret-key={{HMAC SECRET KEY}}
- Put your service account JSON file as
gcp-sa.json
insecrets
directory. - Working directory should be as follows:
.
├── docker-compose.yml
├── hive
│ ├── Dockerfile
│ ├── core-site.xml
│ ├── docker-entrypoint.sh
│ └── hive-site.xml
├── issues
├── secrets
│ └── gcp-sa.json
├── superset
│ ├── Dockerfile
│ ├── docker-init.sh
│ └── superset_config.py
└── trino
├── Dockerfile
├── catalog
│ ├── hive.properties
│ └── jmx.properties
├── common
│ └── exchange-manager.properties
├── coordinator
│ ├── config.properties
│ ├── jmx-config.yaml
│ ├── jvm.config
│ └── mysql-event-listener.properties
└── worker
└── config.properties
docker compose up -d
trino --server http://localhost:8080 --catalog hive
Login to Superset using admin
as username and admin
as password.
Made with ❤️ by nil1729