Golang Apache Impala Driver

The actively supported Apache Impala driver for Go's database/sql package

This driver started as a fork of github.com/bippio/go-impala, which hasn't been updated in over four years and appears to be abandoned. Several issues have been fixed since - some quite severe. The original codebase also didn't support Go modules.

Install

Add impala-go to your Go module:

go get github.com/sclgo/impala-go

Alternatively, see below how to use as a CLI.

Connection Parameters and DSN

The connection string uses a URL format: impala://username:password@host:port?param1=value&param2=value

Parameters:

• auth - string. Authentication mode. Supported values: "noauth", "ldap"
• tls - boolean. Enable TLS
• ca-cert - The file that contains the public key certificate of the CA that signed the Impala certificate
• batch-size - integer value (default: 1024). Maximum number of rows fetched per request
• buffer-size- in bytes (default: 4096); Buffer size for the Thrift transport
• mem-limit - string value (example: 3m); Memory limit for query

A string of this format can be constructed using the URL type in the net/url package.

  query := url.Values{}
  query.Add("auth", "ldap")

  u := &url.URL{
      Scheme:   "impala",
      User:     url.UserPassword(username, password),
      Host:     net.JoinHostPort(hostname, port),
      RawQuery: query.Encode(),
  }
  db, err := sql.Open("impala", u.String())

Also, you can bypass string-base data source name by using sql.OpenDB:

  opts := impala.DefaultOptions
  opts.Host = hostname
  opts.UseLDAP = true
  opts.Username = username
  opts.Password = password

  connector := impala.NewConnector(&opts)
  db, err := sql.OpenDB(connector)

CLI

impala-go is included in xo/usql - the universal SQL CLI, inspired by psql.

Install usql, start it, then on its prompt, run:

\connect impala DSN

where DSN is a data source name in the format above. Review the usql documentation for other options.

The latest version of usql typically comes with the latest version of impala-go but if you need to use a different one, you can prepare a custom build using usqlgen. For example, the following command builds a usql binary in the working directory using impala-go from master branch:

go run github.com/sclgo/usqlgen@latest build --get github.com/sclgo/impala-go@master -- -tags impala

Example Go code

package main

// Simple program to list databases and the tables

import (
	"context"
	"database/sql"
	"log"

	"github.com/sclgo/impala-go"
)

func main() {
	opts := impala.DefaultOptions

	opts.Host = "localhost" // impala host
	opts.Port = "21050"

	// enable LDAP authentication:
	//opts.UseLDAP = true
	//opts.Username = "<ldap username>"
	//opts.Password = "<ldap password>"
	//
	// enable TLS
	//opts.UseTLS = true
	//opts.CACertPath = "/path/to/cacert"

	connector := impala.NewConnector(&opts)
	db := sql.OpenDB(connector)
	defer func() {
		_ = db.Close()
	}()

	ctx := context.Background()

	rows, err := db.QueryContext(ctx, "SHOW DATABASES")
	if err != nil {
		log.Fatal(err)
	}

	var name, comment string
	databases := make([]string, 0) // databases will contain all the DBs to enumerate later
	for rows.Next() {
		if err := rows.Scan(&name, &comment); err != nil {
			log.Fatal(err)
		}
		databases = append(databases, name)
	}
	if err := rows.Err(); err != nil {
		log.Fatal(err)
	}
	log.Println("List of Databases", databases)

	tables, err := impala.NewMetadata(db).GetTables(ctx, "%", "%")
	if err != nil {
		log.Fatal(err)
	}
	log.Println("List of Tables", tables)
}

Check out also an open data end-to-end demo.

Support

The library is actively tested with Impala 4.4 and 3.4. All 3.x and 4.x minor versions should work well. 2.x is also supported on a best-effort basis.

File any issues that you encounter as Github issues.

Copyright and acknowledgements

This library started as a fork of github.com/bippio/go-impala, under the MIT license. This library retains the same license.

The project logo combines the Golang Gopher from github.com/golang-samples/gopher-vector with the Apache Impala logo, licensed under the Apache 2 license.