Skip to content

Commit

Permalink
feat(tags): replace tags identifier from record urn and type to asset…
Browse files Browse the repository at this point in the history 
… id (#122)

* feat(tags): replace record urn and type in tags to asset id

* chore: update readme

* chore: update docs

* chore: update docs

* chore: update docs and makefile

* fix(tags): fix inconsistent behaviour

* fix(tags): add template not found error check in update tag handler

* chore: fix broken tests in tag
  • Loading branch information
@mabdh
mabdh authored May 12, 2022
1 parent 004001c commit b761ace
Show file tree
Hide file tree
Showing 38 changed files with 6,079 additions and 6,319 deletions.
20 changes: 10 additions & 10 deletions Makefile
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
NAME="github.com/odpf/compass"
VERSION=$(shell git describe --always --tags 2>/dev/null)
COVERFILE="/tmp/compass.coverprofile"
PROTON_COMMIT := "efc71a54e643624f2f809bac8c095c069576c4dd"
PROTON_COMMIT := "4d2fb0f0b145c31c02ccd65fb4a83510d58712e2"

.PHONY: all build test clean install proto

Expand Down Expand Up @@ -40,12 +40,12 @@ proto: ## Generate the protobuf files
install: ## install required dependencies
@echo "> installing dependencies"
go mod tidy
go get github.com/vektra/mockery/v2@v2.10.4
go get google.golang.org/protobuf/cmd/protoc-gen-go@v1.27.1
go get google.golang.org/protobuf/proto@v1.27.1
go get google.golang.org/grpc@v1.45.0
go get google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2.0
go get github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway@v2.8.0
go get github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2@v2.8.0
go get github.com/bufbuild/buf/cmd/buf@v1.3.1
go get github.com/envoyproxy/protoc-gen-validate@v0.6.7
go install github.com/vektra/mockery/v2@v2.12.2
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28.0
go get google.golang.org/protobuf/proto@v1.28.0
go get google.golang.org/grpc@v1.46.0
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2.0
go install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway@v2.9.0
go install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2@v2.9.0
go install github.com/bufbuild/buf/cmd/buf@v1.4.0
go install github.com/envoyproxy/protoc-gen-validate@v0.6.7
76 changes: 47 additions & 29 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,68 +37,86 @@ Explore the following resources to get started with Compass:

## Requirements

Compass is written in golang, and requires go version >= 1.16. Please make sure that the go tool chain is available on your machine. See golang’s [documentation](https://golang.org/) for installation instructions. Compass is also using [mockery](https://github.com/vektra/mockery) v2.10.0 to generate mocks.
Compass is written in Golang, and requires go version >= 1.16. Please make sure that the go toolchain is available on your machine. See Golang’s [documentation](https://golang.org/) for installation instructions.

Alternatively, you can use docker to build compass as a docker image. More on this in the next section.
Alternatively, you can use docker to build Compass as a docker image. More on this in the next section.

Compass uses elasticsearch v7 as the query and storage backend. In order to run compass locally, you’ll need to have an instance of elasticsearch running. You can either download elasticsearch and run it manually, or you can run elasticsearch inside docker by running the following command in a terminal
```
$ docker run -d -p 9200:9200 -e "discovery.type=single-node" elasticsearch:7.6.1
Compass uses PostgreSQL 13 as its main storage and Elasticsearch v7 as the secondary storage to power the search. In order to run compass locally, you’ll need to have an instance of postgres and elasticsearch running. You can either download them and run it manually, or you can run them inside docker by using `docker-compose` with `docker-compose.yaml` provided in the root of this project.

PostgreSQL details and Elasticsearch brokers can alternatively be specified via the environment variable, `ELASTICSEARCH_BROKERS` for elasticsearch and `DB_HOST`, `DB_NAME`, `DB_USER`, `DB_PASSWORD` for postgres.

If you use Docker to build compass, then configuring networking requires extra steps. Following is one of doing it by running postgres and elasticsearch inside with `docker-compose` first.

Go to the root of this project and run `docker-compose`.

```text
$ docker-compose up
```
Once postgres and elasticsearch has been ready, we can run Compass by passing in the config of postgres and elasticsearch defined in `docker-compose.yaml` file.

## Building Compass

Begin by cloning this repository then you have two ways in which you can build compass

## Running locally
Begin by cloning this repository, then you have two ways in which you can build compass
* As a native executable
* As a docker image

To build compass as a native executable, run `make` inside the cloned repository.
```

```text
$ make
```

This will create the `compass` binary in the root directory

Building compass' Docker image is just a simple, just run docker build command and optionally name the image
```

```text
$ docker build . -t compass
```

Compass interfaces with an elasticsearch cluster. Run compass using:
## Migration
Before serving Compass app, we need to run the migration first. Run this docker command to migrate Compass.

```text
$ docker run --rm --net compass_storage -p 8080:8080 -e ELASTICSEARCH_BROKERS=http://es:9200 -e DB_HOST=postgres -e DB_PORT=5432 -e DB_NAME=compass -e DB_USER=compass -e DB_PASSWORD=compass_password odpf/compass compass migrate
```
./compass -elasticsearch-brokers "http://<broker-host-name>"

If you are using Compass binary, you can run this command.
```text
./compass -elasticsearch-brokers "http://<broker-host-name>" -db-host "<postgres-host-name>" -db-port 5432 -db-name "<postgres-db-name>" -db-user "<postgres-db-user>" -db-password "<postgres-db-password> migrate"
```

Elasticsearch brokers can alternatively be specified via the `ELASTICSEARCH_BROKERS` environment variable.
## Serving Locally

If you used Docker to build compass, then configuring networking requires extra steps. Following is one of doing it, running elasticsearch inside docker
Once the migration has been done, Compass server can be started with this command.

```text
$ docker run --net compass_storage -p 8080:8080 -e ELASTICSEARCH_BROKERS=http://es:9200 -e DB_HOST=postgres -e DB_PORT=5432 -e DB_NAME=compass -e DB_USER=compass -e DB_PASSWORD=compass_password odpf/compass compass serve
```
# create a docker network where compass and elasticsearch will reside
$ docker network create compass-net

# run elasticsearch, bound to the network we created. Since we are using the -d flag to docker run, the command inside the subshell returns the container id
$ ES_CONTAINER_ID=$(docker run -d -e "discovery.type=single-node" --net compass-net elasticsearch:7.5.2)
# run compass, passing in the hostname (container id) of the elasticsearch server
# if everything goes ok, you should say something like this:
# time="2020-04-01T18:41:00Z" level=info msg="compass v0.1.0-103-g83b909b starting on 0.0.0.0:8080" reporter=main
# time="2020-04-01T18:41:00Z" level=info msg="connected to elasticsearch cluster \"docker-cluster\" (server version 7.5.2)" reporter=main
$ docker run --net compass-net compass -p 8080:8080 -elasticsearch-brokers http://${ES_CONTAINER_ID}:9200
If you are using Compass binary, you can run this command.
```text
./compass -elasticsearch-brokers "http://<broker-host-name>" -db-host "<postgres-host-name>" -db-port 5432 -db-name "<postgres-db-name>" -db-user "<postgres-db-user>" -db-password "<postgres-db-password> serve"
```

If everything goes ok, you should see something like this:
```text
time="2022-04-27T09:18:08Z" level=info msg="compass starting" version=v0.2.0
time="2022-04-27T09:18:08Z" level=info msg="connected to elasticsearch cluster" config="\"docker-cluster\" (server version 7.6.1)"
time="2022-04-27T09:18:08Z" level=info msg="New Relic monitoring is disabled."
time="2022-04-27T09:18:08Z" level=info msg="statsd metrics monitoring is disabled."
time="2022-04-27T09:18:08Z" level=info msg="connected to postgres server" host=postgres port=5432
time="2022-04-27T09:18:08Z" level=info msg="server started"
```
## Running tests

```
# Run unit tests
$ make unit-test
# Run integration tests
# Run tests
$ make test
```

The integration test suite requires docker to run elasticsearch. In case you wish to test against an existing
The tests combine both unit and integration tests, the test suite requires docker to run elasticsearch. In case you wish to test against an existing
elasticsearch cluster, set the value of `ES_TEST_SERVER_URL` to the URL of the elasticsearch server.


Expand Down
Loading

0 comments on commit b761ace

Please sign in to comment.