Skip to content
This repository has been archived by the owner on Apr 12, 2022. It is now read-only.
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: lifechurch/dnaquery
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 9bece45e6dfdb371b54a765a672429f9958bc2ca
Choose a base ref
...
head repository: lifechurch/dnaquery
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v0.1.0
Choose a head ref
  • 20 commits
  • 953 files changed
  • 2 contributors

Commits on Sep 23, 2017

  1. slight improvements

    chrisvaughn committed Sep 23, 2017
    Copy the full SHA
    d887b1b View commit details

Commits on Oct 27, 2017

  1. Copy the full SHA
    c3fbecd View commit details

Commits on Oct 28, 2017

  1. Copy the full SHA
    789a948 View commit details
  2. Copy the full SHA
    2c31cfe View commit details

Commits on Oct 30, 2017

  1. Copy the full SHA
    1908201 View commit details

Commits on Nov 2, 2017

  1. Copy the full SHA
    d0144b1 View commit details
  2. Copy the full SHA
    fad7fda View commit details

Commits on Nov 7, 2017

  1. Copy the full SHA
    9a401ac View commit details

Commits on Nov 8, 2017

  1. update deps

    chrisvaughn committed Nov 8, 2017
    Copy the full SHA
    87ce872 View commit details
  2. Copy the full SHA
    8e53941 View commit details

Commits on Nov 9, 2017

  1. tweaks to logging

    chrisvaughn committed Nov 9, 2017
    Copy the full SHA
    9307fa2 View commit details
  2. Copy the full SHA
    a5a81f8 View commit details

Commits on Nov 12, 2017

  1. Create LICENSE

    chrisvaughn authored Nov 12, 2017
    Copy the full SHA
    589bf2d View commit details
  2. Merge pull request #1 from lifechurch/add-license-1

    Create LICENSE
    chrisvaughn authored Nov 12, 2017
    Copy the full SHA
    7bec3b2 View commit details
  3. Create README.md

    chrisvaughn authored Nov 12, 2017
    Copy the full SHA
    c96334f View commit details
  4. a bit more in readme

    chrisvaughn committed Nov 12, 2017
    Copy the full SHA
    cc18dd3 View commit details
  5. Copy the full SHA
    13141b9 View commit details
  6. Copy the full SHA
    a071317 View commit details
  7. Copy the full SHA
    0548d2a View commit details
  8. Copy the full SHA
    c4ac4f5 View commit details
Showing 953 changed files with 383,556 additions and 30 deletions.
13 changes: 13 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
dnaquery
dnaquery.toml
gcp_credentials.json

# Test binary, build with `go test -c`
*.test

# Output of the go coverage tool, specifically when used with LiteIDE
*.out


logs/
results/
92 changes: 92 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Contributing

When contributing to this repository, please first discuss the change you wish to make via issue,
email, or any other method with the owners of this repository before making a change.

Please note we have a code of conduct, please follow it in all your interactions with the project.

## Pull Request Process

1. Ensure any install or build dependencies are removed before the end of the layer when doing a
build.
2. Update the README.md with details of changes to the interface, this includes new environment
variables, exposed ports, useful file locations and container parameters.
3. Increase the version numbers in any examples files and the README.md to the new version that this
Pull Request would represent. The versioning scheme we use is [SemVer](http://semver.org/).
4. You may merge the Pull Request in once you have the sign-off of two other developers, or if you
do not have permission to do that, you may request the second reviewer to merge it for you.

## Code of Conduct

### Our Pledge

In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to making participation in our project and
our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, gender identity and expression, level of experience,
nationality, personal appearance, race, religion, or sexual identity and
orientation.

### Our Standards

Examples of behavior that contributes to creating a positive environment
include:

* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members

Examples of unacceptable behavior by participants include:

* The use of sexualized language or imagery and unwelcome sexual attention or
advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic
address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting

### Our Responsibilities

Project maintainers are responsible for clarifying the standards of acceptable
behavior and are expected to take appropriate and fair corrective action in
response to any instances of unacceptable behavior.

Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct, or to ban temporarily or
permanently any contributor for other behaviors that they deem inappropriate,
threatening, offensive, or harmful.

### Scope

This Code of Conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community. Examples of
representing a project or community include using an official project e-mail
address, posting via an official social media account, or acting as an appointed
representative at an online or offline event. Representation of a project may be
further defined and clarified by project maintainers.

### Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team at [INSERT EMAIL ADDRESS]. All
complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
Further details of specific enforcement policies may be posted separately.

Project maintainers who do not follow or enforce the Code of Conduct in good
faith may face temporary or permanent repercussions as determined by other
members of the project's leadership.

### Attribution

This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
available at [http://contributor-covenant.org/version/1/4][version]

[homepage]: http://contributor-covenant.org
[version]: http://contributor-covenant.org/version/1/4/
98 changes: 98 additions & 0 deletions Gopkg.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

42 changes: 42 additions & 0 deletions Gopkg.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@

# Gopkg.toml example
#
# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md
# for detailed Gopkg.toml documentation.
#
# required = ["github.com/user/thing/cmd/thing"]
# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
#
# [[constraint]]
# name = "github.com/user/project"
# version = "1.0.0"
#
# [[constraint]]
# name = "github.com/user/project2"
# branch = "dev"
# source = "github.com/myfork/project2"
#
# [[override]]
# name = "github.com/x/y"
# version = "2.4.0"


[[constraint]]
name = "cloud.google.com/go"
version = "0.16.0"

[[constraint]]
name = "github.com/aws/aws-sdk-go"
version = "1.12.24"

[[constraint]]
branch = "master"
name = "github.com/buger/jsonparser"

[[constraint]]
name = "github.com/pelletier/go-toml"
version = "1.0.1"

[[constraint]]
branch = "master"
name = "google.golang.org/api"
21 changes: 21 additions & 0 deletions LICENSE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2017 Life.Church

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
105 changes: 105 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# DNAQuery

DNAQuery is a command line utility to take [logdna](https://logdna.com) archives and load them into [BigQuery](https://cloud.google.com/bigquery/). This allows long term queryable storage of logs in BigQuery which can be considerably most cost effective. In our use case we want real time access to lots of different logs and long term storage of a subset of those logs. This approach has helped us find a balance between access & cost.

## Getting Started

`go get -u github.com/lifechurch/dnaquery`

DNAQuery has been tested on Go v1.9

## Prerequisites

- logdna account must have [Archiving](https://docs.logdna.com/docs/archiving) to S3 enabled
- a [Google Cloud Platform](https://cloud.google.com) account

## Configuring

`cp example.toml dnaquery.toml`

edit dnaquery.toml

All of these settings are currently required.

### AWS
Note: It's recommended that you create a new [IAM user](https://console.aws.amazon.com/iam/home) with `s3:GetObject` permission on the logdna archive previously setup

```
[aws]
Key = "key123123123132" # Key for AWS IAM user
Secret = "secret123123123123" # Secret for AWS IAM user
Bucket = "logs" # name of bucket set in logdna archive setup above
LogPrefix = "a7112abc9d" # each archive file starts with a prefix specific to your logdna account
```

### Storage

```
[storage]
LogDirectory = 'logs' # temp local directory to download logs to and store results, this directory will be created if it doesn't exist
```

### Containers
Note: the term container here is used to be consistent with logdna's terminology
```
[[containers]]
Name = "production-app" # name of container
Regex = '^([\d.]+) - \[([^\]]*)\] - - \[([^\]]*)\] "([^"]*)" (\d+) (\d+) "([^"]*)" "([^"]*)" (\d+) ([\d.]+) ([\d.:]+) (\d+) ([\d.]+) (\d+)$' # regex used to pull parts of logs out, currently we dont' use any named capture groups
TimeGroup = 3 # the number of the capture group (1-based) in above regex that holds the time component of the log
TimeFormat = "2/Jan/2006:15:04:05 -0700" # the format of the time field using https://golang.org/pkg/time/#Parse
[[containers.excludes]] # an array of tables for exclusions
Group = 4 # the number of the capture group in above regex to be used in the exclusion check
Contains = "ping" # excludes logs if the value here is contained in the string in the above capture group
```

### GCP
```
[gcp] # Google Cloud Project settings
ProjectID = "gcpproj" # name of GCP project
CredentialsFile = "gcp_credentials.json" # path to the credentials file downloaded from GCP
Bucket = "logdna_to_bq" # name of bucket in Cloud Storage to save results for ingestion into BigQuery, bucket will need to be created before first run
Dataset = "logdna" # BigQuery dataset
TemplateTable = "logdna" # currently DNAQuery uses a template table. More details below.
```

#### TemplateTable

Currently the schema for the BigQuery table is specified by creating a table with the schema that matches the regex. This is likely to change in the future as we may need a different schema for each container above.

## Usage

currently dnaquery requires a config file `dnaquery.toml` in the current directory

`dnaquery 2017-11-20`

## Tests

`go test`

## Dependencies

This project uses [dep](https://github.com/golang/de) for dependency management

```
go get -u github.com/golang/dep/cmd/dep
dep ensure -update; dep prune
```

## Contributing

Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on our code of conduct, and the process
for submitting pull requests.

## Versioning

We use [SemVer](http://semver.org/) for versioning. For the versions available, see the [tags on this repository](https://github.com/lifechurch/dnaquery/tags).

## Authors

* **Chris Vaughn** - *Initial work* - [chrisvaughn](https://github.com/chrisvaughn)

See also the list of [contributors](https://github.com/lifechurch/dnaquery/contributors) who participated in this project.

## License

This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details
24 changes: 24 additions & 0 deletions example.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
[aws]
Key = "key123123123132"
Secret = "secret123123123123"
Bucket = "logs"
LogPrefix = "a7112abc9d"

[storage]
LogDirectory = 'logs'

[[containers]]
Name = "production-app"
Regex = '^([\d.]+) - \[([^\]]*)\] - - \[([^\]]*)\] "([^"]*)" (\d+) (\d+) "([^"]*)" "([^"]*)" (\d+) ([\d.]+) ([\d.:]+) (\d+) ([\d.]+) (\d+)$'
TimeGroup = 3
TimeFormat = "2/Jan/2006:15:04:05 -0700"
[[containers.excludes]]
Group = 4
Contains = "ping"

[gcp]
ProjectID = "gcpproj"
CredentialsFile = "gcp_credentials.json"
Bucket = "logdna_to_bq"
Dataset = "logdna"
TemplateTable = "logdna"
397 changes: 367 additions & 30 deletions main.go

Large diffs are not rendered by default.

21 changes: 21 additions & 0 deletions vendor/cloud.google.com/go/.travis.yml
15 changes: 15 additions & 0 deletions vendor/cloud.google.com/go/AUTHORS
152 changes: 152 additions & 0 deletions vendor/cloud.google.com/go/CONTRIBUTING.md
37 changes: 37 additions & 0 deletions vendor/cloud.google.com/go/CONTRIBUTORS
202 changes: 202 additions & 0 deletions vendor/cloud.google.com/go/LICENSE
54 changes: 54 additions & 0 deletions vendor/cloud.google.com/go/MIGRATION.md
541 changes: 541 additions & 0 deletions vendor/cloud.google.com/go/README.md

Large diffs are not rendered by default.

32 changes: 32 additions & 0 deletions vendor/cloud.google.com/go/appveyor.yml
49 changes: 49 additions & 0 deletions vendor/cloud.google.com/go/authexample_test.go
156 changes: 156 additions & 0 deletions vendor/cloud.google.com/go/bigquery/bigquery.go
101 changes: 101 additions & 0 deletions vendor/cloud.google.com/go/bigquery/copy.go
141 changes: 141 additions & 0 deletions vendor/cloud.google.com/go/bigquery/copy_test.go
501 changes: 501 additions & 0 deletions vendor/cloud.google.com/go/bigquery/dataset.go

Large diffs are not rendered by default.

328 changes: 328 additions & 0 deletions vendor/cloud.google.com/go/bigquery/dataset_test.go
296 changes: 296 additions & 0 deletions vendor/cloud.google.com/go/bigquery/doc.go
82 changes: 82 additions & 0 deletions vendor/cloud.google.com/go/bigquery/error.go
110 changes: 110 additions & 0 deletions vendor/cloud.google.com/go/bigquery/error_test.go
757 changes: 757 additions & 0 deletions vendor/cloud.google.com/go/bigquery/examples_test.go

Large diffs are not rendered by default.

398 changes: 398 additions & 0 deletions vendor/cloud.google.com/go/bigquery/external.go
143 changes: 143 additions & 0 deletions vendor/cloud.google.com/go/bigquery/external_test.go
105 changes: 105 additions & 0 deletions vendor/cloud.google.com/go/bigquery/extract.go
118 changes: 118 additions & 0 deletions vendor/cloud.google.com/go/bigquery/extract_test.go
135 changes: 135 additions & 0 deletions vendor/cloud.google.com/go/bigquery/file.go
98 changes: 98 additions & 0 deletions vendor/cloud.google.com/go/bigquery/file_test.go
73 changes: 73 additions & 0 deletions vendor/cloud.google.com/go/bigquery/gcs.go
Loading