Skip to content
This repository has been archived by the owner on Nov 8, 2022. It is now read-only.

Commit

Permalink
Merge pull request #617 from tiffanyfj/documentation
Browse files Browse the repository at this point in the history 
Refactored readmes for building snap
  • Loading branch information
@tiffanyfay
tiffanyfay committed Dec 18, 2015
2 parents 2f1e2e1 + eaa1931 commit 912fdcb
Show file tree
Hide file tree
Showing 15 changed files with 303 additions and 278 deletions.
97 changes: 1 addition & 96 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,99 +47,4 @@ When reporting an issue, details are key. Include the following:
- Environment details (virtual, physical, etc.)
- Steps to reproduce
- Actual results
- Expected results

##Build and Test
###Build
In the /snap directory there's a `Makefile` that builds all dependencies and snap.
To get dependencies and build snap just run:
```
make
```

Alternatively, you can run `make` with any of the following targets:

* `default`: runs make deps and make all
* `deps`: fetches all dependencies using godeps
* `check`: runs test suite
* `all`: builds snapd, snapctl, and the test plugins
* `snap` builds snapd and snapctl
* `install`: installs snapd and snapctl binaries in /usr/local/bin
* `release`: cuts a snap release

###Test
####Creating Tests
Our tests are written using [smartystreets' GoConvey package](https://github.com/smartystreets/goconvey). See https://github.com/smartystreets/goconvey/wiki for an introduction to creating a test.

####Running Tests
#####In local machine
To run all tests in order (it will stop at any failure in a directory before continuing on):
```
./scripts/test.sh
```

TO run all tests *and to continue through all directories even with errors*:
```
go test ./...
```

Tests can be pruned in go test with the `-run` option:
```
go test ./... -v -run <TestName>
```

e.g. `TestLoad` from `control_test.go`:
```
go test ./... -v -run TestLoad
```

e.g. using GoConvey UX:
```
go test -coverprofile=/tmp/coverage.out && go tool cover -html=/tmp/coverage.out
```

#####In Docker
There's a `Dockerfile` located at `./scripts/Dockerfile`:
```
FROM golang:latest
ENV GOPATH=$GOPATH:/app
ENV SNAP_PATH=/go/src/github.com/intelsdi-x/snap/build
RUN apt-get update && \
apt-get -y install facter
WORKDIR /go/src/github.com/intelsdi-x/
RUN git clone https://<GIT_TOKEN>@github.com/intelsdi-x/gomit.git
WORKDIR /go/src/github.com/intelsdi-x/snap
ADD . /go/src/github.com/intelsdi-x/snap
RUN go get github.com/tools/godep && \
go get golang.org/x/tools/cmd/goimports && \
go get golang.org/x/tools/cmd/vet && \
go get golang.org/x/tools/cmd/cover && \
go get github.com/smartystreets/goconvey
RUN scripts/deps.sh
RUN make
```
This is run in the snap directory using `./scripts/run_tests_with_docker.sh`
First you need a [github personal access token](https://help.github.com/articles/creating-an-access-token-for-command-line-use/)
Then export the token using `export GIT_TOKEN=<tokenID>`.

```
#!/bin/bash -e
die() {
echo >&2 $@
exit 1
}
if [ $# -eq 2 ]; then
GIT_TOKEN=$1
fi
if [ -z "${GIT_TOKEN}" ]; then
die "arg missing: github token is required so we can clone a private repo)"
fi
sed s/\<GIT_TOKEN\>/${GIT_TOKEN}/ scripts/Dockerfile > scripts/Dockerfile.tmp
docker build -t intelsdi-x/snap-test -f scripts/Dockerfile.tmp .
rm scripts/Dockerfile.tmp
docker run -it intelsdi-x/snap-test scripts/test.sh
```
- Expected results
81 changes: 44 additions & 37 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -58,26 +58,26 @@ Here are blog posts related to ***snap*** by the team:
**snap** is a framework for enabling the gathering of telemetry from systems. The goals of this project are to:

* Empower systems to expose a consistent set of telemetry data
* Simplify telemetry ingestion across ubiquitous storage system
* Simplify telemetry ingestion across ubiquitous storage systems
* Improve the deployment model, packaging and flexibility for collecting telemetry
* Allow flexible processing of telemetry data on agent (e.g. machine learning)
* Provide powerful clustered control of telemetry workflows across small or large clusters

The key features of snap are:

* **Plugin Architecture**: snap has a simple and smart modular design. The three types of plugins (collectors, processors, and publishers) allow snap to mix and match functionality based on user need. All plugins are designed with versioning, signing and deployment at scale in mind. The open plugin model allows for loading built-in, community, or proprietary plugins into snap.
* **Plugin Architecture**: snap has a simple and smart modular design. The three types of plugins (collectors, processors, and publishers) allow snap to mix and match functionality based on user need. All plugins are designed with versioning, signing and deployment at scale in mind. The **open plugin model** allows for loading built-in, community, or proprietary plugins into snap.
* **Collectors** - Collectors consume telemetry data. Collectors are built-in plugins for leveraging existing telemetry solutions (Facter, CollectD, Ohai) as well as specific plugins for consuming Intel telemetry (Node, DCM, NIC, Disk) and can reach into new architectures through additional plugins (see [Plugin Authoring below](#author-a-plugin)). Telemetry data is organized into a dynamically generated catalog of available data points.
* **Processors** - Extensible workflow injection. Convert telemetry into another data model for consumption by existing consumption systems (like OpenStack Ceilometer). Allows encryption of all or part of the telemetry payload before publishing. Inject remote queries into workflow for tokens, filtering, or other external calls. Implement filtering at an agent level reducing injection load on telemetry consumer.
* **Publishers** - Store telemetry into a wide array of systems. snap decouples the collection of telemetry from the implementation of where to send it. snap comes with a large library of publisher plugins that allow exposure to telemetry analytics systems both custom and common. This flexibility allows snap to be valuable to open source and commercial ecosystems alike by writing a publisher for their architectures.


* **Dynamic Updates**: snap is designed to evolve. Each scheduled workflow automatically uses the most mature plugin for that step, unless the collection is pinned to a specific version (ex: get /intel/server/cpu/load/v1). Loading a new plugin automatically upgrades running workflows in tasks. Load plugins dynamically, without a restart to the service or server. This dynamically extends the metric catalog when loaded, giving access to new measurements immediately. Swapping a newer version plugin for an old one in a safe transaction. All of these behaviors allow for simple and secure bug fixes, security patching, and improving accuracy in production.
* **Dynamic Updates**: snap is designed to evolve. Each scheduled workflow automatically uses the most mature plugin for that step, unless the collection is pinned to a specific version (e.g. get /intel/psutil/load/load1/v1). Loading a new plugin automatically upgrades running workflows in tasks. Load plugins dynamically, without a restart to the service or server. This dynamically extends the metric catalog when loaded, giving access to new measurements immediately. Swapping a newer version plugin for an old one in a safe transaction. All of these behaviors allow for simple and secure bug fixes, security patching, and improving accuracy in production.

* **snap tribe**: snap is designed for ease of administration. With snap tribe, nodes work in groups (aka tribes). Requests are made through agreement- or task-based node groups, designed as a scalable gossip-based node-to-node communication process. Administrators can control all snap nodes in a tribe agreement by messaging just one of them. There is auto-discovery of new nodes and import of tasks & plugins from nodes within a given tribe. It is cluster configuration management made simple.
* **snap tribe**: snap is designed for ease of administration. With snap tribe, nodes work in groups (aka tribes). Requests are made through agreement- or task-based node groups, designed as a scalable gossip-based node-to-node communication process. Administrators can control all snap nodes in a tribe agreement by messaging just one of them. There is auto-discovery of new nodes and import of tasks and plugins from nodes within a given tribe. It is cluster configuration management made simple.

Some additionally important notes about how snap works:

* Multiple management modules including: CLI, REST API, and Web Console (each of which can be turned on or off)
* Multiple management modules including: [CLI](docs/SNAPCTL.md) (snapctl) and [REST API](docs/REST_API.md) (each of which can be turned on or off)
* Secure validation occurs via plugin signing, SSL encryption for APIs and payload encryption for communication between components
* CLI control from Linux or OS X

Expand All @@ -96,32 +96,37 @@ Snap deploys as a binary, which makes requirements quite simple. We've tested on

### Installation

You can get the pre-built binaries for your OS and architecture at snap's [GitHub Releases](https://github.com/intelsdi-x/snap/releases) page. This isn't the comprehensive list of plugins, but they will help you get started. Right now, snap only supports Linux and OS X.
You can get the pre-built binaries for your OS and architecture at snap's [GitHub Releases](https://github.com/intelsdi-x/snap/releases) page. This isn't the comprehensive list of plugins, but they will help you get started. Right now, snap only supports Linux and OS X (Darwin).

If you're looking for the bleeding edge of snap, you can build it by cloning down the `master` branch. To build snap from source, you will need [Golang >= 1.4](https://golang.org) and [GNU Make](https://www.gnu.org/software/make/). More on building snap [here](./CONTRIBUTING.md).
#### Building snap
If you're looking for the bleeding edge of snap, you can build it by getting the `master` branch using `go get github.com/intelsdi-x/snap`. Otherwise you can just use the binaries. To build snap from source, you will need [Golang >= 1.4](https://golang.org) and [GNU Make](https://www.gnu.org/software/make/). For more on building snap check out [BUILD_AND_TEST.md](docs/BUILD_AND_TEST.md).

### Running snap

Start a standalone snap agent:
Start a standalone snap agent (snapd):

If you cloned down the repository and aren't using just downloaded binaries, you can set the following. Also, follow the steps in [BUILD_AND_TEST.md](docs/BUILD_AND_TEST.md):
```
export SNAP_PATH=<snapdirectory>/build
$SNAP_PATH/bin/snapd --plugin-trust 0 --log-level 1
$ export SNAP_PATH=<snapDirectoryPath>/build
```
This will bring up a snap agent without requiring plugin signing and set the logging level to debug. snap's REST API will be listening on port 8181.
And then you can use `$SNAP_PATH` for your `<snapBinPath>`.
```
$ <snapBinPath>/bin/snapd --plugin-trust 0 --log-level 1
```
This will bring up a snap agent without requiring plugin signing and set the logging level to debug. snap's REST API will be listening on port 8181. To learn more about the snap agent and how to use it look at [SNAPD.md](docs/SNAPD.md) and/or run `<snapBinPath>/bin/snapd -h`.

### Running snap in tribe (cluster) mode

The first node
Start the first node:

```
$SNAP_PATH/bin/snapd --tribe
$ <snapBinPath>/bin/snapd --tribe
```

All other nodes who join will need to select any existing member of the cluster.

```
$SNAP_PATH/bin/snapd --tribe-seed <ip or name of another tribe member>
$ <snapBinPath>/bin/snapd --tribe-seed <ip or name of another tribe member>
```

Checkout the [tribe doc](docs/TRIBE.md) for more info.
Expand All @@ -133,35 +138,35 @@ Next, lets load a few of the demo plugins. You can do this via cURL, or `snapct

Using cURL
```sh
curl -X POST -F plugin=@build/plugin/snap-collector-mock1 http://localhost:8181/v1/plugins
curl -X POST -F plugin=@build/plugin/snap-processor-passthru http://localhost:8181/v1/plugins
curl -X POST -F plugin=@build/plugin/snap-publisher-file http://localhost:8181/v1/plugins
$ curl -X POST -F plugin=@build/plugin/snap-collector-mock1 http://localhost:8181/v1/plugins
$ curl -X POST -F plugin=@build/plugin/snap-processor-passthru http://localhost:8181/v1/plugins
$ curl -X POST -F plugin=@build/plugin/snap-publisher-file http://localhost:8181/v1/plugins
```

Or:

Using `snapctl`
```sh
$SNAP_PATH/bin/snapctl plugin load build/plugin/snap-collector-mock1
$SNAP_PATH/bin/snapctl plugin load build/plugin/snap-processor-passthru
$SNAP_PATH/bin/snapctl plugin load build/plugin/snap-publisher-file
Using `snapctl` (snap CLI)
```
$ <snapBinPath>/bin/snapctl plugin load build/plugin/snap-collector-mock1
$ <snapBinPath>/bin/snapctl plugin load build/plugin/snap-processor-passthru
$ <snapBinPath>/bin/snapctl plugin load build/plugin/snap-publisher-file
```

Let's look at what plugins we have loaded now:

```sh
$SNAP_PATH/bin/snapctl plugin list
```
$ <snapBinPath>/bin/snapctl plugin list
NAME VERSION TYPE SIGNED STATUS LOADED TIME
mock1 1 collector false loaded Tue, 17 Nov 2015 14:08:17 PST
mock 1 collector false loaded Tue, 17 Nov 2015 14:08:17 PST
passthru 1 processor false loaded Tue, 17 Nov 2015 14:16:12 PST
file 3 publisher false loaded Tue, 17 Nov 2015 14:16:19 PST
```
### Running Tasks

Tasks can be in JSON or YAML format. Let's start one of the [example tasks](./examples/tasks/mock-file.yaml) from the `examples/` directory:
Tasks can be in JSON or YAML format. Let's start one of the [example tasks](./examples/tasks/mock-file.yaml) from the `examples/tasks/` directory:

```
$SNAP_PATH/bin/snapctl task create -t examples/tasks/mock-file.yaml
$ <snapBinPath>/bin/snapctl task create -t examples/tasks/mock-file.yaml
Using task manifest to create task
Task created
ID: 8b9babad-b3bc-4a16-9e06-1f35664a7679
Expand All @@ -173,12 +178,12 @@ From here, you should be able to do 2 things:

See the data that is being published to the file:
```
$ tail -f /tmp/published
$ tail -f /tmp/snap-published-mock-file.log
```

Or actually tap into the data that snap is collecting:
Or actually tap into the data that snap is collecting using the task ID to watch the task:
```
$SNAP_PATH/bin/snapctl task watch 8b9babad-b3bc-4a16-9e06-1f35664a7679
$ <snapBinPath>/bin/snapctl task watch 8b9babad-b3bc-4a16-9e06-1f35664a7679
```

### Building Tasks
Expand All @@ -190,11 +195,13 @@ All known plugins are tracked in the [plugin catalog](https://github.com/intelsd
If you would like to write your own, read through [Author a Plugin](#author-a-plugin). Let us know if you begin to write one by opening an Issue. When you finish, please open a Pull Request to add yours to the catalog!

## Documentation
Documentation for snap will be kept in this repository for now. We would also like to link to external how-to blog posts as people write them. See our [CONTRIBUTING.md](#contributing) for more details.
Documentation for snap will be kept in this repository for now in the `docs/` directory. We would also like to link to external how-to blog posts as people write them. See our [CONTRIBUTING.md](#contributing) for more details.

* [snapctl](docs/SNAPCTL.md)
* [snapd](docs/SNAPD.md)
* [snapd (snap agent)](docs/SNAPD.md)
* [snapctl (snap CLI)](docs/SNAPCTL.md)
* [build and test](docs/BUILD_AND_TEST.md)
* [REST API](docs/REST_API.md)
* [tasks](docs/TASKS.md)
* [plugin signing](docs/PLUGIN_SIGNING.md)
* [tribe](docs/TRIBE.md)

Expand All @@ -208,13 +215,13 @@ We have a few known features we want to take on from here while we remain open f
* Distributed Workflows (spec pending)
* Workflow Routing (see issue [#539](https://github.com/intelsdi-x/snap/issues/539))

If you would like to propose a feature, please open a Issue that includes RFC in it (for [request for comments](https://en.wikipedia.org/wiki/Request_for_Comments)).
If you would like to propose a feature, please [open an Issue](https://github.com/intelsdi-x/snap/issues)) that includes RFC in it (for [request for comments](https://en.wikipedia.org/wiki/Request_for_Comments)).

## Community Support
This repository is one of **many** projects in the **snap framework**. Discuss your questions about snap by reaching out to us on:
This repository is one of **many** projects in the **snap framework**. Discuss your questions about snap by reaching out to us:

* Through GitHub Issues. Issues is our home for **all** needs: Q&A on everything - installation, request for events, integrations, bug issues, futures. [Open up an Issue](https://github.com/intelsdi-x/snap/issues) and know there's no wrong question for us.
* We also have a Gitter channel opened up on this repository that threads directly into our engineering team Slack (thanks to [Sameroom](https://sameroom.io/)).
* We also have a [Gitter channel](https://gitter.im/intelsdi-x/snap) opened up on this repository that threads directly into our engineering team Slack (thanks to [Sameroom](https://sameroom.io/)).

The full project lives here, at http://github.com/intelsdi-x/snap.

Expand Down Expand Up @@ -264,7 +271,7 @@ Just tag **@intelsdi-x/snap-maintainers** if you need to get some attention on a
</tr>
</table>

We're also looking for new maintainers from the community. Please let us know if you would like to become one by opening an Issue titled "interested in becoming a maintainer."
We're also looking for new maintainers from the community. Please let us know if you would like to become one by opening an Issue titled "interested in becoming a maintainer." We are currently working on a more official process.

## Thank You
And **thank you!** Your contribution, through code and participation, is incredibly important to us.
6 changes: 3 additions & 3 deletions core/serror/serror.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,18 +36,18 @@ type snapError struct {
// The variadic signature allows fields to optionally
// be added at construction.
func New(e error, fields ...map[string]interface{}) *snapError {
// Catch someone trying to wrap a pe around a pe.
// Catch someone trying to wrap a serror around a serror.
// We throw a panic to make them fix this.
if _, ok := e.(SnapError); ok {
panic("You are trying to wrap a snapError around a SnapError. Don't do this.")
panic("You are trying to wrap a snapError around a snapError. Don't do this.")
}

p := &snapError{
err: e,
fields: make(map[string]interface{}),
}

// insert fields into new SnapError
// insert fields into new snapError
for _, f := range fields {
for k, v := range f {
p.fields[k] = v
Expand Down
Loading

0 comments on commit 912fdcb

Please sign in to comment.