Skip to content

Commit

Permalink
Revise README, Getting Started, and flow visibility documents
Browse files Browse the repository at this point in the history
Add Theia overview, getting-started, contributing, community, and
license information to README.md.
Move Getting Started to a separate document, and add more information
about Theia features and usage.
section.
Rename theia.md to theia-cli.md.

Signed-off-by: Jianjun Shen <shenj@vmware.com>
  • Loading branch information
jianjuns committed Jun 14, 2022
1 parent 27afadc commit 479d146
Show file tree
Hide file tree
Showing 5 changed files with 177 additions and 84 deletions.
51 changes: 43 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,45 @@
# Theia

Theia contains Network Flow Visibility functions which are exacted from the
[Antrea main repo](https://github.com/antrea-io/antrea). While flow exporter and
flow aggregator are kept in the Antrea repo, other flow visibility modules, such
as ClickHouse and Grafana related ones, will be moved to this new repo.

During the code migration period, we will still keep related functions available
and stable in the Antrea repo, and the functions in this new repo won't be ready
until we announce the migration is completed.
Theia is a network observability and analytics platform for Kubernetes. It is
built on top of [Antrea](https://github.com/antrea-io/antrea), and consumes
[network flows exported by Antrea](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md)
to provide fine-grained visibility into the communication and NetworkPolicies
among Pods and Services in a Kubernetes cluster.

## Getting Started

Getting started with Theia is simple. You can follow the [Getting Started](docs/getting-started.md)
guide to install Theia and start rocking!

Theia supports network flow visualization and monitoring with Grafana. Check the
[Network Flow Visibility](docs/network-flow-visibility.md) document to learn
more.

Based on the collected network flow information, Theia can recommend appropriate
NetworkPolicy configuration to secure Kubernetes network and applications.
Please refer to the [NetworkPolicy Recommendation](docs/networkpolicy-recommendation.md)
user guide to learn more.

## Contributing

The Antrea community welcomes new contributors. We are waiting for your PRs!

* Before contributing, please get familiar with our
[Code of Conduct](CODE_OF_CONDUCT.md).
* Check out the Antrea [Contributor Guide](CONTRIBUTING.md) for information
about setting up your development environment and our contribution workflow.
* Learn about Antrea's [Architecture and Design](https://github.com/antrea-io/antrea/blob/main/docs/design/architecture.md).
Your feedback is more than welcome!
* Check out [Open Issues](https://github.com/antrea-io/theia/issues).
* Join the Antrea [community](#community) and ask us any question you may have.

## Community

Please refer to the [Antrea community](https://github.com/antrea-io/antrea/blob/main/README.md#community)
information.

## License

Theia is licensed under the [Apache License, version 2.0](LICENSE)

[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fantrea-io%2Fantrea.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fantrea-io%2Fantrea?ref=badge_large)
97 changes: 97 additions & 0 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Getting Started with Theia

## Table of Contents

<!-- toc -->
- [Overview](#overview)
- [Prerequisites](#prerequisites)
- [Theia Installation](#theia-installation)
- [Features](#features)
- [Network Flow Visualization and Monitoring](#network-flow-visualization-and-monitoring)
- [NetworkPolicy Recommendation](#networkpolicy-recommendation)
- [Additional Information](#additional-information)
<!-- /toc -->

## Overview

Theia is a network observability and analytics platform for Kubernetes, built
on top of [Antrea](https://github.com/antrea-io/antrea). Theia consumes [network
flows exported by Antrea](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md)
to provide fine-grained visibility into the communication and NetworkPolicies
among Pods and Services in a Kubernetes cluster.

Theia supports network flow visualization and monitoring with Grafana, and can
recommend appropriate NetworkPolicy configuration to secure Kubernetes network
and applications. This guide describes how to install and get started with
Theia.

## Prerequisites

Theia requires that Antrea v1.7.0 or later is installed in the Kubernetes
cluster. Ensure the Flow Exporter feature of Antrea Agent is enabled in the
Antrea deployment manifest:

```yaml
antrea-agent.conf: |
...
featureGates:
...
FlowExporter: true
```
## Theia Installation
To enable both [Grafana Flow Collector](network-flow-visibility.md) and
[NetworkPolicy Recommendation](networkpolicy-recommendation.md), please install
Flow Aggregator and Theia by runnning the following commands:
```bash
git clone https://github.com/antrea-io/theia.git
helm install flow-aggregator theia/build/charts/flow-aggregator --set clickHouse.enable=true,recordContents.podLabels=true -n flow-aggregator --create-namespace
helm install theia theia/build/charts/theia --set sparkOperator.enable=true -n flow-visibility --create-namespace
```

To enable only Grafana Flow Collector, please install Flow Aggregator and Theia
by runnning the following commands:

```bash
git clone https://github.com/antrea-io/theia.git
helm install flow-aggregator theia/build/charts/flow-aggregator --set clickHouse.enable=true -n flow-aggregator --create-namespace
helm install theia theia/build/charts/theia -n flow-visibility --create-namespace
```

## Features

### Network Flow Visualization and Monitoring

Theia uses Grafana to visualize network flows in the Kubernetes cluster. After
the installation, you can run the following commands to get the Grafana Service
address:

```bash
NODE_NAME=$(kubectl get pod -l app=grafana -n flow-visibility -o jsonpath='{.items[0].spec.nodeName}')
NODE_IP=$(kubectl get nodes ${NODE_NAME} -o jsonpath='{.status.addresses[0].address}')
GRAFANA_NODEPORT=$(kubectl get svc grafana -n flow-visibility -o jsonpath='{.spec.ports[*].nodePort}')
echo "=== Grafana Service is listening on ${NODE_IP}:${GRAFANA_NODEPORT} ==="
```

You can access Grafana in your browser at: `http://[NodeIP]:[NodePort]`, and log
in with username: `admin` and password: `admin`. Navigate to the [Theia
dashboards](network-flow-visibility.md#grafana-dashboards) to view the network
flows in the cluster.

### NetworkPolicy Recommendation

Please follow the instructions in the [NetworkPolicy Recommendation](networkpolicy-recommendation.md)
user guide.

## Additional Information

Refer to Antrea documentation to learn more about
[Flow Exporter](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md#flow-exporter),
[Flow Aggregator](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md#flow-aggregator),
and their advanced configurations.

For more information about Grafana Flow Collector installation and
customization, please refer to Grafana Flow Collector [Deployment Steps](network-flow-visibility.md#deployment-steps),
and [Configuration](network-flow-visibility.md#configuration).
107 changes: 34 additions & 73 deletions docs/network-flow-visibility.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,9 @@
# Theia: Network Flow Visibility for Antrea
# Network Flow Visibility with Theia

## Table of Contents

<!-- toc -->
- [Overview](#overview)
- [Getting Started](#getting-started)
- [Prerequisites](#prerequisites)
- [Theia Installation](#theia-installation)
- [Additional information](#additional-information)
- [Grafana Flow Collector](#grafana-flow-collector)
- [Purpose](#purpose)
- [About Grafana and ClickHouse](#about-grafana-and-clickhouse)
Expand All @@ -22,85 +18,39 @@
- [Credentials Configuration](#credentials-configuration-1)
- [Service Customization](#service-customization-1)
- [Performance Configuration](#performance-configuration)
- [Persistent Volumes](#persistent-volumes)
- [Persistent Volumes](#persistent-volumes)
- [Grafana Dashboards](#grafana-dashboards)
- [Pre-built Dashboards](#pre-built-dashboards)
- [Flow Records Dashboard](#flow-records-dashboard)
- [Pod-to-Pod Flows Dashboard](#pod-to-pod-flows-dashboard)
- [Pod-to-External Flows Dashboard](#pod-to-external-flows-dashboard)
- [Pod-to-Service Flows Dashboard](#pod-to-service-flows-dashboard)
- [Node-to-Node Flows Dashboard](#node-to-node-flows-dashboard)
- [Network-Policy Flows Dashboard](#network-policy-flows-dashboard)
- [Dashboards Customization](#dashboards-customization)
- [Dashboard Customization](#dashboard-customization)
<!-- /toc -->

## Overview

[Antrea](https://github.com/antrea-io/antrea/blob/main/docs/design/architecture.md)
is a Kubernetes network plugin that provides network connectivity and security
features for Pod workloads. Considering the scale and dynamism of Kubernetes
workloads in a cluster, Network Flow Visibility helps in the management and
configuration of Kubernetes resources such as Network Policy, Services, Pods
etc., and thereby provides opportunities to enhance the performance and security
aspects of Pod workloads.

For visualizing the network flows, Antrea monitors the flows in Linux conntrack
module. These flows are converted to flow records, and then flow records are post-processed
before they are sent to the configured external flow collector.

## Getting Started

### Prerequisites

To install Theia, ensure the Flow Exporter feature is enabled in Antrea deployment
manifest, in the following way:

```yaml
antrea-agent.conf: |
...
featureGates:
...
FlowExporter: true
```
### Theia Installation
To enable both Grafana Flow Collector and
[NetworkPolicy Recommendation](networkpolicy-recommendation.md), please install
Theia and Flow Aggregator by runnning the following commands:
```bash
git clone https://github.com/antrea-io/theia.git
helm install flow-aggregator theia/build/charts/flow-aggregator --set clickHouse.enable=true,recordContents.podLabels=true -n flow-aggregator --create-namespace
helm install theia theia/build/charts/theia --set sparkOperator.enable=true -n flow-visibility --create-namespace
```

To enable only Grafana Flow Collector, please install Theia and Flow Aggregator
by runnning the following commands:

```bash
git clone https://github.com/antrea-io/theia.git
helm install flow-aggregator theia/build/charts/flow-aggregator --set clickHouse.enable=true -n flow-aggregator --create-namespace
helm install theia theia/build/charts/theia -n flow-visibility --create-namespace
```

### Additional information

Refer to Antrea documentation to learn more about
[Flow Exporter](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md#flow-exporter),
[Flow Aggregator](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md#flow-aggregator)
and their advanced configurations.
Theia is a network observability and analytics platform for Kubernetes, built
on top of [Antrea](https://github.com/antrea-io/antrea). It supports network
flow visualization and monitoring with Grafana. This document describes the
Grafana Flow Collector and network flow visualization functionality of Theia.

## Grafana Flow Collector

### Purpose

Antrea supports sending IPFIX flow records through the Flow Exporter and Flow Aggregator
feature described in [doc](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md).
The Grafana Flow Collector works as the visualization tool for flow records and
flow-related information. We use ClickHouse as the data storage, which collects flow
records data from the Flow Aggregator and load the data to Grafana. This document
provides the guidelines for deploying the Grafana Flow Collector with support for
Antrea-specific IPFIX fields in a Kubernetes cluster.
Antrea supports exporting IPFIX flow records with Kubernetes workload and
NetworkPolicy information and sending them to a flow collector, through the Flow
Exporter and Flow Aggregator features as described in the [Antrea Network Flow
Visibility](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md)
document. The Grafana Flow Collector works as the visualization and monitoring
tool for flow records and flow-related information. It uses ClickHouse as the
data storage, which collects flow records data from the Flow Aggregator and
loads the data to Grafana. This document provides the guidelines for deploying
the Grafana Flow Collector with support for Antrea-specific IPFIX fields in a
Kubernetes cluster.

### About Grafana and ClickHouse

Expand All @@ -112,6 +62,11 @@ ClickHouse as the data storage, and use Grafana as the data visualization and mo

### Deployment Steps

This section talks about detailed steps to deploy the Grafana Flow Collector
using Helm charts or YAML manifests. For quick steps of installing Theia
including the Grafana Flow Collector, please refer to the [Getting Started](getting-started.md)
guide.

We support deploying the Grafana Flow Collector with Helm. Here is the
[Helm chart](../build/charts/theia/) for the Grafana Flow Collector. Please follow
the instructions from the Helm chart [README](../build/charts/theia/README.md)
Expand Down Expand Up @@ -156,7 +111,7 @@ kubectl apply -f https://raw.githubusercontent.com/antrea-io/theia/main/build/ya
Run the following command to check if ClickHouse and Grafana are deployed properly:

```bash
kubectl get all -n flow-visibility
kubectl get all -n flow-visibility
```

The expected results will be like:
Expand Down Expand Up @@ -217,6 +172,10 @@ kubectl delete -f https://raw.githubusercontent.com/antrea-io/theia/main/build/c

### Configuration

So far, we went through Grafana Flow Collector deployment with the default
configuration. The Grafana Flow Collector also exposes a few configuration
parameters for you to customize the deployment. Read this section to learn more.

#### With Helm

If you install the Grafana Flow Collector using the Helm command, please refer
Expand Down Expand Up @@ -448,7 +407,7 @@ The time interval between the batch commits to the ClickHouse is specified in th
as `commitInterval`. The ClickHouse throughput grows sightly when the commit interval
grows from 1s to 8s. A commit interval larger than 8s provides little improvement on the throughput.

##### Persistent Volumes
###### Persistent Volumes

By default, ClickHouse is deployed in memory. We support deploying ClickHouse with
Persistent Volumes.
Expand Down Expand Up @@ -594,9 +553,11 @@ should be set to your storage size.
name: clickhouse-storage-volume
```

## Grafana Dashboards

### Pre-built Dashboards

The following dashboards are pre-built and are recommended for Antrea flow
The following dashboards are pre-built and are recommended for Theia flow
visualization. They can be found in the Home page of Grafana, by clicking
the Magnifier button on the left menu bar.
<img src="https://downloads.antrea.io/static/02152022/flow-visibility-grafana-intro-1.png" width="900" alt="Grafana Search Dashboards Guide">
Expand All @@ -620,7 +581,7 @@ and resources to render them.

If you want to stop filtering traffic by Namespace, or edit the panel limit,
you will need to edit the ClickHouse SQL query for each panel. Please follow
the [dashboards customization](#dashboards-customization) section for more
the [Dashboard Customization](#dashboard-customization) section for more
information. As a special case, to edit the panel limit for pie charts, instead
of editing the query, please follow the
[doc](https://grafana.com/docs/grafana/latest/visualizations/pie-chart-panel/#limit)
Expand Down Expand Up @@ -728,7 +689,7 @@ Mouse out or click on the background will bring all the traffic back.

<img src="https://downloads.antrea.io/static/05232022/flow-visibility-np-3.png" width="900" alt="Network-Policy Flows Dashboard">

### Dashboards Customization
### Dashboard Customization

If you would like to make any changes to any of the pre-built dashboards, or build
a new dashboard, please follow this [doc](https://grafana.com/docs/grafana/latest/dashboards/)
Expand Down
6 changes: 3 additions & 3 deletions docs/networkpolicy-recommendation.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,15 @@ their applications according to Zero Trust principles.

## Prerequisite

Please follow [Getting Started](network-flow-visibility.md#getting-started) to
install Antrea Flow Aggregator and Theia.
Please follow the [Getting Started](getting-started.md) guide to install Antrea
Flow Aggregator and Theia.

## Perform NetworkPolicy Recommendation

Users can leverage Theia's NetworkPolicy Recommendation feature through `theia`
CLI. `theia` is the command-line tool which provides access to Theia network
flow visibility capabilities. To get more information about `theia`, please
refer to this [doc](theia.md).
refer to its [user guide](theia-cli.md).

There are 3 `theia` commands for the NetworkPolicy Recommendation feature:

Expand Down
File renamed without changes.

0 comments on commit 479d146

Please sign in to comment.