Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revise README, Getting Started, and flow visibility documents #45

Merged
merged 1 commit into from
Jun 15, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 43 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,45 @@
# Theia

Theia contains Network Flow Visibility functions which are exacted from the
[Antrea main repo](https://github.com/antrea-io/antrea). While flow exporter and
flow aggregator are kept in the Antrea repo, other flow visibility modules, such
as ClickHouse and Grafana related ones, will be moved to this new repo.

During the code migration period, we will still keep related functions available
and stable in the Antrea repo, and the functions in this new repo won't be ready
until we announce the migration is completed.
Theia is a network observability and analytics platform for Kubernetes. It is
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@salv-orlando : I believe you can write a better version. But could we start from the simple introduction?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to keep the readme simple, I think your intro is great.

I would just replace
"provide fine-grained visibility into the network communications in a Kubernetes cluster"

with

"provide fine-grained visibility into connections across pod and services in a Kubernetes cluster"

Unless you chose "network communications" for a specific reason

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about "provide fine-grained visibility into communications among Pods and Services in a Kubernetes cluster"? I feel people might think connections as TCP connections.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just propose "visibility on workload among Pods, Service and external network" as alternative.

built on top of [Antrea](https://github.com/antrea-io/antrea), and consumes
[network flows exported by Antrea](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md)
to provide fine-grained visibility into the communication and NetworkPolicies
among Pods and Services in a Kubernetes cluster.

## Getting Started

Getting started with Theia is simple. You can follow the [Getting Started](docs/getting-started.md)
guide to install Theia and start rocking!

Theia supports network flow visualization and monitoring with Grafana. Check the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now, Theia only supports flow visibility and NP recommendation for a single cluster. Maybe we should highlight this before we can support more?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel no need to mention not implemented features in README (e.g. we can have a ROADMAP doc for that if we want). There are other K8s visibility solutions support a single cluster only too. Do not think multi-cluster support is a must.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not include "forward-looking" statements in the README files.
As @jianjuns mentioned, we can have them either in the roadmap doc, or maybe simply track them on Github!

[Network Flow Visibility](docs/network-flow-visibility.md) document to learn
more.

Based on the collected network flow information, Theia can recommend appropriate
NetworkPolicy configuration to secure Kubernetes network and applications.
Please refer to the [NetworkPolicy Recommendation](docs/networkpolicy-recommendation.md)
user guide to learn more.

## Contributing

The Antrea community welcomes new contributors. We are waiting for your PRs!

* Before contributing, please get familiar with our
[Code of Conduct](CODE_OF_CONDUCT.md).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took a look at the workflow failure, it seems the CODE_OF_CONDUCT.md is not added? Or shall we pointing to the file in Antrea repo?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will be added in #40

* Check out the Antrea [Contributor Guide](CONTRIBUTING.md) for information
about setting up your development environment and our contribution workflow.
* Learn about Antrea's [Architecture and Design](https://github.com/antrea-io/antrea/blob/main/docs/design/architecture.md).
Your feedback is more than welcome!
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not about this PR, but we also need an arch doc for Theia to integrate Antrea's one.

* Check out [Open Issues](https://github.com/antrea-io/theia/issues).
* Join the Antrea [community](#community) and ask us any question you may have.
dreamtalen marked this conversation as resolved.
Show resolved Hide resolved

## Community

Please refer to the [Antrea community](https://github.com/antrea-io/antrea/blob/main/README.md#community)
information.

## License

Theia is licensed under the [Apache License, version 2.0](LICENSE)

[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fantrea-io%2Fantrea.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fantrea-io%2Fantrea?ref=badge_large)
97 changes: 97 additions & 0 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Getting Started with Theia

## Table of Contents

<!-- toc -->
- [Overview](#overview)
- [Prerequisites](#prerequisites)
- [Theia Installation](#theia-installation)
- [Features](#features)
- [Network Flow Visualization and Monitoring](#network-flow-visualization-and-monitoring)
- [NetworkPolicy Recommendation](#networkpolicy-recommendation)
- [Additional Information](#additional-information)
<!-- /toc -->

## Overview

Theia is a network observability and analytics platform for Kubernetes, built
on top of [Antrea](https://github.com/antrea-io/antrea). Theia consumes [network
flows exported by Antrea](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md)
to provide fine-grained visibility into the communication and NetworkPolicies
among Pods and Services in a Kubernetes cluster.

Theia supports network flow visualization and monitoring with Grafana, and can
recommend appropriate NetworkPolicy configuration to secure Kubernetes network
and applications. This guide describes how to install and get started with
Theia.

## Prerequisites

Theia requires that Antrea v1.7.0 or later is installed in the Kubernetes
cluster. Ensure the Flow Exporter feature of Antrea Agent is enabled in the
Antrea deployment manifest:

```yaml
antrea-agent.conf: |
...
featureGates:
...
FlowExporter: true
```

## Theia Installation

To enable both [Grafana Flow Collector](network-flow-visibility.md) and
[NetworkPolicy Recommendation](networkpolicy-recommendation.md), please install
Flow Aggregator and Theia by runnning the following commands:

```bash
git clone https://github.com/antrea-io/theia.git
helm install flow-aggregator theia/build/charts/flow-aggregator --set clickHouse.enable=true,recordContents.podLabels=true -n flow-aggregator --create-namespace
helm install theia theia/build/charts/theia --set sparkOperator.enable=true -n flow-visibility --create-namespace
```

To enable only Grafana Flow Collector, please install Flow Aggregator and Theia
by runnning the following commands:

```bash
git clone https://github.com/antrea-io/theia.git
helm install flow-aggregator theia/build/charts/flow-aggregator --set clickHouse.enable=true -n flow-aggregator --create-namespace
helm install theia theia/build/charts/theia -n flow-visibility --create-namespace
```

## Features

### Network Flow Visualization and Monitoring

Theia uses Grafana to visualize network flows in the Kubernetes cluster. After
heanlan marked this conversation as resolved.
Show resolved Hide resolved
the installation, you can run the following commands to get the Grafana Service
address:

```bash
NODE_NAME=$(kubectl get pod -l app=grafana -n flow-visibility -o jsonpath='{.items[0].spec.nodeName}')
NODE_IP=$(kubectl get nodes ${NODE_NAME} -o jsonpath='{.status.addresses[0].address}')
GRAFANA_NODEPORT=$(kubectl get svc grafana -n flow-visibility -o jsonpath='{.spec.ports[*].nodePort}')
echo "=== Grafana Service is listening on ${NODE_IP}:${GRAFANA_NODEPORT} ==="
```

You can access Grafana in your browser at: `http://[NodeIP]:[NodePort]`, and log
in with username: `admin` and password: `admin`. Navigate to the [Theia
dashboards](network-flow-visibility.md#grafana-dashboards) to view the network
flows in the cluster.

### NetworkPolicy Recommendation

Please follow the instructions in the [NetworkPolicy Recommendation](networkpolicy-recommendation.md)
user guide.

## Additional Information

Refer to Antrea documentation to learn more about
[Flow Exporter](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md#flow-exporter),
[Flow Aggregator](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md#flow-aggregator),
and their advanced configurations.

For more information about Grafana Flow Collector installation and
customization, please refer to Grafana Flow Collector [Deployment Steps](network-flow-visibility.md#deployment-steps),
and [Configuration](network-flow-visibility.md#configuration).
107 changes: 34 additions & 73 deletions docs/network-flow-visibility.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,9 @@
# Theia: Network Flow Visibility for Antrea
# Network Flow Visibility with Theia

## Table of Contents

<!-- toc -->
- [Overview](#overview)
- [Getting Started](#getting-started)
- [Prerequisites](#prerequisites)
- [Theia Installation](#theia-installation)
- [Additional information](#additional-information)
- [Grafana Flow Collector](#grafana-flow-collector)
- [Purpose](#purpose)
- [About Grafana and ClickHouse](#about-grafana-and-clickhouse)
Expand All @@ -22,85 +18,39 @@
- [Credentials Configuration](#credentials-configuration-1)
- [Service Customization](#service-customization-1)
- [Performance Configuration](#performance-configuration)
- [Persistent Volumes](#persistent-volumes)
- [Persistent Volumes](#persistent-volumes)
- [Grafana Dashboards](#grafana-dashboards)
- [Pre-built Dashboards](#pre-built-dashboards)
- [Flow Records Dashboard](#flow-records-dashboard)
- [Pod-to-Pod Flows Dashboard](#pod-to-pod-flows-dashboard)
- [Pod-to-External Flows Dashboard](#pod-to-external-flows-dashboard)
- [Pod-to-Service Flows Dashboard](#pod-to-service-flows-dashboard)
- [Node-to-Node Flows Dashboard](#node-to-node-flows-dashboard)
- [Network-Policy Flows Dashboard](#network-policy-flows-dashboard)
- [Dashboards Customization](#dashboards-customization)
- [Dashboard Customization](#dashboard-customization)
<!-- /toc -->

## Overview

[Antrea](https://github.com/antrea-io/antrea/blob/main/docs/design/architecture.md)
is a Kubernetes network plugin that provides network connectivity and security
features for Pod workloads. Considering the scale and dynamism of Kubernetes
workloads in a cluster, Network Flow Visibility helps in the management and
configuration of Kubernetes resources such as Network Policy, Services, Pods
etc., and thereby provides opportunities to enhance the performance and security
aspects of Pod workloads.

For visualizing the network flows, Antrea monitors the flows in Linux conntrack
module. These flows are converted to flow records, and then flow records are post-processed
before they are sent to the configured external flow collector.

## Getting Started

### Prerequisites

To install Theia, ensure the Flow Exporter feature is enabled in Antrea deployment
manifest, in the following way:

```yaml
antrea-agent.conf: |
...
featureGates:
...
FlowExporter: true
```

### Theia Installation

To enable both Grafana Flow Collector and
[NetworkPolicy Recommendation](networkpolicy-recommendation.md), please install
Theia and Flow Aggregator by runnning the following commands:

```bash
git clone https://github.com/antrea-io/theia.git
helm install flow-aggregator theia/build/charts/flow-aggregator --set clickHouse.enable=true,recordContents.podLabels=true -n flow-aggregator --create-namespace
helm install theia theia/build/charts/theia --set sparkOperator.enable=true -n flow-visibility --create-namespace
```

To enable only Grafana Flow Collector, please install Theia and Flow Aggregator
by runnning the following commands:

```bash
git clone https://github.com/antrea-io/theia.git
helm install flow-aggregator theia/build/charts/flow-aggregator --set clickHouse.enable=true -n flow-aggregator --create-namespace
helm install theia theia/build/charts/theia -n flow-visibility --create-namespace
```

### Additional information

Refer to Antrea documentation to learn more about
[Flow Exporter](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md#flow-exporter),
[Flow Aggregator](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md#flow-aggregator)
and their advanced configurations.
Theia is a network observability and analytics platform for Kubernetes, built
on top of [Antrea](https://github.com/antrea-io/antrea). It supports network
flow visualization and monitoring with Grafana. This document describes the
Grafana Flow Collector and network flow visualization functionality of Theia.

## Grafana Flow Collector

### Purpose

Antrea supports sending IPFIX flow records through the Flow Exporter and Flow Aggregator
feature described in [doc](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md).
The Grafana Flow Collector works as the visualization tool for flow records and
flow-related information. We use ClickHouse as the data storage, which collects flow
records data from the Flow Aggregator and load the data to Grafana. This document
provides the guidelines for deploying the Grafana Flow Collector with support for
Antrea-specific IPFIX fields in a Kubernetes cluster.
Antrea supports exporting IPFIX flow records with Kubernetes workload and
NetworkPolicy information and sending them to a flow collector, through the Flow
Exporter and Flow Aggregator features as described in the [Antrea Network Flow
Visibility](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md)
document. The Grafana Flow Collector works as the visualization and monitoring
tool for flow records and flow-related information. It uses ClickHouse as the
data storage, which collects flow records data from the Flow Aggregator and
loads the data to Grafana. This document provides the guidelines for deploying
the Grafana Flow Collector with support for Antrea-specific IPFIX fields in a
Kubernetes cluster.

### About Grafana and ClickHouse

Expand All @@ -112,6 +62,11 @@ ClickHouse as the data storage, and use Grafana as the data visualization and mo

### Deployment Steps

This section talks about detailed steps to deploy the Grafana Flow Collector
using Helm charts or YAML manifests. For quick steps of installing Theia
including the Grafana Flow Collector, please refer to the [Getting Started](getting-started.md)
Copy link

@dreamtalen dreamtalen Jun 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting Started link needs to be updated in networkpolicy-recommendation.md L29 too.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated it.

guide.

We support deploying the Grafana Flow Collector with Helm. Here is the
[Helm chart](../build/charts/theia/) for the Grafana Flow Collector. Please follow
the instructions from the Helm chart [README](../build/charts/theia/README.md)
Expand Down Expand Up @@ -156,7 +111,7 @@ kubectl apply -f https://raw.githubusercontent.com/antrea-io/theia/main/build/ya
Run the following command to check if ClickHouse and Grafana are deployed properly:

```bash
kubectl get all -n flow-visibility
kubectl get all -n flow-visibility
```

The expected results will be like:
Expand Down Expand Up @@ -217,6 +172,10 @@ kubectl delete -f https://raw.githubusercontent.com/antrea-io/theia/main/build/c

### Configuration

So far, we went through Grafana Flow Collector deployment with the default
configuration. The Grafana Flow Collector also exposes a few configuration
dreamtalen marked this conversation as resolved.
Show resolved Hide resolved
parameters for you to customize the deployment. Read this section to learn more.

#### With Helm

If you install the Grafana Flow Collector using the Helm command, please refer
Expand Down Expand Up @@ -448,7 +407,7 @@ The time interval between the batch commits to the ClickHouse is specified in th
as `commitInterval`. The ClickHouse throughput grows sightly when the commit interval
grows from 1s to 8s. A commit interval larger than 8s provides little improvement on the throughput.

##### Persistent Volumes
###### Persistent Volumes

By default, ClickHouse is deployed in memory. We support deploying ClickHouse with
Persistent Volumes.
Expand Down Expand Up @@ -594,9 +553,11 @@ should be set to your storage size.
name: clickhouse-storage-volume
```

## Grafana Dashboards

### Pre-built Dashboards

The following dashboards are pre-built and are recommended for Antrea flow
The following dashboards are pre-built and are recommended for Theia flow
visualization. They can be found in the Home page of Grafana, by clicking
the Magnifier button on the left menu bar.
<img src="https://downloads.antrea.io/static/02152022/flow-visibility-grafana-intro-1.png" width="900" alt="Grafana Search Dashboards Guide">
Expand All @@ -620,7 +581,7 @@ and resources to render them.

If you want to stop filtering traffic by Namespace, or edit the panel limit,
you will need to edit the ClickHouse SQL query for each panel. Please follow
the [dashboards customization](#dashboards-customization) section for more
the [Dashboard Customization](#dashboard-customization) section for more
information. As a special case, to edit the panel limit for pie charts, instead
of editing the query, please follow the
[doc](https://grafana.com/docs/grafana/latest/visualizations/pie-chart-panel/#limit)
Expand Down Expand Up @@ -728,7 +689,7 @@ Mouse out or click on the background will bring all the traffic back.

<img src="https://downloads.antrea.io/static/05232022/flow-visibility-np-3.png" width="900" alt="Network-Policy Flows Dashboard">

### Dashboards Customization
### Dashboard Customization

If you would like to make any changes to any of the pre-built dashboards, or build
a new dashboard, please follow this [doc](https://grafana.com/docs/grafana/latest/dashboards/)
Expand Down
6 changes: 3 additions & 3 deletions docs/networkpolicy-recommendation.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,15 @@ their applications according to Zero Trust principles.

## Prerequisite

Please follow [Getting Started](network-flow-visibility.md#getting-started) to
install Antrea Flow Aggregator and Theia.
Please follow the [Getting Started](getting-started.md) guide to install Antrea
Flow Aggregator and Theia.

## Perform NetworkPolicy Recommendation

Users can leverage Theia's NetworkPolicy Recommendation feature through `theia`
CLI. `theia` is the command-line tool which provides access to Theia network
flow visibility capabilities. To get more information about `theia`, please
refer to this [doc](theia.md).
refer to its [user guide](theia-cli.md).

There are 3 `theia` commands for the NetworkPolicy Recommendation feature:

Expand Down
File renamed without changes.