From 740c31938d9bc56d80d526e5f160fded685dac2a Mon Sep 17 00:00:00 2001 From: Jianjun Shen Date: Fri, 10 Jun 2022 21:32:11 -0400 Subject: [PATCH] Revise README and user documents Add Theia overview, getting-started, contributting, community, and license information to README.md. Revise the Theia introduction, and add more information about Theia features and usage in network-flow-visibility.md and the Getting Started section. Rename theia.md to theia-cli.md. Signed-off-by: Jianjun Shen --- README.md | 51 ++++++++++++--- docs/getting-started.md | 96 ++++++++++++++++++++++++++++ docs/network-flow-visibility.md | 88 +++++++------------------ docs/networkpolicy-recommendation.md | 2 +- docs/{theia.md => theia-cli.md} | 0 5 files changed, 164 insertions(+), 73 deletions(-) create mode 100644 docs/getting-started.md rename docs/{theia.md => theia-cli.md} (100%) diff --git a/README.md b/README.md index 4122e92a4..9badbb13a 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,45 @@ # Theia -Theia contains Network Flow Visibility functions which are exacted from the -[Antrea main repo](https://github.com/antrea-io/antrea). While flow exporter and -flow aggregator are kept in the Antrea repo, other flow visibility modules, such -as ClickHouse and Grafana related ones, will be moved to this new repo. - -During the code migration period, we will still keep related functions available -and stable in the Antrea repo, and the functions in this new repo won't be ready -until we announce the migration is completed. +Theia is a network observability and analytics platform for Kubernetes. It is +built on top of [Antrea](https://github.com/antrea-io/antrea), and consumes +[network flows exported by Antrea](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md) +to provide fine-grained visibility into the network communications in a +Kubernetes cluster. + +## Getting Started + +Getting started with Theia is simple. You can follow the [Getting Started](docs/getting-started.md) +guide to install Theia and start rocking! + +Theia supports network flow visualization and monitoring with Grafana. Check the +[Grafana Flow Collector](docs/network-flow-visibility.md) document to learn +more. + +Based on the collected network flow information, Theia can recommend appropriate +NetworkPolicy configuration to secure Kubernetes network and applications. +Please refer to the [NetworkPolicy Recommendation](docs/networkpolicy-recommendation.md) +user guide to learn more. + +## Contributing + +The Antrea community welcomes new contributors. We are waiting for your PRs! + +* Before contributing, please get familiar with our +[Code of Conduct](CODE_OF_CONDUCT.md). +* Check out the Antrea [Contributor Guide](CONTRIBUTING.md) for information +about setting up your development environment and our contribution workflow. +* Learn about Antrea's [Architecture and Design](https://github.com/antrea-io/antrea/blob/main/docs/design/architecture.md). +Your feedback is more than welcome! +* Check out [Open Issues](https://github.com/antrea-io/theia/issues). +* Join the Antrea [community](#community) and ask us any question you may have. + +## Community + +Please refer to the [Antrea community](https://github.com/antrea-io/antrea/blob/main/README.md#community) +information. + +## License + +Antrea is licensed under the [Apache License, version 2.0](LICENSE) + +[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fantrea-io%2Fantrea.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fantrea-io%2Fantrea?ref=badge_large) diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 000000000..a3eb110f6 --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,96 @@ +# Getting Started with Theia + +## Table of Contents + + +- [Overview](#overview) +- [Prerequisites](#prerequisites) +- [Theia Installation](#theia-installation) +- [Features](#features) + - [Network Flow Visualization and Monitoring](#network-flow-visualization-and-monitoring) + - [NetworkPolicy Recommendation](#networkpolicy-recommendation) +- [Additional Information](#additional-information) + + +## Overview + +Theia is a network observability and analytics platform for Kubernetes, bulit +on top of [Antrea](https://github.com/antrea-io/antrea). Theia consumes [network +flows exported by Antrea](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md) +to provide fine-grained visibility into the network communications in a +Kubernetes cluster. + +Theia supports network flow visualization and monitoring with Grafana, and can +recommend appropriate NetworkPolicy configuration to secure Kubernetes network +and applications. This guide describes how to install and get started with +Theia. + +## Prerequisites + +To install Theia, ensure the Flow Exporter feature is enabled in Antrea deployment +manifest, in the following way: + +```yaml + antrea-agent.conf: | + ... + featureGates: + ... + FlowExporter: true +``` + +## Theia Installation + +To enable both [Grafana Flow Collector](network-flow-visibility.md) and +[NetworkPolicy Recommendation](networkpolicy-recommendation.md), please install +Flow Aggregator and Theia by runnning the following commands: + +```bash +git clone https://github.com/antrea-io/theia.git +helm install flow-aggregator theia/build/charts/flow-aggregator --set clickHouse.enable=true,recordContents.podLabels=true -n flow-aggregator --create-namespace +helm install theia theia/build/charts/theia --set sparkOperator.enable=true -n flow-visibility --create-namespace +``` + +To enable only Grafana Flow Collector, please install Flow Aggregator and Theia +by runnning the following commands: + +```bash +git clone https://github.com/antrea-io/theia.git +helm install flow-aggregator theia/build/charts/flow-aggregator --set clickHouse.enable=true -n flow-aggregator --create-namespace +helm install theia theia/build/charts/theia -n flow-visibility --create-namespace +``` + +## Features + +### Network Flow Visualization and Monitoring + +Theia uses Grafana to visualize network flows in the Kubernetes cluster. After +the installation, you can run the following commands to get the Grafana Service +address: + +```bash +NODE_NAME=$(kubectl get pod -l app=grafana -n flow-visibility -o jsonpath='{.items[0].spec.nodeName}') +NODE_IP=$(kubectl get nodes ${NODE_NAME} -o jsonpath='{.status.addresses[0].address}') +GRAFANA_NODEPORT=$(kubectl get svc grafana -n flow-visibility -o jsonpath='{.spec.ports[*].nodePort}') +echo "=== Grafana Service is listening on ${NODE_IP}:${GRAFANA_NODEPORT} ===" +``` + +You now access Grafana in your browser at: `http://[NodeIP]:[NodePort]`, and log +in with username: `admin` and password: `admin`. Navigate to the [Theia +dashboards](network-flow-visibility.md#grafana-dashboards) to view the network +flows in the cluster. + +### NetworkPolicy Recommendation + +Please follow the instructions in the [NetworkPolicy Recommendation](networkpolicy-recommendation.md) +user guide. + +## Additional Information + +Refer to Antrea documentation to learn more about +[Flow Exporter](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md#flow-exporter), +[Flow Aggregator](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md#flow-aggregator) +and their advanced configurations. + +For more information about Grafana Flow Collector installation and +customization, please refer to Grafana Flow Collector [Deployment Steps](network-flow-visibility.md#deployment-steps), +and [Configuration](network-flow-visibility.md#configuration). diff --git a/docs/network-flow-visibility.md b/docs/network-flow-visibility.md index 9414f8eb5..d2e2f5b83 100644 --- a/docs/network-flow-visibility.md +++ b/docs/network-flow-visibility.md @@ -1,13 +1,9 @@ -# Theia: Network Flow Visibility for Antrea +# Network Flow Visibility with Theia ## Table of Contents - [Overview](#overview) -- [Getting Started](#getting-started) - - [Prerequisites](#prerequisites) - - [Theia Installation](#theia-installation) - - [Additional information](#additional-information) - [Grafana Flow Collector](#grafana-flow-collector) - [Purpose](#purpose) - [About Grafana and ClickHouse](#about-grafana-and-clickhouse) @@ -23,6 +19,7 @@ - [Service Customization](#service-customization-1) - [Performance Configuration](#performance-configuration) - [Persistent Volumes](#persistent-volumes) +- [Grafana Dashboards](#grafana-dashboards) - [Pre-built Dashboards](#pre-built-dashboards) - [Flow Records Dashboard](#flow-records-dashboard) - [Pod-to-Pod Flows Dashboard](#pod-to-pod-flows-dashboard) @@ -30,65 +27,17 @@ - [Pod-to-Service Flows Dashboard](#pod-to-service-flows-dashboard) - [Node-to-Node Flows Dashboard](#node-to-node-flows-dashboard) - [Network-Policy Flows Dashboard](#network-policy-flows-dashboard) - - [Dashboards Customization](#dashboards-customization) + - [Dashboard Customization](#dashboard-customization) ## Overview -[Antrea](https://github.com/antrea-io/antrea/blob/main/docs/design/architecture.md) -is a Kubernetes network plugin that provides network connectivity and security -features for Pod workloads. Considering the scale and dynamism of Kubernetes -workloads in a cluster, Network Flow Visibility helps in the management and -configuration of Kubernetes resources such as Network Policy, Services, Pods -etc., and thereby provides opportunities to enhance the performance and security -aspects of Pod workloads. - -For visualizing the network flows, Antrea monitors the flows in Linux conntrack -module. These flows are converted to flow records, and then flow records are post-processed -before they are sent to the configured external flow collector. - -## Getting Started - -### Prerequisites - -To install Theia, ensure the Flow Exporter feature is enabled in Antrea deployment -manifest, in the following way: - -```yaml - antrea-agent.conf: | - ... - featureGates: - ... - FlowExporter: true -``` - -### Theia Installation - -To enable both Grafana Flow Collector and -[NetworkPolicy Recommendation](networkpolicy-recommendation.md), please install -Theia and Flow Aggregator by runnning the following commands: - -```bash -git clone https://github.com/antrea-io/theia.git -helm install flow-aggregator theia/build/charts/flow-aggregator --set clickHouse.enable=true,recordContents.podLabels=true -n flow-aggregator --create-namespace -helm install theia theia/build/charts/theia --set sparkOperator.enable=true -n flow-visibility --create-namespace -``` - -To enable only Grafana Flow Collector, please install Theia and Flow Aggregator -by runnning the following commands: - -```bash -git clone https://github.com/antrea-io/theia.git -helm install flow-aggregator theia/build/charts/flow-aggregator --set clickHouse.enable=true -n flow-aggregator --create-namespace -helm install theia theia/build/charts/theia -n flow-visibility --create-namespace -``` - -### Additional information - -Refer to Antrea documentation to learn more about -[Flow Exporter](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md#flow-exporter), -[Flow Aggregator](https://github.com/antrea-io/antrea/blob/main/docs/network-flow-visibility.md#flow-aggregator) -and their advanced configurations. +Theia is a network observability and analytics platform for Kubernetes, bulit +on top of [Antrea](https://github.com/antrea-io/antrea). Theia consumes network +flows exported by Antrea to provide fine-grained visibility into the network +communications in a Kubernetes cluster. It supports network flow visualization +and monitoring with Grafana. This document describes the Grafana Flow Collector +and network flow visualization functionality of Theia. ## Grafana Flow Collector @@ -112,6 +61,11 @@ ClickHouse as the data storage, and use Grafana as the data visualization and mo ### Deployment Steps +This section talks about detailed steps to deploy the Grafana Flow Collector +using Helm charts or YAML manifests. For quick steps of installing Theia +including the Grafana Flow Collector, please refer to the [Getting Started](getting-started.md) +guide. + We support deploying the Grafana Flow Collector with Helm. Here is the [Helm chart](../build/charts/theia/) for the Grafana Flow Collector. Please follow the instructions from the Helm chart [README](../build/charts/theia/README.md) @@ -156,7 +110,7 @@ kubectl apply -f https://raw.githubusercontent.com/antrea-io/theia/main/build/ya Run the following command to check if ClickHouse and Grafana are deployed properly: ```bash -kubectl get all -n flow-visibility +kubectl get all -n flow-visibility ``` The expected results will be like: @@ -217,6 +171,10 @@ kubectl delete -f https://raw.githubusercontent.com/antrea-io/theia/main/build/c ### Configuration +So far, we went through Grafana Flow Collector deployment with the default +configuration. The Grafana Flow Collector also exposes a few configuration +parameters for you to customize the deployment. Read this section to learn more. + #### With Helm If you install the Grafana Flow Collector using the Helm command, please refer @@ -594,9 +552,11 @@ should be set to your storage size. name: clickhouse-storage-volume ``` +## Grafana Dashboards + ### Pre-built Dashboards -The following dashboards are pre-built and are recommended for Antrea flow +The following dashboards are pre-built and are recommended for Theia flow visualization. They can be found in the Home page of Grafana, by clicking the Magnifier button on the left menu bar. Grafana Search Dashboards Guide @@ -620,7 +580,7 @@ and resources to render them. If you want to stop filtering traffic by Namespace, or edit the panel limit, you will need to edit the ClickHouse SQL query for each panel. Please follow -the [dashboards customization](#dashboards-customization) section for more +the [Dashboard Customization](#dashboard-customization) section for more information. As a special case, to edit the panel limit for pie charts, instead of editing the query, please follow the [doc](https://grafana.com/docs/grafana/latest/visualizations/pie-chart-panel/#limit) @@ -728,7 +688,7 @@ Mouse out or click on the background will bring all the traffic back. Network-Policy Flows Dashboard -### Dashboards Customization +### Dashboard Customization If you would like to make any changes to any of the pre-built dashboards, or build a new dashboard, please follow this [doc](https://grafana.com/docs/grafana/latest/dashboards/) diff --git a/docs/networkpolicy-recommendation.md b/docs/networkpolicy-recommendation.md index f154ac830..aa613735a 100644 --- a/docs/networkpolicy-recommendation.md +++ b/docs/networkpolicy-recommendation.md @@ -34,7 +34,7 @@ install Antrea Flow Aggregator and Theia. Users can leverage Theia's NetworkPolicy Recommendation feature through `theia` CLI. `theia` is the command-line tool which provides access to Theia network flow visibility capabilities. To get more information about `theia`, please -refer to this [doc](theia.md). +refer to its [user guide](theia-cli.md). There are 3 `theia` commands for the NetworkPolicy Recommendation feature: diff --git a/docs/theia.md b/docs/theia-cli.md similarity index 100% rename from docs/theia.md rename to docs/theia-cli.md