Added Receive + Prometheus Agent tutorial. (#4817)

* Added Receive + Prometheus Agent tutorial.

Also updated tutorial 3.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Apply suggestions from code review

Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>

* Added minor nits.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Apply suggestions from code review

Co-authored-by: Ian Billett <ibillett@redhat.com>

* Moved to Prometheus main.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* Reverted due to broken main.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

* nits.

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Co-authored-by: Ian Billett <ibillett@redhat.com>

tutorials/katacoda/thanos-pathway.json

@@ -1,6 +1,6 @@
 {
   "title": "Learn Thanos",
-  "description": "Introduction, Tips and Advanced Tutorials for Thanos: the CNCF, Global, Scalable System for Prometheus Metrics with cheap, Long-term Storage Capabilities. Course version: v0.3",
+  "description": "Introduction, Tips and Advanced Tutorials for Thanos: the CNCF, Global, Scalable System for Prometheus Metrics with cheap, Long-term Storage Capabilities. Course version: v0.4",
   "icon": "fa-thanos",
   "courses": [
     {
@@ -15,8 +15,13 @@
     },
     {
       "course_id": "3-receiver",
-      "title": "Intermediate: Ingesting metrics data from unreachable sources with Thanos Receive",
-      "description": "Learn how to ingest and query metrics data from unreachable sources with Thanos Receive."
+      "title": "Intermediate: Streaming metrics from remote source with Thanos Receive",
+      "description": "Learn how to ingest and then query metrics data from egress-only sources with Thanos Receive."
+    },
+    {
+      "course_id": "4-receiver-agent",
+      "title": "Bonus: Using Prometheus Agent for streaming metrics to Thanos Receive",
+      "description": "Learn how to use Prometheus in agent mode to efficiently stream metrics out of cluster."
     },
     {
       "course_id": "6-query-caching",

tutorials/katacoda/thanos/3-receiver/finish.md

@@ -1,10 +1,11 @@
 # Summary
 
 Congratulations! 🎉🎉🎉
-You completed this Thanos Receive tutorial. Let's summarize what we learned:
+You completed this `Thanos Receive` tutorial. Let's summarize what we learned:
 
-* Thanos Receive is a component that implements the `Prometheus Remote Write` protocol.
-* Prometheus can be configured to remote write its metric data in real-time to another server that implements the Remote Write protocol.
+* `Thanos Receive` is a component that implements the `Prometheus Remote Write` protocol.
+* Prometheus can be configured to remote-write its metric data in real-time to another server that implements the `Remote Write` protocol.
+* Thanos Receive allows us to ingest the data from sources which have limited accessibility, or that has no querying / storing capabilities and they have to forward data somewhere else (e.g Prometheus in Agent mode explained in next tutorial).
 
 See next courses for other tutorials about different deployment models and more advanced features of Thanos!
 

tutorials/katacoda/thanos/3-receiver/index.json

@@ -1,6 +1,6 @@
 {
-  "title": "Intermediate: Ingesting metrics data from unreachable sources with Thanos Receive",
-  "description": "Learn how to ingest and query metrics data from unreachable sources with Thanos Receive.",
+  "title": "Intermediate: Streaming metrics from remote source with Thanos Receive",
+  "description": "Learn how to ingest and then query metrics data from egress-only sources with Thanos Receive.",
   "difficulty": "Moderate",
   "details": {
     "steps": [

tutorials/katacoda/thanos/3-receiver/intro.md

@@ -1,17 +1,15 @@
-# Intermediate: Ingesting metrics data from unreachable sources with Thanos Receive
+# Intermediate: Streaming metrics from remote source with Thanos Receive
 
 The [Thanos](thanos.io) project defines a set of components that can be composed together into a highly available metric system with **unlimited storage capacity** that **seamlessly** integrates into your existing Prometheus deployments.
 
 In this course you get first-hand experience building and deploying this infrastructure yourself.
 
 In this tutorial, you will learn:
 
-* How to ingest metrics data from Prometheus instances that are unreachable from your infrastructure.
+* How to ingest metrics data from Prometheus instances without ingress traffic and need to store data locally for longer time.
 * How to setup a Thanos Querier to access this data.
 * How Thanos Receive is different from Thanos Sidecar, and when is the right time to use each of them.
 
-This will allow you to setup infrastucture
-
 > NOTE: This course uses docker containers with pre-built Thanos, Prometheus, and Minio Docker images available publicly.
 
 ### Prerequisites

tutorials/katacoda/thanos/3-receiver/step1.md

@@ -7,6 +7,7 @@ Let's imagine that you run a company called `Wayne Enterprises`. This company ru
 However, these sites are special. For security reasons, they do not expose public endpoints to the Prometheus instances running there, and so cannot be accessed directly from other parts of your infrastructure.
 
 As the person responsible for implementing monitoring these sites, you have two requirements to meet:
+
 1. Implement a global view of this data. `Wayne Enterprises` needs to know what is happening in all parts of the company - including secret ones!
 1. Global view must be queryable in near real-time. We can't afford any delay in monitoring these locations!
 
@@ -57,9 +58,10 @@ Verify that `prometheus-batcave` is running by navigating to the [Batcave Promet
   This lets us tell Prometheus to dynamically reload its configuration, which will be useful later in this tutorial.
 </details>
 
+
 ### Batcomputer
 
-Almost exactly the same configuration as above, execpt we run the Prometheus instance on port `9091`.
+Almost exactly the same configuration as above, except we run the Prometheus instance on port `9091`.
 
 <pre class="file" data-filename="prometheus-batcomputer.yaml" data-target="replace">
 global:

tutorials/katacoda/thanos/3-receiver/step2.md

@@ -27,11 +27,12 @@ Since we cannot access the `Thanos Sidecar` directly - we cannot query metrics d
 This means that the Global View would be at least 2 hours out of date, and does not satisfy requirement #2.
 </details>
 
+
 ## Thanos Receive
 
 Enter [Thanos Receive](https://thanos.io/tip/components/receive.md/).
 
-`Thanos Receive` is a component that implements the [Prometheus Remote Write API](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write). This means that it will accept metrics data that is sent to it by other Prometheus instances.
+`Thanos Receive` is a component that implements the [Prometheus Remote Write API](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write). This means that it will accept metrics data that is sent to it by other Prometheus instances (or any other process that implements Remote Write API).
 
 Prometheus can be configured to `Remote Write`. This means that Prometheus will send all of its metrics data to a remote endpoint as they are being ingested - useful for our requirements!
 
@@ -58,6 +59,8 @@ docker run -d --rm \
     --remote-write.address 127.0.0.1:10908
 ```{{execute}}
 
+This starts Thanos Receive that listens on `http://127.0.0.1:10908/api/v1/receive' endpoint for Remote Write and on `127.0.0.1:10907` for Thanos StoreAPI.
+
 Let's talk about some important parameters:
 * `--label` - `Thanos Receive` requires at least one label to be set. These are called 'external labels' and are used to uniquely identify this instance of `Thanos Receive`.
 * `--remote-write.address` - This is the address that `Thanos Receive` is listening on for Prometheus' remote write requests.
@@ -91,10 +94,11 @@ Now we are done right? Try querying for some data...
 ![alt text](./assets/receive-empty-query-result.png)
 
 <details>
- <summary>Uh-oh! Why are we seeing 'Empty Query Result' responses?</summary>
+ <summary>Uh-oh! Do you know why are we seeing 'Empty Query Result' responses?</summary>
 
 We have correctly configured `Thanos Receive` & `Thanos Query`, but we have not yet configured Prometheus to write to remote write its data to the right place.
 
 </details>
 
+
 Hit continue and we will fix this setup!

tutorials/katacoda/thanos/3-receiver/step3.md

@@ -1,6 +1,6 @@
 # Configure Prometheus Remote Write
 
-Our problem in the last step was that we have have not yet configured Prometheus to `remote_write` to our `Thanos Receive` instance.
+Our problem in the last step was that we have not yet configured Prometheus to `remote_write` to our `Thanos Receive` instance.
 
 We need to tell `prometheus-batcave` & `prometheus-batcomputer` where to write their data to.
 

tutorials/katacoda/thanos/4-receiver-agent/courseBase.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+docker pull quay.io/bwplotka/prometheus:agent1
+docker pull quay.io/thanos/thanos:v0.21.0
+
+mkdir /root/editor

tutorials/katacoda/thanos/4-receiver-agent/finish.md

@@ -0,0 +1,20 @@
+# Summary
+
+Congratulations! 🎉🎉🎉
+You completed this Prometheus Agent tutorial. Let's summarize what we learned:
+
+* Thanos Receive is a component that implements the `Prometheus Remote Write` protocol.
+* Prometheus Agent can be deployed to remote write its metric data in real-time to another server that implements the Remote Write protocol.
+* Prometheus Agent uses a _fraction_ of the resources as normal Prometheus, by not enabling features like: local metrics storage and local query APIs.
+
+See next courses for other tutorials about different deployment models and more advanced features of Thanos!
+
+## Further Reading
+
+TBD
+
+### Feedback
+
+Do you see any bug, typo in the tutorial or you have some feedback for us?
+
+let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io

tutorials/katacoda/thanos/4-receiver-agent/index.json

@@ -0,0 +1,51 @@
+{
+  "title": "Bonus: Using Prometheus Agent for streaming metrics to Thanos Receive",
+  "description": "Learn how to use Prometheus in agent mode to efficiently stream metrics out of cluster.",
+  "difficulty": "Moderate",
+  "details": {
+    "steps": [
+      {
+        "title": "Problem Statement & Setup",
+        "text": "step1.md",
+        "verify": "step1-verify.sh"
+      },
+      {
+        "title": "Setup Prometheus Agents",
+        "text": "step2.md",
+        "verify": "step2-verify.sh"
+      },
+      {
+        "title": "Verify Setup",
+        "text": "step3.md",
+        "verify": "step3-verify.sh"
+      }
+    ],
+    "intro": {
+      "text": "intro.md",
+      "courseData": "courseBase.sh",
+      "credits": "https://thanos.io"
+    },
+    "finish": {
+      "text": "finish.md",
+      "credits": "test"
+    }
+  },
+  "files": [
+    "prom-agent-batmobile.yaml",
+    "prom-agent-batcopter.yaml"
+  ],
+  "environment": {
+    "uilayout": "editor-terminal",
+    "uisettings": "yaml",
+    "uieditorpath": "/root/editor",
+    "showdashboard": true,
+    "dashboards": [
+      {"name": "Prometheus Agent Batmobile", "port": 9090},
+      {"name": "Prometheus Agent Batcopter", "port": 9091},
+      {"name": "Thanos Query", "port": 39090}
+  ]
+  },
+  "backend": {
+    "imageid": "docker-direct"
+  }
+}

tutorials/katacoda/thanos/4-receiver-agent/intro.md

@@ -0,0 +1,30 @@
+# Intermediate: (Bonus) Using Prometheus Agent for pure forward-only metric streaming with Thanos Receive
+
+The [Thanos](thanos.io) project defines a set of components that can be composed together into a highly available metric system with **unlimited storage capacity** that **seamlessly** integrates into your existing Prometheus deployments.
+
+But [Prometheus](https://prometheus.io/) project is far from slowing down the development. Together with the community, it constantly evolves to bring value to different network and cluster topologies that we see. Thanos brings distributed and cloud storage and querying to the table, built on the core Prometheus code and design. It allowed Prometheus to focus on critical collection and single cluster monitoring functionalities.
+
+As we learned in the previous tutorial, certain situations require us to collect (pull) data from applications and stream them out of the cluster as soon as possible. `Thanos Receive` allows doing that by ingesting metrics using the Remote Write protocol that the sender can implement. Typically we recommended using Prometheus with short retention and blocked read and query API as a "lightweight" sender.
+
+In November 2021, however, we, the Prometheus community, introduced a brand new Prometheus mode called "Agent mode". The implementation itself was already battle tested on https://github.com/grafana/agent, where it was available and authored by [Robert Fratto](https://github.com/rfratto) since 2020.
+
+The agent mode is optimized for efficient metric scraping and forwarding (i.e. immediate metric removal once it's securely delivered to a remote location). Since this is incredibly helpful for the Thanos community, we wanted to give you first-hand experience deploying Prometheus Agent together with Thanos Receive in this course.
+
+In this tutorial, you will learn:
+
+* How to reduce Prometheus based client-side metric collection to a minimum, using the new Prometheus "Agent mode" with `Thanos Receiver`, explained in the previous tutorial.
+
+> NOTE: This course uses docker containers with pre-built Thanos, Prometheus, and Minio Docker images available publicly.
+
+### Prerequisites
+
+Please complete tutorial #3 first: [Intermediate: Streaming metrics from remote source with Thanos Receive](https://www.katacoda.com/thanos/courses/thanos/3-receiver) 🤗
+
+### Feedback
+
+Do you see any bug, typo in the tutorial, or do you have some feedback for us?
+Let us know on https://github.com/thanos-io/thanos or #thanos slack channel linked on https://thanos.io
+
+### Contributed by:
+
+* Bartek Plotka [@bwplotka](http://bwplotka.dev)

tutorials/katacoda/thanos/4-receiver-agent/step1-verify.sh

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+
+# receive
+curl -s 127.0.0.1:10909/metrics >/dev/null || exit 1
+# query
+curl -s 127.0.0.1:39090/metrics >/dev/null || exit 1
+
+echo '"done"'

tutorials/katacoda/thanos/4-receiver-agent/step1.md

@@ -0,0 +1,56 @@
+## Problem Statement
+
+Let's get back to our example from [Tutorial 3](https://www.katacoda.com/thanos/courses/thanos/3-receiver). Imagine you run a company called `Wayne Enterprises`. In tutorial 3, we established monitoring of two special clusters: `Batcave` & `Batcomputer`.  These are special because they do not expose public endpoints to the Prometheus instances running there for security reasons, so we used the Remote Write protocol to stream all metrics to Thanos Receive in our centralized space.
+
+Let's imagine we want to expand our `Wayne Enterprises` by adding metrics collection to applications running on smaller devices inside more mobile Batman tools: `Batmobile` and `Batcopter`.
+
+Each of these vehicles has a smaller computer that runs applications from which we want to scrape Prometheus-like metrics using OpenMetrics format.
+
+As the person responsible for implementing monitoring in these environments, you have three requirements to meet:
+
+1. Implement a global view of this data. `Wayne Enterprises` needs to know what is happening in all company parts - including secret ones!
+2. `Batmobile` and `Batcopter` can be out of network for some duration of time. You don't want to lose precious data.
+3. `Batmobile` and `Batcopter` environments are very **resource constrained**, so you want an efficient solution that avoids extra computations and storage.
+
+Firstly, let us set up Thanos as we explained in the previous tutorial.
+
+## Setup Central Platform
+
+As you might remember from the previous tutorial, in the simplest form for streaming use cases, we need to deploy Thanos Receive and Thanos Querier.
+
+Let's run `Thanos Receive`:
+
+```
+docker run -d --rm \
+    -v $(pwd)/receive-data:/receive/data \
+    --net=host \
+    --name receive \
+    quay.io/thanos/thanos:v0.21.0 \
+    receive \
+    --tsdb.path "/receive/data" \
+    --grpc-address 127.0.0.1:10907 \
+    --http-address 127.0.0.1:10909 \
+    --label "receive_replica=\"0\"" \
+    --label "receive_cluster=\"wayne-enterprises\"" \
+    --remote-write.address 127.0.0.1:10908
+```{{execute}}
+
+This starts Thanos Receive that listens on `http://127.0.0.1:10908/api/v1/receive` endpoint for Remote Write and on `127.0.0.1:10907` for Thanos StoreAPI.
+
+Next, let us run a `Thanos Query` instance connected to Thanos Receive:
+
+```
+docker run -d --rm \
+--net=host \
+--name query \
+quay.io/thanos/thanos:v0.21.0 \
+query \
+--http-address "0.0.0.0:39090" \
+--store "127.0.0.1:10907"
+```{{execute}}
+
+Verify that `Thanos Query` is working and configured correctly by looking at the 'stores' tab [here](https://[[HOST_SUBDOMAIN]]-39090-[[KATACODA_HOST]].environments.katacoda.com/stores) (try refreshing the Thanos UI if it does not show up straight away).
+
+We should see Receive store on this page and, as expected, no metric data since we did not connect any Remote Write sender yet.
+
+With our "central" platform that is ready to ingest metrics, we can now start to architect our collection pipeline that will stream all metrics to our `Wayne Enterprises`.

tutorials/katacoda/thanos/4-receiver-agent/step2-verify.sh

@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+
+# prometheus-batcave
+curl -s 127.0.0.1:9090/metrics >/dev/null || exit 1
+# prometheus-batcomputer
+curl -s 127.0.0.1:9091/metrics >/dev/null || exit 1
+# receive
+curl -s 127.0.0.1:10909/metrics >/dev/null || exit 1
+# query
+curl -s 127.0.0.1:39090/metrics >/dev/null || exit 1
+
+echo '"done"'