Update idle runners after reconciliation (#421)

* Update idle runners after reconciliation Adds the difference between expected and active plus idle * update docs * update changelog --------- Co-authored-by: Christopher Bartz <christopher.bartz@canonical.com>
canonical · Dec 17, 2024 · 318550e · 318550e
1 parent 92ef42f
commit 318550e
Show file tree

Hide file tree

Showing 5 changed files with 657 additions and 626 deletions.
diff --git a/docs/changelog.md b/docs/changelog.md
@@ -1,5 +1,9 @@
 # Changelog
 
+### 2024-12-13
+
+- Add the difference between expected and actual runners to the "Runners after reconciliation" dashboard panel.
+
 ### 2024-12-05
 
 - Bugfix to no longer stop the reconciliation when a runner's health check fails.

diff --git a/docs/reference/cos.md b/docs/reference/cos.md
@@ -13,7 +13,7 @@ The "GitHub Self-Hosted Runner Metrics" metrics dashboard presents the following
 - General: Displays general metrics about the charm and runners, such as:
   - Lifecycle counters: Tracks the frequency of Runner initialisation, start, stop, and crash events.
   - Available runners: A horizontal bar graph showing the number of runners available (and max expected) during the last reconciliation event. Note: This data is updated after each reconciliation event and is not real-time. 
-  - Runners after reconciliation: A time series graph showing the number of runners marked as active/idle and the number of expected runners during the last reconciliation event over time. Note: This data is updated after each reconciliation event and is not real-time.
+  - Runners after reconciliation: A time series graph showing the number of runners marked as active/idle, the number of expected runners, and the difference between expected and the former (unknown) during the last reconciliation event over time. Note: This data is updated after each reconciliation event and is not real-time.
   - Duration observations: Each data point aggregates the last hour and shows the 50th, 90th, 95th percentile and maximum durations for:
       - Runner installation
       - Runner idle duration

diff --git a/github-runner-manager/src-docs/openstack_cloud.health_checks.md b/github-runner-manager/src-docs/openstack_cloud.health_checks.md
@@ -13,7 +13,7 @@ Collection of functions related to health checks for a runner VM.
 
 ---
 
-<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/health_checks.py#L29"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/health_checks.py#L33"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ## <kbd>function</kbd> `check_runner`
 
@@ -43,7 +43,7 @@ This check applies to runners in any OpenStack state (ACTIVE, STOPPED, etc).
 
 ---
 
-<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/health_checks.py#L58"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/health_checks.py#L70"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ## <kbd>function</kbd> `check_active_runner`
 
@@ -67,6 +67,12 @@ Run a health check for a runner whose openstack instance is ACTIVE.
 
 
 
+**Raises:**
+
+ - <b>`OpenstackHealthCheckError`</b>:  If the health check could not be completed. 
+
+
+
 **Returns:**
  Whether the runner should be considered healthy. 
 

diff --git a/github-runner-manager/src-docs/openstack_cloud.openstack_runner_manager.md b/github-runner-manager/src-docs/openstack_cloud.openstack_runner_manager.md
@@ -14,11 +14,12 @@ Manager for self-hosted runner on OpenStack.
 - **MAX_METRICS_FILE_SIZE**
 - **RUNNER_STARTUP_PROCESS**
 - **OUTDATED_METRICS_STORAGE_IN_SECONDS**
+- **HEALTH_CHECK_ERROR_LOG_MSG**
 
 
 ---
 
-<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L82"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L85"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ## <kbd>class</kbd> `OpenStackServerConfig`
 Configuration for OpenStack server. 
@@ -49,7 +50,7 @@ __init__(image: str, flavor: str, network: str) → None
 
 ---
 
-<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L97"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L100"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ## <kbd>class</kbd> `OpenStackRunnerManagerConfig`
 Configuration for OpenStack runner manager. 
@@ -92,7 +93,7 @@ __init__(
 
 ---
 
-<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L133"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L138"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ## <kbd>class</kbd> `OpenStackRunnerManager`
 Manage self-hosted runner on OpenStack cloud. 
@@ -103,7 +104,7 @@ Manage self-hosted runner on OpenStack cloud.
 
  - <b>`name_prefix`</b>:  The name prefix of the runners created. 
 
-<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L140"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L145"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ### <kbd>method</kbd> `__init__`
 
@@ -135,7 +136,7 @@ The prefix of runner names.
 
 ---
 
-<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L350"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L362"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ### <kbd>method</kbd> `cleanup`
 
@@ -158,7 +159,7 @@ Cleanup runner and resource on the cloud.
 
 ---
 
-<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L179"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L184"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ### <kbd>method</kbd> `create_runner`
 
@@ -188,7 +189,7 @@ Create a self-hosted runner.
 
 ---
 
-<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L285"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L297"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ### <kbd>method</kbd> `delete_runner`
 
@@ -212,7 +213,7 @@ Delete self-hosted runners.
 
 ---
 
-<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L319"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L331"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ### <kbd>method</kbd> `flush_runners`
 
@@ -235,7 +236,7 @@ Remove idle and/or busy runners.
 
 ---
 
-<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L222"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L227"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ### <kbd>method</kbd> `get_runner`
 
@@ -258,7 +259,7 @@ Get a self-hosted runner by instance id.
 
 ---
 
-<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L251"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../../github-runner-manager/src/github_runner_manager/openstack_cloud/openstack_runner_manager.py#L261"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ### <kbd>method</kbd> `get_runners`