Skip to content

Commit

Permalink
use batch gh-workflow-stats-action with separate table (#9722)
Browse files Browse the repository at this point in the history
We found that exporting GH Workflow Runs in batch is more efficient due
to
- better utilisation of Github API
- and gh runners usage is rounded to minutes, so even when ad-hoc export
is done in 5-10 seconds, we billed for one minute usage

So now we introduce batch exporting, with version v0.2.x of github
workflow stats exporter.
How it's expected to work now:
- every 15 minutes we query for the workflow runs, created in last 2
hours
- to avoid missing workflows that ran for more than 2 hours, every night
(00:25) we will query workflows created in past 24 hours and export them
as well
- should we have query for even longer periods?
- lets see how it works with current schedule
- for longer periods like for days or weeks, it may require to adjust
logic and concurrency of querying data, so lets for now use simpler
version
  • Loading branch information
fedordikarev authored Nov 11, 2024
1 parent 5a138d0 commit fde16f8
Showing 1 changed file with 29 additions and 0 deletions.
29 changes: 29 additions & 0 deletions .github/workflows/report-workflow-stats-batch.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
name: Report Workflow Stats Batch

on:
schedule:
- cron: '*/15 * * * *'
- cron: '25 0 * * *'

jobs:
gh-workflow-stats-batch:
name: GitHub Workflow Stats Batch
runs-on: ubuntu-22.04
permissions:
actions: read
steps:
- name: Export Workflow Run for the past 2 hours
uses: neondatabase/gh-workflow-stats-action@v0.2.1
with:
db_uri: ${{ secrets.GH_REPORT_STATS_DB_RW_CONNSTR }}
db_table: "gh_workflow_stats_batch_neon"
gh_token: ${{ secrets.GITHUB_TOKEN }}
duration: '2h'
- name: Export Workflow Run for the past 24 hours
if: github.event.schedule == '25 0 * * *'
uses: neondatabase/gh-workflow-stats-action@v0.2.1
with:
db_uri: ${{ secrets.GH_REPORT_STATS_DB_RW_CONNSTR }}
db_table: "gh_workflow_stats_batch_neon"
gh_token: ${{ secrets.GITHUB_TOKEN }}
duration: '24h'

1 comment on commit fde16f8

@github-actions
Copy link

@github-actions github-actions bot commented on fde16f8 Nov 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5463 tests run: 5232 passed, 1 failed, 230 skipped (full report)


Failures on Postgres 16

  • test_compaction_l0_memory[github-actions-selfhosted]: release-x86-64
# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_compaction_l0_memory[release-pg16-github-actions-selfhosted]"
Flaky tests (3)

Postgres 17

Postgres 14

Code coverage* (full report)

  • functions: 31.7% (7881 of 24824 functions)
  • lines: 49.4% (62405 of 126225 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
fde16f8 at 2024-11-11T23:03:35.340Z :recycle:

Please sign in to comment.