[feature] dashboard resyncPeriod to not change version

[NissesSenap]

Describe the feature
Currently every time a dashboard is reconciled, the version gets updated.
I don't want a new version should be created for each resync.

Version
v5.0.2

To Reproduce

Apply the following yaml

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  labels:
    dashboards: "grafana"
spec:
  config:
    log:
      mode: "console"
    auth:
      disable_login_form: "false"
    security:
      admin_user: root
      admin_password: secret

Dashboard

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: grafanadashboard-sample
  namespace: grafana
spec:
  resyncPeriod: 30s
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  json: >
    {
      "annotations": {
        "list": [
          {
            "builtIn": 1,
            "datasource": {
              "type": "grafana",
              "uid": "-- Grafana --"
            },
            "enable": true,
            "hide": true,
            "iconColor": "rgba(0, 211, 255, 1)",
            "name": "Annotations & Alerts",
            "type": "dashboard"
          }
        ]
      },
      "editable": true,
      "fiscalYearStartMonth": 0,
      "graphTooltip": 1,
      "id": 47,
      "links": [],
      "liveNow": false,
      "panels": [],
      "refresh": "5s",
      "schemaVersion": 38,
      "style": "dark",
      "tags": [],
      "templating": {
        "list": []
      },
      "time": {
        "from": "now-6h",
        "to": "now"
      },
      "timepicker": {
        "refresh_intervals": [],
        "time_options": []
      },
      "timezone": "browser",
      "title": "edvin Dashboard",
      "uid": "6a3fe7e9-cff2-4391-b0a0-83e10ea6959b",
      "version": 3,
      "weekStart": ""
    }

wait for 30 seconds and see the bump of the versions

Expected behavior
I expect the dashboard to not get a new version when a normal resyncPeriod happens.
It should only change if a change to the dashboard have been done.

Screenshots

Shows all the version that I currently have

Here we can see that it's only version that is diffing

[NissesSenap]

I should have known this but I just didn't remember it. But this is the expected behavior, we could remake it but it woulden't be a small change.
If you think this is a good idea please up vote it.

In the mean time I will lower how often my resyncPeriod runs.

[pb82]

@NissesSenap I think the default versions to keep is 20, so it won't go beyond that.

[owenhaynes]

@NissesSenap This is also causing a increase in diskspace usage since the move to v5 of the operator usage has gone up from 2GB and has risen to 15GB. We currently have over 4000+ versions for each of our dashboards.

We disabled the resync for now by setting it to a year as we have our Grafana in readonly mode

[pb82]

@owenhaynes wow, that's a lot. I thought Grafana capped the maximum number of versions at 20?
If that's not the case, we need to reconsider limiting that @NissesSenap @weisdd

[owenhaynes]

could be bug in Grafana as the UI is only showing 20, need to inspect the db some more as has increased in db size over the past month

[NissesSenap]

At least set a default value of max 20 versions by default in the resources so we don't create these kinds of sizes in db.
Should be a relatively small change.

[pb82]

I can take that one.

[pb82]

PR here: #1168
@owenhaynes have you had a chance to inspect the database?

[owenhaynes]

@pb82 yeah the consistent updates to the dashboard was causing the WAL to become big

[pb82]

thanks @owenhaynes , I'll check if that setting has an effect on the size of the database. It should be the same behaviour with the default sqlite database.

[7840vz]

This one actually blocking me from switching from operator 4.x where I don't have such thing.

What is really annoying is when the dashboards get updated and page is reloaded while users are working with them.

[NissesSenap]

@7840vz you can always set a reload value that is infinitely long, so it never is reapplied. I changed my value to be run once a day. This way I still know that my dashboards are up to date.
Remember to do the same for your datasources as well. Else there will still be a reload thing due to how grafana works.

[7840vz]

@NissesSenap , thanks,

yes, I also tried to set value to once a day. But in that case users have to wait up to 24h for dashboards to appear after Grafana restart/first start if Grafana is stateless. And I run multiple stateless copies of Grafana with operator....

[pb82]

The versions_to_keep setting in Grafana doesn't seem to have any effect. I have set it to 20, but Grafana happily creates more versions. I'll raise a bug in Grafana.

[pb82]

grafana/grafana#72256

[gil-brown-argus]

Is there any workaround for this issue?
In my case the version gets updated all the time (for all dashboards that are imported via the CRD) and i don't want users to wait for 24 hours for a change to take affect.

Our team suffer from when the dashboards get updated and page is reloaded while users are working with them as well.

[pb82]

@gil-brown-argus the idea is that the Grafana cleanup job removes the outdated dashboard versions from time to time. However, it's not clear to me how often this job runs, have you seen it cleaning up old dashboards after some time (maybe a few hours or a day)?

It looks like we will have to work around this and that means only submitting dashboard updates if there is a change. So before an update, we need to retrieve the dashboard and compare.

[gil-brown-argus]

Hi @pb82 , thanks for your quick response !
It seems like the job is cleaning up "old" dashboard old versions (currently i have only 15 versions of the dashboards).
I do wait for the dashboard updates only in case of a change.

[pb82]

@gil-brown-argus Do you know how often the job is running? I couldn't find anything in the Grafana codebase. If it was possible to adjust that job to run more often, that would probably also work.

[gil-brown-argus]

I have no idea.

[NissesSenap]

See #1213 for a potential fix

[gil-brown-argus]

thanks, I will watch it.

[NissesSenap]

This should now be solved. To improve this further, we have also created #1221
I will close this issue.