Unexpected keyword argument 'datasource'

[apepojken]

When I try to explore datasources it always fail with a TypeError. No idea why this happens.

I have tried versions 0.15.0 - 0.17.0

This is what happens:

➜ ~ export GRAFANA_URL=https://grafana.example.com                          
➜ ~ export GRAFANA_TOKEN=*************     
➜ ~ grafana-wtf explore datasources --format=yaml --cache-ttl=0 --drop-cache
2023-11-14 12:57:30,811 [grafana_wtf.commands                ] INFO   : Using Grafana at https://grafana.example.com
2023-11-14 12:57:30,814 [grafana_wtf.core                    ] INFO   : Response cache will expire immediately (expire_after=0)
2023-11-14 12:57:30,815 [grafana_wtf.core                    ] INFO   : Response cache database location is /home/user/.cache/grafana-wtf.sqlite
2023-11-14 12:57:30,815 [grafana_wtf.core                    ] INFO   : Dropping response cache
2023-11-14 12:57:30,815 [grafana_wtf.core                    ] INFO   : Clearing cache
2023-11-14 12:57:30,815 [requests_cache.backends.base        ] INFO   : Clearing all items from the cache
2023-11-14 12:57:30,933 [grafana_wtf.core                    ] INFO   : Scanning dashboards
2023-11-14 12:57:31,421 [grafana_wtf.core                    ] INFO   : Found 172 dashboard(s)
0it [00:00, ?it/s]
2023-11-14 12:57:31,429 [grafana_wtf.core                    ] INFO   : Fetching dashboards in parallel with 5 concurrent requests     | 0/172 [00:00<?, ?it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 172/172 [00:02<00:00, 69.62it/s]
2023-11-14 12:57:34,122 [grafana_wtf.core                    ] INFO   : Scanning datasources██████████████████████████████▉  | 169/172 [00:02<00:00, 75.00it/s]
2023-11-14 12:57:34,177 [grafana_wtf.core                    ] INFO   : Found 35 data source(s)
Traceback (most recent call last):
  File "/usr/local/bin/grafana-wtf", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/grafana_wtf/commands.py", line 309, in run
    results = engine.explore_datasources()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/grafana_wtf/core.py", line 420, in explore_datasources
    ix = Indexer(engine=self)
         ^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/grafana_wtf/core.py", line 560, in __init__
    self.index()
  File "/home/user/.local/lib/python3.11/site-packages/grafana_wtf/core.py", line 564, in index
    self.index_dashboards()
  File "/home/user/.local/lib/python3.11/site-packages/grafana_wtf/core.py", line 627, in index_dashboards
    item = DatasourceItem.from_payload(item)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/grafana_wtf/model.py", line 171, in from_payload
    return cls(**payload)
           ^^^^^^^^^^^^^^
TypeError: DatasourceItem.__init__() got an unexpected keyword argument 'datasource'

So far everything else seem to work, and grafana-wtf has been of great help to me. Would really appreciate if you could help me with this.

Thanks!

[amotl]

Hi there.

Thank you for reporting this flaw. Let's investigate it. May I ask you about the Grafana version you are using?

With kind regards,
Andreas.

[apepojken]

Hey!
This is the Grafana version:

{
  "commit": "cfcea75916",
  "database": "ok",
  "version": "9.5.2"
}

[amotl]

Hi again,

thank you for providing the Grafana version. Apologies that we haven't been able to look into this, yet.

With kind regards,
Andreas.

[amotl]

Dear @apepojken,

there is a patch now.

• Accept datasource attribute on DatasourceItem, but ignore it #131

However, we are not sure if that solves the problem. Is there any chance you would be able to convey corresponding dashboard JSON representations to us, so we may be able to inspect them, and reflect the data structure through code, to improve where are apparently still rough edges right now?

That would tremendously support us on the debugging process.

With kind regards,
Andreas.

[amotl]

Hi again,

we just invoked your command on play.grafana.org, without the patch from GH-131, and the outcome is that it did not cause any exception.

export GRAFANA_URL=https://play.grafana.org
grafana-wtf explore datasources --format=yaml

So, it seems it might be really an edge case on your end. However, everything is possible, as this sample sure enough is not exhaustive at all. Also, play.grafana.org is running on Grafana version 11.1.0-68793 already.

In this spirit, it would be really good to get some data samples (dashboard JSON dumps) from your installation, when possible, in order to be able to run grafana-wtf on the real data which causes issues for you. Otherwise, we are not sure about including the patch from GH-131 into the upcoming release.

With kind regards,
Andreas.

[apepojken]

Hey!
The GH-131 patch actually solved my problem. Now I get:

/home/user/.local/lib/python3.12/site-packages/grafana_wtf/model.py:181: UserWarning: The `datasource` attribute is ignored for the time being.
See also: https://github.com/panodata/grafana-wtf/issues/110
Please report back this occurrence to the grafana-wtf issue tracker,
so the maintainers can improve the situation.
The context of this error is in `DatasourceItem`, using this ingress data:
{'datasource': Munch({'type': 'prometheus', 'uid': 'f4bkXCb4z'}), 'type': 'prometheus', 'uid': 'f4bkXCb4z'}
  warnings.warn(
2024-04-05 15:44:53,251 [grafana_wtf.core                    ] WARNING : Data source 'prometheus.example.com' not found
2024-04-05 15:44:53,252 [grafana_wtf.core                    ] WARNING : Data source 'Prometheus' not found
/home/user/.local/lib/python3.12/site-packages/grafana_wtf/model.py:181: UserWarning: The `datasource` attribute is ignored for the time being.
See also: https://github.com/panodata/grafana-wtf/issues/110
Please report back this occurrence to the grafana-wtf issue tracker,
so the maintainers can improve the situation.
The context of this error is in `DatasourceItem`, using this ingress data:
{'datasource': Munch({'type': 'prometheus', 'uid': 'LeC8GuI4k'}), 'type': 'prometheus', 'uid': 'LeC8GuI4k'}
  warnings.warn(
2024-04-05 15:44:53,254 [grafana_wtf.core                    ] WARNING : Data source 'dev-prometheus.example.com' not found
2024-04-05 15:44:53,254 [grafana_wtf.core                    ] WARNING : Data source '$__all' not found
2024-04-05 15:44:53,255 [grafana_wtf.core                    ] WARNING : Data source 'prometheus' not found
2024-04-05 15:44:53,256 [grafana_wtf.core                    ] WARNING : Data source 'dev-prometheus.example.com' not found
2024-04-05 15:44:53,257 [grafana_wtf.core                    ] WARNING : Data source '' not found
2024-04-05 15:44:53,257 [grafana_wtf.core                    ] WARNING : Data source 'None' not found
2024-04-05 15:44:53,257 [grafana_wtf.core                    ] WARNING : Data source 'None' not found
2024-04-05 15:44:53,259 [grafana_wtf.core                    ] WARNING : Data source 'VictoriaMetrics' not found
2024-04-05 15:44:53,265 [grafana_wtf.commands                ] WARNING : Found 12 unused data source(s)

And then a list of all the datasources.

Regarding the dashboard JSON dumps, We currently have 194 of them. Do you want me to send the output of grafana-wtf explore dashboards or the whole JSON representation of the dashboards?

Thanks for helping out!

[apepojken]

Also, please tell me if you need any additional info on the datasources that trigger the warning

[amotl]

Dear @apepojken,

The patch GH-131 actually solved my problem.

Excellent, thanks! It will be merged and included into the upcoming release.

Regarding the dashboard JSON dumps, We currently have 194 of them. Do you want me to send the output of all of them?

I don't think that is needed now that it works for you.

So, it seems it might be really an edge case on your end.

From the output shared above, the spot that encodes references to data sources apparently has a redundant bit of information in your Grafana instance:

{
  "datasource": {
    "type": "prometheus",
    "uid": "LeC8GuI4k"
  },
  "type": "prometheus",
  "uid": "LeC8GuI4k"
}

I think the "datasource" item within the data source item is redundant and not used at all.

With kind regards,
Andreas.