system tables for inner metrics

[MichaelScofield]

What problem does the new feature solve?

It's good to have a metrics table like the one in Databend or ClickHouse. Users can easily "select" on the metrics table to see db's important running status. With metrics table, we can integrate other visualization tools or dashboards that supports SQL like Grafana, to export some meaningful timeseries data.

What does the feature do?

Adds system tables for inner metrics.

Implementation challenges

1. In datanode, create a new table called metrics, with schema like the MetricsTable in Databend. Register it into default catalog like our ScriptsTable do.
2. Implement "select" for metrics table. Instead of actually storing the metrics, call the render() method of PROMETHEUS_HANDLE, and parse the result (prometheus lines) as table ouput. Refer to the code here in Databend for details.
3. In frontend, create the same metrics table as well. Unlike DistTable, the metrics table does not do distributed query.

[azhsmesos]

Can you describe it in more detail？

[killme2008]

@azhsmesos Looks like the metrics table in clickhouse

https://clickhouse.com/docs/en/operations/system-tables/metrics/

[waynexia]

This issue requests one or more table in system catalog that contains information about running status. E.g., load, how much memory is used, how many running queries, how many opened fds, etc. It exposes a convenient way to do system analysis and debugging

[Lilit0x]

I don't know if this has been implemented since I can't find any indication that it has been, but I find it interesting, and I'd like to work on it. I still have some pending PRs, so this is just like an indication of my interest.
I'll come back to outline my thought process for it coupled with the implementation details that have been described above.
Thanks

[MichaelScofield]

@Lilit0x Are you still interested on this issue?

[Lilit0x]

Hi @MichaelScofield , yes, I am, but I just got assigned to another issue.
If it isn't urgent, I'll like to come back to it.

[MichaelScofield]

@Lilit0x Not urgent. Just a friendly ping. Come back to this issue when you have time, thx~

[waynexia]

Hi @MichaelScofield , yes, I am, but I just got assigned to another issue. If it isn't urgent, I'll like to come back to it.

Thanks for your enthusiasm! This is not hurry, and we are making some changes related to our internal metrics #1906. Will update this issue when it's unblocked ❤️

[tisonkun]

@waynexia @killme2008 I think some of the features needed by this issue have been implemented in information_schema.

Perhaps we can close this issue and open a new issue if there is any ongoing items.

[killme2008]

@waynexia @killme2008 I think some of the features needed by this issue have been implemented in information_schema.

Perhaps we can close this issue and open a new issue if there is any ongoing items.

Yes. We have runtime_metrics and try to implement xxx_stats etc. in information_schema.