Improve scrapers/mdatagen tool to produce more user-friendly code and support user settings

[dmitryax]

Metrics Builder for Scraper Receivers

Design document: https://docs.google.com/document/d/1OwDbIFlugS1rR0STv7rYBYcUjDZQ45pyVUvOSFfaRp4

Goals

Provide a user-friendly interface for metrics reporting by mdatagen tool

Currently mdatagen tool produces generated_metrics.go files that contain mostly definitions of the metrics taken from metadata.yaml files. Scraper authors have to add some amount of repetitive code to each of the scrapers to fill the metrics with data points and send them downstream. We want to reduce the amount of the repetitive code and provide scraper authors with a simple interface for reporting metrics that will be generated from metadata.yaml files.

Ability to change the default metrics representation by applying user settings

We need a way to customize the default representation of the metrics generated by mdatagen based on user's config. Once we move metrics generation to the Metrics Builder generated by mdatagen, it'll be be possible to customize metrics based on user settings.

As the first step, we want to provide a way to enable or disable any metrics based on user config. This functionality should be encapsulated in the Metrics Builder generated by mdatagen, so scraper authors don't need to deal with user settings. This will also provide a way to introduce optional metrics which are disabled by default just by changing metadata.yaml properties.

Progress

Metrics builder functionality:

• Ability to disable particular metrics.
• [metrics builder] Ability to re-aggregate metric by attributes #10726
• [cmd/mdatagen] Ability to change aggregation temporality #10903
• [cmd/mdatagen] Ability to disable/enable all metrics #10074

Migrated scrapers:

• Migrate apachereceiver to the new Metrics Builder opentelemetry-collector-contrib#7135
• receiver/hostmetricsreceiver/internal/scraper/cpuscraper
• receiver/hostmetricsreceiver/internal/scraper/diskscraper
• Migrate hostmetrics / filesystemscraper to the new Metrics Builder opentelemetry-collector-contrib#7136
• Migrate hostmetrics / loadscraper to the new Metrics Builder opentelemetry-collector-contrib#7137
• Migrate hostmetrics / memoryscraper to the new Metrics Builder opentelemetry-collector-contrib#7138
• receiver/hostmetricsreceiver/internal/scraper/networkscraper
• Migrate hostmetrics / pagingscraper to the new Metrics Builder opentelemetry-collector-contrib#7139
• Migrate hostmetrics / processesscraper to the new Metrics Builder opentelemetry-collector-contrib#7140
• Migrate hostmetrics / processscraper to the new Metrics Builder opentelemetry-collector-contrib#7141
• Migrate kafkametricsreceiver to the new Metrics Builder opentelemetry-collector-contrib#7142
• Migrate kubeletstatsreceiver to the new Metrics Builder opentelemetry-collector-contrib#7143
• Migrate memcachedreceiver to the new Metrics Builder opentelemetry-collector-contrib#7144
• Migrate mongodbatlasreceiver to the new Metrics Builder opentelemetry-collector-contrib#7145
• Migrate mysqlreceiver to the new Metrics Builder opentelemetry-collector-contrib#7146
• Migrate nginxreceiver to the new Metrics Builder opentelemetry-collector-contrib#7147
• Migrate postgresqlreceiver to the new Metrics Builder opentelemetry-collector-contrib#7148
• receiver/redisreceiver
• receiver/zookeeperreceiver

Additional improvements

• [Metrics builder] Allow attribute values of any types opentelemetry-collector-contrib#8127
• [Metric builder] Replace attributes value enums with typed constants opentelemetry-collector-contrib#9595
• [Metrics builder] Generate whole pdata.Metrics object opentelemetry-collector-contrib#8358
• Scrapers: separate resource attributes in metadata.yaml opentelemetry-collector-contrib#7301
• [Metrics builder] Remove generation of Attribute constants opentelemetry-collector-contrib#8842
• [Metric builder] Add external system states to be used in metric constraints opentelemetry-collector-contrib#15995
• [Metrics builder] Generate tests to cover the generated code opentelemetry-collector-contrib#16375
• [cmd/mdatagen] Generate test helpers that can be used in scraper tests opentelemetry-collector-contrib#18063
• [metrics builder] Non-string support for enum attributes opentelemetry-collector-contrib#16419
• mdatagen support for slice and map attributes opentelemetry-collector-contrib#18272

[abhide]

@dmitryax I would like to work on networkscraper receiver.

[abhide]

@dmitryax I would like to work on filesystemscraper receiver.

[dmitryax]

@abhide sounds good, thanks! I converted the action item into another issue open-telemetry/opentelemetry-collector-contrib#7136

[rogercoll]

@dmitryax I would like to work on the memoryscraper receiver.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• cmd/mdatagen: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• cmd/mdatagen: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• cmd/mdatagen: @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[codeboten]

transferred issue as mdatagen lives in the core repo now