[cmd/mdatagen] Document metadata.yaml schema (open-telemetry#22962)

* Start adding the status information to the metadata schema documentation

* Remove cmd from the list of possible classes

* Fix spelling mistake

Co-authored-by: Antoine Toulme <antoine@toulme.name>

* Update the value used for the development stability status

Co-authored-by: Antoine Toulme <antoine@toulme.name>

* Fix the tests after updating the example metadata file

* Document how to use mdatagen and the schema of metadata.yaml

* Add the stability levels for connectors

* Change the receiver type so the tests pass

* Update the generated documentation

* Change the syntax used for string array types so that it is valid YAML

---------

Co-authored-by: Antoine Toulme <antoine@toulme.name>

cmd/mdatagen/README.md

@@ -1,42 +1,56 @@
 # Metadata Generator
 
-Components must contain a `metadata.yaml` file that documents various aspects of the component including:
+Every component's documentation should include a brief description of the component and guidance on how to use it.
+There is also some information about the component (or metadata) that should be included to help end-users understand the current state of the component and whether it is right for their use case.
+Examples of this metadata about a component are:
 
 * its stability level
 * the distributions containing it
 * the types of pipelines it supports
 * metrics emitted in the case of a scraping receiver
 
-Current examples:
+The metadata generator defines a schema for specifying this information to ensure it is complete and well-formed.
+The metadata generator is then able to ingest the metadata, validate it against the schema and produce documentation in a standardized format.
+An example of how this generated documentation looks can be found in [documentation.md](./documentation.md).
 
-* hostmetricsreceiver scrapers like the [cpuscraper](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/hostmetricsreceiver/internal/scraper/cpuscraper/metadata.yaml)
+## Using the Metadata Generator
 
-See [metric-metadata.yaml](metric-metadata.yaml) for file format documentation.
+In order for a component to benefit from the metadata generator (`mdatagen`) these requirements need to be met:
+1. A `metadata.yaml` file containing the metadata needs to be included in the component
+2. The component should declare a `go:generate mdatagen` directive which tells `mdatagen` what to generate
 
-If adding a new receiver a `doc.go` file should also be added to trigger the generation. See below for details.
+As an example, here is a minimal `metadata.yaml` for the [OTLP receiver](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver):
+```yaml
+type: otlp
+status:
+  class: receiver
+  stability:
+    beta: [logs]
+    stable: [metrics, traces]
+```
 
-## Generating
+Detailed information about the schema of `metadata.yaml` can be found in [metadata-schema.yaml](./metadata-schema.yaml).
 
-`make generate` triggers the following actions:
+The `go:generate mdatagen` directive is usually defined in a `doc.go` file in the same package as the component, for example:
+```go
+//go:generate mdatagen metadata.yaml
 
-1. `cd cmd/mdatagen && $(GOCMD) install .` to install `mdatagen` tool in`GOBIN`
+package main
+```
 
-2. All `go:generate mdatagen` directives that are usually defined in `doc.go` files of the metric scrapers will be 
-   executed. For example,
-   [/receiver/hostmetricsreceiver/internal/scraper/cpuscraper/doc.go](../../receiver/hostmetricsreceiver/internal/scraper/cpuscraper/doc.go)
-   runs `mdatagen` for the [metadata.yaml](../../receiver/hostmetricsreceiver/internal/scraper/cpuscraper/metadata.yaml)
-   which generates the metrics builder code in
-   [internal/metadata](../../receiver/hostmetricsreceiver/internal/scraper/cpuscraper/internal/metadata)
-   and [documentation.md](../../receiver/hostmetricsreceiver/internal/scraper/cpuscraper/internal/metadata) with
-   generated documentation about emitted metrics.
+Below are some more examples that can be used for reference:
 
-## Development
+* The ElasticSearch receiver has an extensive [metadata.yaml](../../receiver/elasticsearchreceiver/metadata.yaml)
+* The host metrics receiver has internal subcomponents, each with their own `metadata.yaml` and `doc.go`. See [cpuscraper](../../receiver/hostmetricsreceiver/internal/scraper/cpuscraper) for example.
 
-In order to introduce support of a new functionality in metadata.yaml:
+You can run `cd cmd/mdatagen && $(GOCMD) install .` to install the `mdatagen` tool in `GOBIN` and then run `mdatagen metadata.yaml` to generate documentation for a specific component or you can run `make generate` to generate documentation for all components.
 
-1. Make code changes in the (generating code)[./loader.test] and (templates)[./templates/].
-2. Add usage of the new functionality in (metadata.yaml)[./metadata.yaml].
-3. Run `make mdatagen-test`.
-4. Make sure all tests are passing including (generated tests)[./internal/metadata/generated_metrics_test.go].
-5. Run `make generate`.
+## Contributing to the Metadata Generator
 
+The code for generating the documentation can be found in [loader.go](./loader.go) and the templates for rendering the documentation can be found in [templates](./templates).
+When making updates to the metadata generator or introducing support for new functionality:
+
+1. Ensure the [metadata-schema.yaml](./metadata-schema.yaml) and [./metadata.yaml](metadata.yaml) files reflect the changes.
+2. Run `make mdatagen-test`.
+3. Make sure all tests are passing including [generated tests](./internal/metadata/generated_metrics_test.go).
+4. Run `make generate`.

cmd/mdatagen/documentation.md

@@ -1,6 +1,6 @@
 [comment]: <> (Code generated by mdatagen. DO NOT EDIT.)
 
-# test
+# file
 
 ## Default Metrics
 

cmd/mdatagen/loader_test.go

@@ -20,8 +20,18 @@ func Test_loadMetadata(t *testing.T) {
 		{
 			name: "metadata.yaml",
 			want: metadata{
-				Type:           "test",
+				Type:           "file",
 				SemConvVersion: "1.9.0",
+				Status: &Status{
+					Class: "receiver",
+					Stability: map[string][]string{
+						"development": {"logs"},
+						"beta":        {"traces"},
+						"stable":      {"metrics"},
+					},
+					Distributions: []string{"contrib"},
+					Warnings:      []string{"Any additional information that should be brought to the consumer's attention"},
+				},
 				ResourceAttributes: map[attributeName]attribute{
 					"string.resource.attr": {
 						Description: "Resource attribute with any string value.",
@@ -146,12 +156,6 @@ func Test_loadMetadata(t *testing.T) {
 					},
 				},
 				ScopeName: "otelcol",
-				Status: &Status{
-					Class: "receiver",
-					Stability: map[string][]string{
-						"development": {"metrics"},
-					},
-				},
 			},
 		},
 		{

cmd/mdatagen/metric-metadata.yaml → cmd/mdatagen/metadata-schema.yaml

@@ -1,6 +1,23 @@
-# Required: type of the receiver.
+# Required: The type of the component - Usually the name. The type and class combined uniquely identify the component (eg. receiver/otlp) or subcomponent (eg. receiver/hostmetricsreceiver/cpu)
 type:
 
+# Required for components (Optional for subcomponents): A high-level view of the development status and use of this component
+status:
+  # Required: The class of the component (For example receiver)
+  class: <receiver|processor|exporter|connector|extension>
+  # Required: The stability of the component - See https://github.com/open-telemetry/opentelemetry-collector#stability-levels
+  stability:
+    development: [<metrics|traces|logs|traces_to_metrics|metrics_to_metrics|logs_to_metrics>]
+    alpha: [<metrics|traces|logs|traces_to_metrics|metrics_to_metrics|logs_to_metrics>]
+    beta: [<metrics|traces|logs|traces_to_metrics|metrics_to_metrics|logs_to_metrics>]
+    stable: [<metrics|traces|logs|traces_to_metrics|metrics_to_metrics|logs_to_metrics>]
+    deprecated: [<metrics|traces|logs|traces_to_metrics|metrics_to_metrics|logs_to_metrics>]
+    unmaintained: [<metrics|traces|logs|traces_to_metrics|metrics_to_metrics|logs_to_metrics>]
+  # Optional: The distributions that this component is bundled with (For example core or contrib). See statusdata.go for a list of common distros.
+  distributions: [string]
+  # Optional: A list of warnings that should be brought to the attention of users looking to use this component
+  warnings: [string]
+
 # Optional: OTel Semantic Conventions version that will be associated with the scraped metrics.
 # This attribute should be set for metrics compliant with OTel Semantic Conventions.
 sem_conv_version: 1.9.0
@@ -13,7 +30,7 @@ resource_attributes:
     # Required: description of the attribute.
     description:
     # Optional: array of attribute values if they are static values (currently, only string type is supported).
-    enum:
+    enum: [string]
     # Required: attribute value type.
     type: <string|int|double|bool|bytes|slice|map>
 
@@ -27,16 +44,16 @@ attributes:
     # Required: description of the attribute.
     description:
     # Optional: array of attribute values if they are static values (currently, only string type is supported).
-    enum:
+    enum: [string]
     # Required: attribute value type.
     type: <string|int|double|bool|bytes|slice|map>
 
-# Required: map of metric names with the key being the metric name and value
+# Optional: map of metric names with the key being the metric name and value
 # being described below.
 metrics:
   <metric.name>:
     # Required: whether the metric is collected by default.
-    enabled: # true | false
+    enabled: bool
     # Required: metric description.
     description:
     # Optional: extended documentation of the metric.
@@ -57,11 +74,11 @@ metrics:
     # Required: metric type with its settings.
     <sum|gauge>:
       # Required for sum and gauge metrics: type of number data point values.
-      value_type: # int | double
+      value_type: <int|double>
       # Required for sum metric: whether the metric is monotonic (no negative delta values).
-      monotonic: # true | false
+      monotonic: bool
       # Required for sum metric: whether reported values incorporate previous measurements
       # (cumulative) or not (delta).
-      aggregation: # delta | cumulative
+      aggregation: <delta|cumulative>
     # Optional: array of attributes that were defined in the attributes section that are emitted by this metric.
-    attributes:
+    attributes: [string]

cmd/mdatagen/metadata.yaml

@@ -1,13 +1,18 @@
 # Sample metric metadata file with all available configurations.
 
-type: test
+type: file
 
 sem_conv_version: 1.9.0
 
 status:
   class: receiver
   stability:
-    development: [metrics]
+    development: [logs]
+    beta: [traces]
+    stable: [metrics]
+  distributions: [contrib]
+  warnings:
+    - Any additional information that should be brought to the consumer's attention
 
 resource_attributes:
   string.resource.attr: