Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add semantic conventions for cloud provider-specific resource attributes #1099

Merged
merged 9 commits into from
Nov 25, 2020
2 changes: 1 addition & 1 deletion semantic_conventions/resource/cloud.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ groups:
brief: 'Amazon Web Services'
- id: Azure
value: 'azure'
brief: 'Amazon Web Services'
brief: 'Microsoft Azure'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😄

- id: GCP
value: 'gcp'
brief: 'Google Cloud Platform'
Expand Down
38 changes: 38 additions & 0 deletions semantic_conventions/resource/cloud_provider/aws/ecs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
groups:
- id: aws.ecs
prefix: aws.ecs
brief: >
Resources used by AWS Elastic Container Service (ECS).
attributes:
- id: container.arn
type: string
brief: >
The Amazon Resource Name (ARN) of an [ECS container instance](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_instances.html).
examples: ['arn:aws:ecs:us-west-1:123456789123:container/32624152-9086-4f0e-acae-1a75b14fe4d9']
- id: cluster.arn
type: string
brief: >
The ARN of an [ECS cluster](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/clusters.html).
examples: ['arn:aws:ecs:us-west-2:123456789123:cluster/my-cluster']
- id: launchtype
type:
allow_custom_values: false
members:
- id: ec2
value: "EC2"
- id: fargate
value: "Fargate"
brief: >
The [launch type](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/launch_types.html) for an ECS task.
examples: ['EC2', 'Fargate']
- id: task.arn
type: string
brief: >
The ARN of an [ECS task definition](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html).
examples: ['arn:aws:ecs:us-west-1:123456789123:task/10838bed-421f-43ef-870a-f43feacbbb5b']
- id: task.family
type: string
brief: >
The task definition family this task definition is a member of.
examples: ['opentelemetry-family']

38 changes: 38 additions & 0 deletions semantic_conventions/resource/cloud_provider/aws/logs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
groups:
- id: aws.log
prefix: aws.log
brief: >
Resources specific to Amazon Web Services.
attributes:
- id: group.names
type: string[]
brief: >
The name(s) of the AWS log group(s) an application is writing to.
examples: ['/aws/lambda/my-function', 'opentelemetry-service']
note: >
Multiple log groups must be supported for cases like multi-container applications,
where a single application has sidecar containers, and each write to their own log
group.
- id: group.arns
type: string[]
brief: >
The Amazon Resource Name(s) (ARN) of the AWS log group(s).
examples: ['arn:aws:logs:us-west-1:123456789012:log-group:/aws/my/group:*']
note: >
See the
[log group ARN format documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/iam-access-control-overview-cwl.html#CWL_ARN_Format).
- id: stream.names
type: string[]
brief: >
The name(s) of the AWS log stream(s) an application is writing to.
examples: ['logs/main/10838bed-421f-43ef-870a-f43feacbbb5b']
- id: stream.arns
type: string[]
brief: >
The ARN(s) of the AWS log stream(s).
examples: ['arn:aws:logs:us-west-1:123456789012:log-group:/aws/my/group:log-stream:logs/main/10838bed-421f-43ef-870a-f43feacbbb5b']
note: >
See the
[log stream ARN format documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/iam-access-control-overview-cwl.html#CWL_ARN_Format).
One log group can contain several log streams, so these ARNs necessarily identify both a log
group and a log stream.
11 changes: 11 additions & 0 deletions specification/resource/semantic_conventions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,3 +135,14 @@ Attributes defining a running environment (e.g. Operating System, Cloud, Data Ce
Version attributes, such as `service.version`, are values of type `string`. They are
the exact version used to identify an artifact. This may be a semantic version, e.g., `1.2.3`, git hash, e.g.,
`8ae73a`, or an arbitrary version string, e.g., `0.1.2.20210101`, whatever was used when building the artifact.

## Cloud-Provider-Specific Attributes

Attributes that are only applicable to resources from a specific cloud provider. Currently, these
resources can only be defined for providers listed as a valid `cloud.provider` in
[Cloud](./cloud.md) and below. Provider-specific attributes all reside in the `cloud_provider` directory.
Valid cloud providers are:

- [Amazon Web Services](https://aws.amazon.com/) ([`aws`](cloud_provider/aws/README.md))
- [Google Cloud Platform](https://cloud.google.com/) (`gcp`)
- [Microsoft Azure](https://azure.microsoft.com/) (`azure`)
2 changes: 1 addition & 1 deletion specification/resource/semantic_conventions/cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,6 @@
| Value | Description |
|---|---|
| `aws` | Amazon Web Services |
| `azure` | Amazon Web Services |
| `azure` | Microsoft Azure |
| `gcp` | Google Cloud Platform |
<!-- endsemconv -->
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# AWS Semantic Conventions

This directory defines standards for resource attributes that only apply to Amazon
Web Services (AWS) resources. If an attribute could apply to resources from more than one cloud
provider (like account ID, operating system, etc), it belongs in the parent
`semantic_conventions` directory.

## Generic AWS Attributes

Attributes that relate to AWS or use AWS-specific terminology, but are used by several
services within AWS or are abstracted away from any particular service:

- [AWS Logs](./logs.md)

## Services

Attributes that relate to an individual AWS service:

- [Elastic Container Service](./ecs.md)
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# AWS ECS

**type:** `aws.ecs`

**Description:** Resources used by AWS Elastic Container Service (ECS).

<!-- semconv aws.ecs -->
| Attribute | Type | Description | Example | Required |
|---|---|---|---|---|
| `aws.ecs.container.arn` | string | The Amazon Resource Name (ARN) of an [ECS container instance](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_instances.html). | `arn:aws:ecs:us-west-1:123456789123:container/32624152-9086-4f0e-acae-1a75b14fe4d9` | No |
| `aws.ecs.cluster.arn` | string | The ARN of an [ECS cluster](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/clusters.html). | `arn:aws:ecs:us-west-2:123456789123:cluster/my-cluster` | No |
Copy link

@hossain-rayhan hossain-rayhan Nov 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we get task metadata from ECS Task Metadata Endpoint, it gives us the ClusterName instead of ClusterARN for EC2 Launch Type. Can't we add one for aws.ecs.cluster.name?

Copy link
Contributor Author

@willarmiros willarmiros Nov 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I originally had it as cluster.name but, since I was populating the ARN for nearly all other fields and consumers could easily derive cluster name from the ARN, I figured recording the cluster ARN made more sense. On Fargate we are provided the cluster ARN by default, and on EC2 launch type, we have the region and account ID from the task ARN, so you can always construct the cluster ARN like I do here.

All this being said, I can still add a new field for the cluster name if you need it. I just wanted to reduce the possibility for redundant info.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess very few customers care about the whole ARN. Most useful info here is the resource name (ClusterName or TaskID). It's also implies that all the exporters should write their logic to extract resource name from ARN. It's not only extracting the resource name, they need to do other checks before that. If resource attribute map does not have the name fields, extract them and apply. The logic is not super complex but extra work and check.
So, in a sense, we are blocking our customers if any specific exporter doesn't come with these logics.

Also, I am kind of convinced with the idea of less redundant info. I am OK to follow the experts opinion.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed offline: We're ok with this as is for now, and in the future if customers ask for it we can add cluster name, task ID, etc to this spec.

| `aws.ecs.launchtype` | string enum | The [launch type](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/launch_types.html) for an ECS task. | `EC2`<br>`Fargate` | No |
| `aws.ecs.task.arn` | string | The ARN of an [ECS task definition](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definitions.html). | `arn:aws:ecs:us-west-1:123456789123:task/10838bed-421f-43ef-870a-f43feacbbb5b` | No |
| `aws.ecs.task.family` | string | The task definition family this task definition is a member of. | `opentelemetry-family` | No |

`aws.ecs.launchtype` MUST be one of the following:

| Value | Description |
|---|---|
| `EC2` | ec2 |
| `Fargate` | fargate |
<!-- endsemconv -->
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# AWS Logs

**Type:** `aws.log`

**Description:** Log attributes for Amazon Web Services.

<!-- semconv aws.log -->
| Attribute | Type | Description | Example | Required |
|---|---|---|---|---|
| `aws.log.group.names` | string[] | The name(s) of the AWS log group(s) an application is writing to. [1] | `/aws/lambda/my-function`<br>`opentelemetry-service` | No |
| `aws.log.group.arns` | string[] | The Amazon Resource Name(s) (ARN) of the AWS log group(s). [2] | `arn:aws:logs:us-west-1:123456789012:log-group:/aws/my/group:*` | No |
| `aws.log.stream.names` | string[] | The name(s) of the AWS log stream(s) an application is writing to. | `logs/main/10838bed-421f-43ef-870a-f43feacbbb5b` | No |
| `aws.log.stream.arns` | string[] | The ARN(s) of the AWS log stream(s). [3] | `arn:aws:logs:us-west-1:123456789012:log-group:/aws/my/group:log-stream:logs/main/10838bed-421f-43ef-870a-f43feacbbb5b` | No |

**[1]:** Multiple log groups must be supported for cases like multi-container applications, where a single application has sidecar containers, and each write to their own log group.

**[2]:** See the [log group ARN format documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/iam-access-control-overview-cwl.html#CWL_ARN_Format).

**[3]:** See the [log stream ARN format documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/iam-access-control-overview-cwl.html#CWL_ARN_Format). One log group can contain several log streams, so these ARNs necessarily identify both a log group and a log stream.
<!-- endsemconv -->