Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkg/ottl: add section about design principles #29424

Merged
merged 7 commits into from
Dec 7, 2023
Merged

pkg/ottl: add section about design principles #29424

Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
7d8609c
pkg/ottl: add section about design principles
axw Nov 21, 2023
c58ee35
Fix link
axw Nov 21, 2023
4f7a2d9
Clarifications
axw Nov 28, 2023
b50deca
Address review comments
axw Dec 4, 2023
35e437a
Apply suggestions from code review
axw Dec 4, 2023
84eb4bb
Merge branch 'main' into ottl-readme-terminating
axw Dec 4, 2023
be2cd7f
Update pkg/ottl/LANGUAGE.md
axw Dec 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion pkg/ottl/LANGUAGE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@

The OTTL grammar includes function invocations, Values and Boolean Expressions. These parts all fit into a Statement, which is the basis of execution in the OTTL.

### Design principles
axw marked this conversation as resolved.
Show resolved Hide resolved

The OTTL and built-in [ottlfuncs](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/ottlfuncs) are designed according to these principles:
- OTTL programs operating in separate contexts cannot influence one another -- an OTTL program may have side-effects only within its own execution [context](#contexts).
- OTTL programs cannot loop forever, except through the use of non built-in functions.
Copy link
Member

@TylerHelmuth TylerHelmuth Nov 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like the idea of documenting design principals, but I am not sure I want to commit to these yet.

OTTL programs operating in separate contexts cannot influence one another -- an OTTL program may have side-effects only within its own execution context.

While the current contexts have restricted cross-cutting between signals, there isn't anything stopping someone from using OTTL with a custom context that does allow this. OTTL is designed to not worry about how the getting and setting is done or what data is being manipulated. I'd prefer to not restrict ourselves with this principal because some day it may be necessary to support more cross-cutting scenarios.

OTTL programs cannot loop forever, except through the use of non built-in functions.

Can you provide more what you mean by this statement? I disagree with the term program as OTTL is not a programming language.

A design principles for OTTL that I do feel confident about documenting are:

  1. Context cannot provide paths that reach "lower" in the OTLP heirarchy. A function in the metric context, for example, can provide access to the DataPointSlice, but not individual datapoints.
  2. OTTL Functions should trend towards erroring when dealing with unexpected results or situations.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TylerHelmuth thanks for your thoughts!

While the current contexts have restricted cross-cutting between signals, there isn't anything stopping someone from using OTTL with a custom context that does allow this. OTTL is designed to not worry about how the getting and setting is done or what data is being manipulated. I'd prefer to not restrict ourselves with this principal because some day it may be necessary to support more cross-cutting scenarios.

Sorry, I worded this poorly. What I intended was to (roughly) capture this Starlark design principle:

  • Hermetic execution. Execution cannot access the file system, network, system clock. It is safe to execute untrusted code.

Execution should definitely not be able to access the system or network. I don't see any reason to exclude the system clock, and doing so would would exclude the Now() converter.

In addition to that, what I meant that execution of one OTTL program (sorry, need another word) must not be able to leak into the execution of another.

OTTL programs cannot loop forever, except through the use of non built-in functions.

Can you provide more what you mean by this statement?

I was trying to capture what CEL says:

  • terminating: CEL programs cannot loop forever;

The point being to guarantee that CEL can be evaluated without taking down the system evaluating it. It doesn't matter so much for single-tenant systems, since you can just say "don't do that." But for multi-tenant systems it will affect others.

It's not quite enough to say that about the grammar -- as long as there are built-in functions, they would be an escape hatch if they're not also covered by this principle. We can't really enforce what a non built-in function does, hence the comment "except through the use of non built-in functions".

I disagree with the term program as OTTL is not a programming language.

Right, I wasn't sure what to call it, and it felt a little wrong even if "program" doesn't necessarily mean "general purpose program". FWIW, I went looking for inspiration in cuelang, and found a few references to "CUE program" at:

Having said that, it doesn't appear to be defined anywhere formally 🤷‍♂️

Is there an established word to describe a series of OTTL statements?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hermetic execution. Execution cannot access the file system, network, system clock. It is safe to execute untrusted code.

I'm not sure we want to restrict functions in this way. Although we wouldn't include it in the transform processor, OTTL as a framework allows users to create functions that interact with the network or filesystem. Since function inclusion is per-component and a compile-time decision there is no untrusted code - OTTL does not support dynamic function loading or remote code execution.

In addition to that, what I meant that execution of one OTTL program (sorry, need another word) must not be able to leak into the execution of another.

This I do agree with. OTTL statements should be completely independent.

It's not quite enough to say that about the grammar -- as long as there are built-in functions, they would be an escape hatch if they're not also covered by this principle. We can't really enforce what a non built-in function does, hence the comment "except through the use of non built-in functions".

Ok I see what you're saying. Yes I think it is safe to say we wouldn't create any functions in ottlfuncs that would loop forever.

Is there an established word to describe a series of OTTL statements?

Not officially. I refer to them a statements.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we move these design goals to be in the ottlfuncs readme? I think they make a lot of sense for the OTTL standard library, which I would expect to be limited in scope and to have good security guarantees. For the language itself, since OTTL statements run directly in the Go VM we can't make any guarantees for what a user-authored function could do in its implementation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we move these design goals to be in the ottlfuncs readme? I think they make a lot of sense for the OTTL standard library, which I would expect to be limited in scope and to have good security guarantees.

Sounds good, I'll have a go at that.

For the language itself, since OTTL statements run directly in the Go VM we can't make any guarantees for what a user-authored function could do in its implementation.

I agree that we can't make any guarantees about user-defined functions, but I still think it's important to address the language to cover any future changes. For example #29289 has a few options, one of which is to add looping to the language. That's fine because the proposed loop syntax would guarantee termination. (Obviously you know all that, just giving an example to explain my rationale.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since function inclusion is per-component and a compile-time decision there is no untrusted code - OTTL does not support dynamic function loading or remote code execution.

Understood. Should we capture that here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a mention of no dynamic loading/evaluation, and moved some bits to ottlfuncs.


### Editors

Editors are functions that transform the underlying telemetry payload. They may return a value, but typically do not. There must be a single Editor Invocation in each OTTL statement.
Expand Down Expand Up @@ -288,4 +294,4 @@ It is possible to update the Value in a telemetry field using a Setter. For read

## Logging inside a OTTL function

To emit logs inside a OTTL function, add a parameter of type [`component.TelemetrySettings`](https://pkg.go.dev/go.opentelemetry.io/collector/component#TelemetrySettings) to the function signature. The OTTL will then inject the TelemetrySettings that were passed to `NewParser` into the function. TelemetrySettings can be used to emit logs.
To emit logs inside a OTTL function, add a parameter of type [`component.TelemetrySettings`](https://pkg.go.dev/go.opentelemetry.io/collector/component#TelemetrySettings) to the function signature. The OTTL will then inject the TelemetrySettings that were passed to `NewParser` into the function. TelemetrySettings can be used to emit logs.
Loading