Telemetry collection p2 #222
Conversation
…y of DBOS context and add it to every span
maxdml
force-pushed
the
telemetry-collection-p2
branch
from
2fb9444 to
2e63bfd
Compare
| "@opentelemetry/exporter-logs-otlp-proto": "0.46.0", | ||
| "@opentelemetry/exporter-trace-otlp-proto": "0.46.0", |
There was a problem hiding this comment.
Use the latest version and lock it. These are experimental packages and breaking changes happen often.
| @@ -276,7 +283,7 @@ export class DBOSExecutor { | |||
| } catch (err) { | |||
| (err as Error).message = `failed to initialize workflow executor: ${(err as Error).message}` | |||
| this.logger.error(err); | |||
| throw new DBOSInitializationError(`failed to initialize workflow executor: ${(err as Error).message}`); | |||
| throw new DBOSInitializationError(`${(err as Error).message}`); | |||
There was a problem hiding this comment.
Remove duplicate text in this error message.
| export const OperationType = { | ||
| HANDLER: "handler", | ||
| WORKFLOW: "workflow", | ||
| TRANSACTION: "transaction", | ||
| COMMUNICATOR: "communicator", | ||
| } as const; |
There was a problem hiding this comment.
Useful as span metadata
| if (koaContext.request.headers && koaContext.request.headers[DBOSExecutorIDHeader]) { | ||
| this.executorID = koaContext.request.headers[DBOSExecutorIDHeader] as string; | ||
| } |
There was a problem hiding this comment.
Now get executor ID as soon as we create the handler context
| traceId: string; | ||
| spanId: string; | ||
| includeContextMetadata: boolean; // Should the console transport formatter include the context metadata? | ||
| span: Span; // All context metadata should be attributes of the context's span |
There was a problem hiding this comment.
We use the span attributes as bearer for all contextual metadata
| const logRecordProcessor = { | ||
| forceFlush: async () => { | ||
| // no-op | ||
| }, | ||
| onEmit(logRecord: LogRecord) { | ||
| telemetryCollector.push(logRecord); | ||
| }, | ||
| shutdown: async () => { | ||
| // no-op | ||
| }, | ||
| }; |
There was a problem hiding this comment.
This is a custom log processor we can given to the OTel logger. It's only job is to push log records in our collector queue.
We might want to tweak its parameters in the future.
| @@ -146,7 +146,7 @@ describe("recovery-tests", () => { | |||
| ExecutorRecovery.resolve2 = resolve; | |||
| }); | |||
|
|
|||
| const execHandle = await testRuntime.invoke(ExecutorRecovery, undefined, { authenticatedUser: "cloud_user", request: { headers: { "dbos-executor-id": "fcvm123" } } }).executorWorkflow(5); | |||
| const execHandle = await testRuntime.invoke(ExecutorRecovery, undefined, { authenticatedUser: "cloud_user", request: { headers: { "dbos-executor-id": "fcvm123", executorID: "fcvm123" } } }).executorWorkflow(5); | |||
There was a problem hiding this comment.
works around the fact that the testing runtime does not use the handlers
| @@ -27,7 +27,17 @@ export class WorkflowContextDebug extends DBOSContextImpl implements WorkflowCon | |||
| readonly isTempWorkflow: boolean; | |||
|
|
|||
| constructor(dbosExec: DBOSExecutor, parentCtx: DBOSContextImpl | undefined, workflowUUID: string, readonly workflowConfig: WorkflowConfig, workflowName: string) { | |||
| const span = dbosExec.tracer.startSpan(workflowName, { workflowUUID: workflowUUID }, parentCtx?.span); | |||
| const span = dbosExec.tracer.startSpan( | |||
| workflowName, | |||
There was a problem hiding this comment.
Why are we doing any sort of tracing in debug workflows? @qianl15 what do you think?
There was a problem hiding this comment.
Yeah right now the usage is unclear. Maybe we could need it for retro action, in the future. Happy to remove it if @qianl15 doesn't find it useful.
Correct |
executorID1) a property of the context 2) parsed from headers at the handler (earlier than where it was before, when creating a workflow context). ExecutorID is "local" by default and will take on the header's if provided. Note we have to keep the logic ininternalWorkflowto accommodate the testing runtime. Eventually we should just expose the executor ID to the environment.