[MLOB-1560] LLMObs Span Processor #4738

sabrenner · 2024-09-30T13:11:02Z

What does this PR do?

Adds a span processor for the LLM Observability product. An instance of it is now a property on the tracer span processor, and it reads the temporary tags on the span and extracts them into an LLM Obs payload to append to the writer. Then, these tags are not read when formatting the span.

Additionally, makes an internal change to how these are stored on tags. Since JSON.stringify and JSON.parse can be expensive, it is better to only serialize at the writer flushing stage. However, by storing tags as plain objects and not stringifying at the time they are added, we do not get the benefit of checking if they are will not throw when stringified. This is fixed by adding some specific guards in-place at the tagger (for metrics, documents, and messages) and span processor (for metadata, which has ambiguous value types).

Future work is planned to try and replace tagging the span with using some kind of external, namespaced storage.

Motivation

Follow-up PR in a series of PRs introducing an LLM Observability SDK into the Node.js tracer.

github-actions · 2024-09-30T13:12:03Z

Overall package size

Self size: 7.24 MB
Deduped: 62.61 MB
No deduping: 62.89 MB

Dependency sizes

| name | version | self size | total size | |------|---------|-----------|------------| | @datadog/native-appsec | 8.1.1 | 18.67 MB | 18.68 MB | | @datadog/native-iast-taint-tracking | 3.1.0 | 12.27 MB | 12.28 MB | | @datadog/pprof | 5.3.0 | 9.85 MB | 10.22 MB | | protobufjs | 7.2.5 | 2.77 MB | 5.16 MB | | @datadog/native-iast-rewriter | 2.4.1 | 2.14 MB | 2.23 MB | | @opentelemetry/core | 1.14.0 | 872.87 kB | 1.47 MB | | @datadog/native-metrics | 2.0.0 | 898.77 kB | 1.3 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | jsonpath-plus | 9.0.0 | 580.4 kB | 1.03 MB | | import-in-the-middle | 1.8.1 | 71.67 kB | 785.15 kB | | msgpack-lite | 0.1.26 | 201.16 kB | 281.59 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.0 | 109.9 kB | 109.9 kB | | semver | 7.6.3 | 95.82 kB | 95.82 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | lru-cache | 7.14.0 | 74.95 kB | 74.95 kB | | ignore | 5.3.1 | 51.46 kB | 51.46 kB | | int64-buffer | 0.1.10 | 49.18 kB | 49.18 kB | | shell-quote | 1.8.1 | 44.96 kB | 44.96 kB | | istanbul-lib-coverage | 3.2.0 | 29.34 kB | 29.34 kB | | rfdc | 1.3.1 | 25.21 kB | 25.21 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | dc-polyfill | 0.1.4 | 23.1 kB | 23.1 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | path-to-regexp | 0.1.10 | 6.38 kB | 6.38 kB | | module-details-from-path | 1.0.3 | 4.47 kB | 4.47 kB |

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

packages/dd-trace/src/exporters/agent/index.js

packages/dd-trace/src/llmobs/span_processor.js

pr-commenter · 2024-10-02T23:19:32Z

Benchmarks

Benchmark execution time: 2024-10-08 20:43:19

Comparing candidate commit c2408d0 in PR branch sabrenner/llmobs-span-processor with baseline commit bddfa3d in branch sabrenner/llmobs-sdk-release.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 260 metrics, 6 unstable metrics.

packages/dd-trace/src/llmobs/span_processor.js

packages/dd-trace/src/format.js

packages/dd-trace/src/llmobs/span_processor.js

packages/dd-trace/src/llmobs/tagger.js

packages/dd-trace/src/span_processor.js

packages/dd-trace/src/llmobs/tagger.js

… tags

* [MLOB-1540] add llmobs configuration to global tracer config (#4696) add llmobs config * [MLOB-1555] LLM Observability writers (#4699) LLM Observability writers * [MLOB-1556] LLM Observability tagger (#4718) LLM Observability tagger * [MLOB-1560] LLMObs Span Processor (#4738) * span processor * tests * remove agent exporter log and do not stringify tags * remove llmobs from exporter tests * add in default unserializable value * review comments * warning log for metric * todo-ify * remove some duplicate logic * decouple llmobs span processing with a channel * use a static weakmap to store llmobs tags/annotations instead of span tags * do not register span in map if it does not have an llmobs span kind * span is passed on an object from sp publisher * re-clarify TODOs * only send span in publish * log multiple warnings and return conditional undefined * update error logic * [MLOB-1561] LLM Observability SDK API (#4773) * wip * type definitions * active + try/catch eval metric writer append * test ts * use tagger map and processor as a channel subscriber * change decorate and add in dev changes * try some api changes * add decorate to noop * fix breaking proxy tests * experimental decorators for TS docs * api changes, fix unit + e2e tests * try removing global log mocks * add some util tests * remove logger mocks * add module tests + do not enable when not specified * fix eval metric integration test * wip * memoize getFunctionArguments * move any subscriber and global writer to the module enablement level instead of sdk * should fix TS tests * add ts integration test and fix decorator * devex for ts versions * add noop typescript test * remove startSpan * remove unneeded change * dedup decorator code * Update index.d.ts Co-authored-by: Yun Kim <35776586+Yun-Kim@users.noreply.github.com> * map metrics names * change validKind to validateKind and throw * tagger for metrics follow-up * review feedback * add some tests for not auto-annotating in certain cases --------- Co-authored-by: Yun Kim <35776586+Yun-Kim@users.noreply.github.com> * hard fail instead of soft fail, except for `wrap` span name * add ml-observability codeowners * resolve ts test * update auto-annotation check * tagger can soft fail * using custom ASL instance and scope activation * fix test comments and remove * address review comments * remove llmobs.apiKey config, only rely on global * fix evaulations test * make llmobs storage accessible --------- Co-authored-by: Yun Kim <35776586+Yun-Kim@users.noreply.github.com>

sabrenner added 2 commits September 27, 2024 15:54

span processor

0d8aea0

tests

c6e2dba

sabrenner commented Sep 30, 2024

View reviewed changes

packages/dd-trace/src/exporters/agent/index.js Outdated Show resolved Hide resolved

sabrenner commented Sep 30, 2024

View reviewed changes

packages/dd-trace/src/llmobs/span_processor.js Outdated Show resolved Hide resolved

sabrenner added 2 commits October 2, 2024 18:40

remove agent exporter log and do not stringify tags

1aae7c3

remove llmobs from exporter tests

62a9cc9

sabrenner marked this pull request as ready for review October 2, 2024 23:10

sabrenner requested a review from a team as a code owner October 2, 2024 23:10

sabrenner commented Oct 3, 2024

View reviewed changes

packages/dd-trace/src/llmobs/span_processor.js Outdated Show resolved Hide resolved

add in default unserializable value

53c1d1a

lievan reviewed Oct 3, 2024

View reviewed changes

sabrenner added 2 commits October 3, 2024 16:32

review comments

792054a

warning log for metric

7b7660d

lievan approved these changes Oct 3, 2024

View reviewed changes

todo-ify

a8eeef8

rochdev requested changes Oct 7, 2024

View reviewed changes

packages/dd-trace/src/llmobs/tagger.js Outdated Show resolved Hide resolved

packages/dd-trace/src/span_processor.js Outdated Show resolved Hide resolved

packages/dd-trace/src/llmobs/tagger.js Outdated Show resolved Hide resolved

sabrenner added 9 commits October 7, 2024 09:32

remove some duplicate logic

80168f8

decouple llmobs span processing with a channel

645577c

use a static weakmap to store llmobs tags/annotations instead of span…

4a19fec

… tags

do not register span in map if it does not have an llmobs span kind

070b564

span is passed on an object from sp publisher

3b12cb5

re-clarify TODOs

8fc59ad

only send span in publish

2698b5f

log multiple warnings and return conditional undefined

bdf5fc0

update error logic

c2408d0

rochdev approved these changes Oct 9, 2024

View reviewed changes

sabrenner merged commit 38225ed into sabrenner/llmobs-sdk-release Oct 9, 2024
191 checks passed

sabrenner deleted the sabrenner/llmobs-span-processor branch October 9, 2024 21:06

sabrenner mentioned this pull request Oct 9, 2024

[MLOB-1524] feat(llmobs): Introduce LLM Observability SDK #4742

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MLOB-1560] LLMObs Span Processor #4738

[MLOB-1560] LLMObs Span Processor #4738

sabrenner commented Sep 30, 2024 •

edited

Loading

github-actions bot commented Sep 30, 2024 •

edited

Loading

pr-commenter bot commented Oct 2, 2024 •

edited

Loading

[MLOB-1560] LLMObs Span Processor #4738

[MLOB-1560] LLMObs Span Processor #4738

Conversation

sabrenner commented Sep 30, 2024 • edited Loading

What does this PR do?

Motivation

github-actions bot commented Sep 30, 2024 • edited Loading

Overall package size

pr-commenter bot commented Oct 2, 2024 • edited Loading

Benchmarks

sabrenner commented Sep 30, 2024 •

edited

Loading

github-actions bot commented Sep 30, 2024 •

edited

Loading

pr-commenter bot commented Oct 2, 2024 •

edited

Loading