aspire-dashboard Structured logs -> Log entry details -> Resource attributes not always correct/seem cached

[dlosch]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Hi,

when exporting otel logging data from a .NET app to both Seq and Aspire Dashboard (8.0.2 and 8.1.0 show same behavior), aspire dashboard doesn't seem to render attribute values under Structured logs -> Log entry details -> Resource -> attributes correctly at all times (sometimes it seems parts of the attributes get cached).

First of all, displayed values for a specific log entry differ between seq (correct) and aspire dashboard (not correct).

1.) Run the app, attribute foo=value1
2.) Seq and Aspire dashboard display the entry with foo=value1
3.) stop app, change value to foo=newValue
4.) re run app
5.) Seq=>foo=newValue, aspire dashboard (no restart of container) foo=value1

Yesterday, with 8.0.2, I also had issues that some attribute pairs did not show up at all at times in aspire dashboard (not a rendering issue).

Expected Behavior

No response

Steps To Reproduce

• asp.net 8 app
• builder.Services.AddLogging(logging => logging.AddOpenTelemetry(openTelemetryLoggerOptions => {
  openTelemetryLoggerOptions.SetResourceBuilder(
  [...]
  .AddAttributes(new Dictionary<string, object> {
  [...]

Exceptions (if any)

No response

.NET Version info

SDK 8.0.303
Runtime: 8.0.7
AOT publish

Anything else?

Affects aspire-dashboard 8.1.0 and 8.0.2

[adamint]

I think the important part is here:

5.) Seq=>foo=newValue, aspire dashboard (no restart of container) foo=value1

[samsp-msft]

James - this seems like an issue with how we associate resources on each batch to the last. We probably need some kind of hash of all the resource properties to account for potential differences.

[JamesNK]

@dlosch What do you expect to happen when a resource adds and remove attributes?

For example:

1.) Run the app, attribute foo=true, record one log entry
2.) Seq and Aspire dashboard display the entry with foo=true
3.) stop app, change attribute to bar=true
4.) re run app, record one log entry
5.) ???

There are now two log entries for the resource. What attributes should the log entry resource display?

Option 1 - resource attributes reflect what was last sent to the dashboard:
Log entry 1, resource attribute bar=true
Log entry 2, resource attribute bar=true

Option 2 - resource attributes are combined over the lifetime of the dashboard:
Log entry 1, resource attribute foo=true, bar=true
Log entry 2, resource attribute foo=true, bar=true

Option 3 - resource attributes are snapshotted to the point in time they were received with the telemetry:
Log entry 1, resource attribute foo=true
Log entry 2, resource attribute bar=true

Option 1 & 2 would be simple to do. Option 3 would require larger changes.

I'm curious what seq does.

[dlosch]

Would prefer option 3 if possible. I only superficially scanned over the open telemetry spec, but my interpretation was that the attributes are effectively scoped to the log entry (at least regarding the output)(?) Are resource attributes transmitted independently (and only if changed) from log entries and then correlated back, or are these attributes attached to each record (when adding a console logger, they are attached to each log entry)? The spec also mentions that a resource is immutable ... how is the intended behavior if a resource is restarted? In my case I did attach the runtimeFrameworkVersion value to the resource as an attribute, which may change between restarts.

A quick test showed that Seq behaves as in option 3.

The dashboard is superb, has helped me track down several not obvious issues with aot (logging+metrics).

[liammclennan]

Seq stores all the resource data on all events - so Option 3.

[JamesNK]

Sounds like option 3 is the way to go. It will be a non-trivial change: the current design is resources are added once, never change, and referenced directly by telemetry. But hopefully not too difficult to change to support having a right resource attribtues against telemetry.

We'll still want to share the resource as much as possible, and then have snapshots of the resource attributes for telemetry. Snapshots should be shared when equal to keep memory usage down.

[JamesNK]

Fixed with #5526