Logs dropped between Seq instances

[PhilBroderickCrezco]

We're in the process of migrating from a Seq instance running in Azure Container Instances to an Azure Virtual Machine for (hopefully) improved performance. We're doing this in a staged fashion, so are currently logging to both for our .NET apps using appsettings.json configuration that looks like:

  "Serilog": {
    "Using": ["Serilog.Sinks.Seq"],
    "MinimumLevel": {
      "Default": "Information"
    },
    "WriteTo": [
      {
        "Name": "Seq",
        "Args": {
          "serverUrl": CONTAINER_INSTANCE_URL
        }
      },
      {
        "Name": "Seq",
        "Args": {
          "serverUrl": VM_URL
        }
      }
    ]
}

We're seeing issues (specifically with one or two applications) that stop logging to the new instance after a period of time. Having enabled Serilog's SelfLog function, we're seeing errors such as:

Serilog.Debugging.LoggingFailedException: Received failed result RequestEntityTooLarge when posting events to Seq.
PeriodicBatchingSink (Serilog.Sinks.Seq.Batched.BatchedSeqSink): dropping the current batch
PeriodicBatchingSink (Serilog.Sinks.Seq.Batched.BatchedSeqSink): failed emitting a batch

But these are only affecting logs sent to the Virtual Machine instance - the Container Instance continues to receive logs just fine.

There are no differences in terms of versioning between the Seq servers (both on 2024.3.12021) and both have the default ingestion limits set (exactly the same). Furthermore, I'm not seeing any issues in the ingestion logs for either.

I've included diagnostics for both instances if that helps.
seq-diagnostics-container-instance.txt
seq-diagnostics-virtual-machine.txt

[KodrAus]

Hi @PhilBroderickCrezco 👋 Is there an application-level proxy between Seq and your virtual machine that could be rejecting those batches? If Seq itself drops incoming requests it will write to the ingestion log and record some metrics. This log doesn't persist across restarts, but your VM has been up for a while so we'd expect to see something in there.

[PhilBroderickCrezco]

We have an nginx proxy running in AKS that we're routing the logs generated by the applications through to Seq. Nothing special on the Virtual Machine running Seq - nothing extra installed. We were seeing some warnings in nginx along these lines:

2024/09/26 08:27:46 [warn] 26#26: *2525 a client request body is buffered to a temporary file /tmp/nginx/client-body/0000000224, client: 10.10.32.4, server: SERVER_URL, request: "POST /api/events/raw HTTP/1.1", host: "SERVER_URL"

But the assumption was nginx would handle itself. Maybe an incorrect assumption?

[nblumhardt]

It looks like the default on nginx could be 1 MiB:

https://stackoverflow.com/questions/28476643/default-nginx-client-max-body-size

Hopefully this will sort it out for you; fingers crossed!

[PhilBroderickCrezco]

Thanks @nblumhardt. We're not actually getting errors - just the warning log. I had a play about with proxy-body-size (cannot use a ConfigMap as we're using the application routing add-on for AKS: https://learn.microsoft.com/en-us/azure/aks/app-routing#limitations) either way and it doesn't seem to be changing things.

However, we're seeing success by manually setting the batchPostingLimit for the new Seq instance sink in our Serilog config. This has made the log ingestion a lot more stable.