Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logs with category "ApplicationGatewayFirewallLog" does not have proper category label in Loki #14176

Open
Kapsztajn opened this issue Sep 19, 2024 · 1 comment

Comments

@Kapsztajn
Copy link

Kapsztajn commented Sep 19, 2024

Describe the bug
This bug is similar to issue in OpenTelemetry Collector: open-telemetry/opentelemetry-collector-contrib#28806
Logs which are forwarder to Loki from Event Hub through Promtail does not have any category for ApplicationGatewayFirewallLog even though it is visible in Event Hub message.

{"records": [{ "timeStamp": "2024-09-18T00:45:09+00:00", "resourceId": "/SUBSCRIPTIONS/hidden/RESOURCEGROUPS/hidden/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/hidden", "operationName": "ApplicationGatewayFirewall", "category": "ApplicationGatewayFirewallLog"

It is probably happening due to missing time field and relabel not properly working:
https://github.com/grafana/agent/pull/3412/files#diff-f7b899798d7f7b22d9a64584a767fa07952fa917226e312ad088de922ea06e71R37
Only timeStamp field is available for ApplicationGatewayFirewallLog.

Image

Normal message from Application Gateway with proper category relabel looks like this:

{"records": [{ "timeStamp": "2024-09-18T00:45:20+00:00", "time": "2024-09-18T00:45:20+00:00", "resourceId": "/SUBSCRIPTIONS/hidden/RESOURCEGROUPS/hidden/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/hidden", "listenerName": "hidden", "ruleName": "hidden", "backendPoolName": "hidden", "backendSettingName": "hidden", "operationName": "ApplicationGatewayAccess", "category": "ApplicationGatewayAccessLog"

To Reproduce
Steps to reproduce the behavior:

  1. Run Promtail with Azure Event Hub
  2. Generate some logs from Application Gateway WAF
  3. Check if category is missing even though logs are visible in Loki/Grafana

Expected behavior
Proper category should be assigned/relabeled by Promtail in Loki for ApplicationGatewayFirewallLog even when only timeStamp filed is available in message.

Environment:

  • Infrastructure: Azure Container Apps
  • Deployment tool: Docker

Screenshots, Promtail config, or terminal output
Promtail config below:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

clients:
  - url: https://hidden.grafana.net/loki/api/v1/push
    basic_auth: 
      username: hidden
      password: hidden

scrape_configs:
  - job_name: test
    azure_event_hubs:
      fully_qualified_namespace: hidden.servicebus.windows.net:9093
      connection_string: 'Endpoint=sb://hidden.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=hidden'
      event_hubs: 
        - hidden
      labels:
        job: azure_event_hub
    relabel_configs:
      - action: replace
        source_labels:
          - __azure_event_hubs_category
        target_label: category
@Kapsztajn
Copy link
Author

@andriikushch tagging you here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant