Support JetStream Advisories Metrics

[ripienaar]

We need to gather JetStream stats:

• API Audit Advisories which should include which API calls yielded an error
• Ack samples
• Delivery Exceeded Advisories
• Delivery terminated advisories Add additional JetStream advisories #32
• Consumer and Stream actions Add additional JetStream advisories #32
• Snapshots and Restores Add additional JetStream advisories #32
• Dashboard => Add Dashboard for "per-Account" JetStream Advisories  #78

[ripienaar]

API audit stuff via #26

# HELP nats_survey_jetstream_advisory_count Number of JetStream Advisory listeners that are running
# TYPE nats_survey_jetstream_advisory_count gauge
nats_survey_jetstream_advisory_count 1
# HELP nats_survey_jetstream_api_audit JetStream API access audit events
# TYPE nats_survey_jetstream_api_audit counter
nats_survey_jetstream_api_audit{account="global",server="NBERSVTO4W7JEIXVJUSH4C6TMOUCBRIOJ6LRG65KJ42OF6UL43UVMMGS",subject="$JS.STREAM.LIST"} 1
nats_survey_jetstream_api_audit{account="global",server="NBERSVTO4W7JEIXVJUSH4C6TMOUCBRIOJ6LRG65KJ42OF6UL43UVMMGS",subject="$JS.STREAM.ORDERS.CREATE"} 1
# HELP nats_survey_jetstream_api_errors JetStream API Errors Count
# TYPE nats_survey_jetstream_api_errors counter
nats_survey_jetstream_api_errors{account="global",server="NBERSVTO4W7JEIXVJUSH4C6TMOUCBRIOJ6LRG65KJ42OF6UL43UVMMGS",subject="$JS.STREAM.ORDERS.CREATE"} 1

[ripienaar]

Delivery exceeded via 52e97c8

# HELP nats_survey_jetstream_delivery_exceeded_count Advisories about JetStream Consumer Delivery Exceeded events
# TYPE nats_survey_jetstream_delivery_exceeded_count counter
nats_survey_jetstream_delivery_exceeded_count{account="global",consumer="SURVEYOR",stream="ORDERS"} 10

[ripienaar]

And acks in e3dbb62

# HELP nats_survey_jetstream_acknowledgement_deliveries How many times messages took to be delivered and Acknowledged
# TYPE nats_survey_jetstream_acknowledgement_deliveries counter
nats_survey_jetstream_acknowledgement_deliveries{account="global",consumer="SURVEYOR",stream="ORDERS"} 12
# HELP nats_survey_jetstream_acknowledgement_duration How long an Acknowledged message took to be Acknowledged
# TYPE nats_survey_jetstream_acknowledgement_duration histogram
nats_survey_jetstream_acknowledgement_duration_bucket{account="global",consumer="SURVEYOR",stream="ORDERS",le="0.005"} 6
nats_survey_jetstream_acknowledgement_duration_bucket{account="global",consumer="SURVEYOR",stream="ORDERS",le="0.01"} 6
nats_survey_jetstream_acknowledgement_duration_bucket{account="global",consumer="SURVEYOR",stream="ORDERS",le="0.025"} 6
nats_survey_jetstream_acknowledgement_duration_bucket{account="global",consumer="SURVEYOR",stream="ORDERS",le="0.05"} 6
nats_survey_jetstream_acknowledgement_duration_bucket{account="global",consumer="SURVEYOR",stream="ORDERS",le="0.1"} 6
nats_survey_jetstream_acknowledgement_duration_bucket{account="global",consumer="SURVEYOR",stream="ORDERS",le="0.25"} 6
nats_survey_jetstream_acknowledgement_duration_bucket{account="global",consumer="SURVEYOR",stream="ORDERS",le="0.5"} 6
nats_survey_jetstream_acknowledgement_duration_bucket{account="global",consumer="SURVEYOR",stream="ORDERS",le="1"} 6
nats_survey_jetstream_acknowledgement_duration_bucket{account="global",consumer="SURVEYOR",stream="ORDERS",le="2.5"} 6
nats_survey_jetstream_acknowledgement_duration_bucket{account="global",consumer="SURVEYOR",stream="ORDERS",le="5"} 6
nats_survey_jetstream_acknowledgement_duration_bucket{account="global",consumer="SURVEYOR",stream="ORDERS",le="10"} 6
nats_survey_jetstream_acknowledgement_duration_bucket{account="global",consumer="SURVEYOR",stream="ORDERS",le="+Inf"} 6
nats_survey_jetstream_acknowledgement_duration_sum{account="global",consumer="SURVEYOR",stream="ORDERS"} 0.011403
nats_survey_jetstream_acknowledgement_duration_count{account="global",consumer="SURVEYOR",stream="ORDERS"} 6

[bwerthmann]

@ColinSullivan1 my understanding is that the initial scope from @ripienaar for this issue is per-account JetStream API Advisories. I've updated the issue title to reflect this.

AFAIK, the only outstanding item is a dashboard.

The work I've done recently related to system/cluster level jetstream metrics is linked to #47.