Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unexpectedly extremely many logs during eventbus restart #3497

Open
ognjen-it opened this issue Feb 26, 2025 · 7 comments
Open

Unexpectedly extremely many logs during eventbus restart #3497

ognjen-it opened this issue Feb 26, 2025 · 7 comments
Labels
bug Something isn't working

Comments

@ognjen-it
Copy link

ognjen-it commented Feb 26, 2025 •
edited
Loading

Describe the bug
After restarting JetStream (EventBus), Argo Events (Sensor) starts producing an excessive number of errors—over 200,000 errors in a few minutes. This significantly impacts system performance and stability. The issue is consistently reproducible, and logs indicate a flood of reconnection or message processing errors.

To Reproduce
Steps to reproduce the behavior:

  1. Deploy JetStream EventBus
  2. Deploy Sensor that will be connected to that eventbus
  3. Update event bus with any information like tolerations or affinity (then evetnbus will start rollout one by one pod)
  4. Check sensor logs or check log volume on grafana with loki :)

Expected behavior
Argo Events should gracefully handle JetStream restarts without producing an overwhelming number of errors. It should retry connections in a controlled manner rather than flooding logs and potentially overloading the system.

Screenshots
Log volume of one sensor:
Image

Example of log:

{"level":"error","ts":1740569807.8508778,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.850854739 +0000 UTC m=+435273.783129224","sensorName":"my-sensor-namegsm","triggerName":"my-sensor-namegsm","sensorName":"my-sensor-namegsm","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}

Environment (please complete the following information):

  • Kubernetes: v1.27.7
  • Argo: 3.6.4
  • Argo Events: 1.9.5
  • JetStream version: 2.10.10

Additional context
Honestly, I wouldn't even have noticed if I didn't have a few dozen sensors and a Loki that had to collect several million logs in those few minutes.

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.
@ognjen-it ognjen-it added the bug Something isn't working label Feb 26, 2025
@ognjen-it
Copy link
Author

ognjen-it commented Feb 26, 2025 •
edited
Loading

It looks like you have fixed it (with adding 10 sec) here:
https://github.com/argoproj/argo-events/blob/release-1.9/pkg/eventbus/jetstream/sensor/trigger_conn.go#L200
But somehow it's not working...

EDIT:
Last logs before connect:

Details 

Common labels: {"app":"my-sensor-name-sensor-hf2hm","container":"main","filename":"/var/log/pods/argo_my-sensor-name-sensor-hf2hm-ggxcv_976736f1-e2d4-46f1-b980-ec687b2e29ec/main/0.log","job":"argo/my-sensor-name-sensor-hf2hm","namespace":"argo","node_name":"aks-notaint-31406283-vmss000007","pod":"my-sensor-name-sensor-hf2hm-ggxcv","service_name":"my-sensor-name-sensor-hf2hm","stream":"stdout"}
Line limit: "100 reached, received logs cover 21.64% (1min 16sec) of your selected time range (5min 51sec)"
Total bytes processed: "878  kB"


2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.86908,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869069961 +0000 UTC m=+435273.801344435","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8690996,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869088095 +0000 UTC m=+435273.801362579","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8691208,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869109866 +0000 UTC m=+435273.801384340","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8691394,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869129413 +0000 UTC m=+435273.801403887","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.869159,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869147617 +0000 UTC m=+435273.801422101","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8691795,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869168757 +0000 UTC m=+435273.801443281","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8692038,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869193023 +0000 UTC m=+435273.801467507","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8692327,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869212379 +0000 UTC m=+435273.801486864","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8692567,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869245492 +0000 UTC m=+435273.801519966","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8692787,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869267102 +0000 UTC m=+435273.801541587","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8693044,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869287451 +0000 UTC m=+435273.801561935","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.869327,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869315514 +0000 UTC m=+435273.801589998","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8693507,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.86933988 +0000 UTC m=+435273.801614354","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8693738,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869362803 +0000 UTC m=+435273.801637287","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8694034,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869391758 +0000 UTC m=+435273.801666242","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.86943,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869418348 +0000 UTC m=+435273.801692832","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8694556,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869443836 +0000 UTC m=+435273.801718370","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"info","ts":1740569807.8694677,"logger":"argo-events.sensor","caller":"base/jetstream.go:112","msg":"Connected to NATS Jetstream server.","sensorName":"my-sensor-name"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.86948,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869468202 +0000 UTC m=+435273.801742676","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8694997,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869489051 +0000 UTC m=+435273.801763525","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8695257,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869512525 +0000 UTC m=+435273.801787010","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.869548,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869537453 +0000 UTC m=+435273.801811937","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.869572,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869560656 +0000 UTC m=+435273.801835141","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.869594,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.86958353 +0000 UTC m=+435273.801858004","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.869618,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869605812 +0000 UTC m=+435273.801880286","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8696377,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869627202 +0000 UTC m=+435273.801901686","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8696575,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869645927 +0000 UTC m=+435273.801920421","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8697517,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869674651 +0000 UTC m=+435273.801949125","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8697984,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869781423 +0000 UTC m=+435273.802055907","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.869823,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869811961 +0000 UTC m=+435273.802086435","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8698459,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869835365 +0000 UTC m=+435273.802109849","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8698702,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869858478 +0000 UTC m=+435273.802132962","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8698936,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869883035 +0000 UTC m=+435273.802157518","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8699157,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869903643 +0000 UTC m=+435273.802178128","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.869	{"level":"error","ts":1740569807.8699343,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869924182 +0000 UTC m=+435273.802198666","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8699522,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869942507 +0000 UTC m=+435273.802216981","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8699718,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869961583 +0000 UTC m=+435273.802236067","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.86999,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869979717 +0000 UTC m=+435273.802254201","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8700097,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.869998703 +0000 UTC m=+435273.802273187","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8700302,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870019602 +0000 UTC m=+435273.802294086","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8700483,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870037726 +0000 UTC m=+435273.802312210","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.870072,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.87006118 +0000 UTC m=+435273.802335665","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.870097,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870085927 +0000 UTC m=+435273.802360401","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8701272,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870108931 +0000 UTC m=+435273.802383415","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8701491,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870138937 +0000 UTC m=+435273.802413421","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8701668,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870156901 +0000 UTC m=+435273.802431385","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.870186,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870175797 +0000 UTC m=+435273.802450271","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8702097,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870199101 +0000 UTC m=+435273.802473585","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8702326,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870221673 +0000 UTC m=+435273.802496157","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8702564,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870245258 +0000 UTC m=+435273.802519742","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8702793,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870269093 +0000 UTC m=+435273.802543567","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8703022,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870291565 +0000 UTC m=+435273.802566039","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8703232,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870312034 +0000 UTC m=+435273.802586508","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8703427,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870331901 +0000 UTC m=+435273.802606385","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8703604,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870350226 +0000 UTC m=+435273.802624710","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8703797,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870369943 +0000 UTC m=+435273.802644417","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.870398,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870388147 +0000 UTC m=+435273.802662621","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.870	{"level":"error","ts":1740569807.8704176,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870406472 +0000 UTC m=+435273.802680946","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8704376,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870427321 +0000 UTC m=+435273.802701805","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8704557,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870445916 +0000 UTC m=+435273.802720390","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8704793,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870464912 +0000 UTC m=+435273.802739386","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8704975,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870487525 +0000 UTC m=+435273.802762009","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8705168,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870506541 +0000 UTC m=+435273.802781035","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8705409,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870529444 +0000 UTC m=+435273.802803928","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.870559,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.87054858 +0000 UTC m=+435273.802823064","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.870582,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870571353 +0000 UTC m=+435273.802845827","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8706017,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.8705913 +0000 UTC m=+435273.802865785","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8706245,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870613963 +0000 UTC m=+435273.802888447","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.870647,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870636636 +0000 UTC m=+435273.802911120","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8706698,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870659389 +0000 UTC m=+435273.802933873","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8706882,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870677984 +0000 UTC m=+435273.802952458","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8707108,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870700056 +0000 UTC m=+435273.802974540","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.870731,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870720745 +0000 UTC m=+435273.802995219","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8707535,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870743337 +0000 UTC m=+435273.803017821","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.870775,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870764818 +0000 UTC m=+435273.803039292","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8707974,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.87078698 +0000 UTC m=+435273.803061454","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8708186,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870807468 +0000 UTC m=+435273.803081952","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8708408,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870830111 +0000 UTC m=+435273.803104585","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8708632,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870852624 +0000 UTC m=+435273.803127098","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8708832,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870873142 +0000 UTC m=+435273.803147626","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8709016,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870890936 +0000 UTC m=+435273.803165420","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8709192,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.87090902 +0000 UTC m=+435273.803183494","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.87094,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870928957 +0000 UTC m=+435273.803203442","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8709629,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870952742 +0000 UTC m=+435273.803227226","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8709936,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.870975144 +0000 UTC m=+435273.803249629","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8710206,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.871005833 +0000 UTC m=+435273.803280316","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.871046,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.871033555 +0000 UTC m=+435273.803308029","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8710692,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.871058792 +0000 UTC m=+435273.803333276","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8710923,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.871081525 +0000 UTC m=+435273.803356009","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8711126,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.871102425 +0000 UTC m=+435273.803376899","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"info","ts":1740569807.871107,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:78","msg":"Successfully located K/V store for sensor my-sensor-name","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8711364,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.871125889 +0000 UTC m=+435273.803400363","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"info","ts":1740569807.8711329,"logger":"argo-events.sensor","caller":"sensors/listener.go:333","msg":"reconnected to EventBus.","sensorName":"my-sensor-name","triggerName":"my-sensor-name","connection":"JetstreamTriggerConn{Sensor:my-sensor-name,Trigger:my-sensor-name}"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8711588,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.871148451 +0000 UTC m=+435273.803422936","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"info","ts":1740569807.871161,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:171","msg":"closing connection...","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name"}
2025-02-26 12:36:47.871	{"level":"info","ts":1740569807.8711789,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:249","msg":"shutting down processMsgs routine","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name"}
2025-02-26 12:36:47.871	{"level":"error","ts":1740569807.8711805,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:202","msg":"failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription\nnats: subscription closed, previousErr=nats: invalid subscription\nnats: subscription closed, previousErrTime=2025-02-26 11:36:47.871170343 +0000 UTC m=+435273.803444827","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name","stacktrace":"github.com/argoproj/argo-events/pkg/eventbus/jetstream/sensor.(*JetstreamTriggerConn).pullSubscribe\n\t/home/runner/work/argo-events/argo-events/pkg/eventbus/jetstream/sensor/trigger_conn.go:202"}
2025-02-26 12:36:47.871	{"level":"info","ts":1740569807.8711991,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:213","msg":"exiting pullSubscribe() for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.vg8Q0Diy7UIus5nuVGjAcr.* Queue: jsi:0xc0015a2000 delivered:183 max:0 conn:0xc001501508 mcb:<nil> mch:<nil> errCh:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] permissionsErr:<nil> typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:2 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name"}
2025-02-26 12:36:49.872	{"level":"info","ts":1740569809.872084,"logger":"argo-events.sensor","caller":"sensors/listener.go:302","msg":"started subscribing to events for trigger my-sensor-name with client connection JetstreamTriggerConn{Sensor:my-sensor-name,Trigger:my-sensor-name}","sensorName":"my-sensor-name","triggerName":"my-sensor-name"}
2025-02-26 12:36:49.872	{"level":"info","ts":1740569809.8721218,"logger":"argo-events.sensor","caller":"sensor/trigger_conn.go:142","msg":"Subscribing to subject default.operational-calendar-event.my-sensor-name with durable name group-2310431384","sensorName":"my-sensor-name","triggerName":"my-sensor-name","sensorName":"my-sensor-name"}

@aaron-arellano
Copy link
Contributor

@ognjen-it . I just saw this issue as well with argo-events v1.9.2. Did you also see memory increase in one of the jetstream eventbus pods since the sensor connection was not up. There were memory spikes on our end with the eventbus and we are not sure they correlate fully but happened at the same time.

failed to fetch messages for subscription &{mu:{state:0 sema:0} sid:2 Subject:_INBOX.gFn0F3rKD1F1vmnjwrtziB.* Queue: jsi:0xc0010a8380 delivered:585 max:0 conn:0xc000e71800 mcb:<nil> mch:<nil> closed:true sc:false connClosed:true draining:false status:0 statListeners:map[] typ:1 pHead:<nil> pTail:<nil> pCond:<nil> pDone:<nil> pMsgs:0 pBytes:0 pMsgsMax:1 pBytesMax:0 pMsgsLimit:65536 pBytesLimit:67108864 dropped:0}, nats: invalid subscription
nats: subscription closed, previousErr=nats: invalid subscription
nats: subscription closed, previousErrTime=2025-03-04 00:52:51.916468352 +0000 UTC m=+132592.699168882

@la0bing
Copy link

la0bing commented Mar 4, 2025

facing this issue too on v1.9.2, this causes huge logs count spikes, seems to be fixed by restarting each sensors.

@ognjen-it
Copy link
Author

Hi @aaron-arellano, I face memory "issues", but that's not worrying... a few hundred megabytes. Do you have some graph/metrics?

Image

@la0bing I think that you don't need to restart to fix it, it will be automatically subscribed to nats jetstream, but logs that sensors have made are problem if you collect those logs..

@la0bing
Copy link

la0bing commented Mar 4, 2025

For me the logs never stop, and some sensors showing failed to subscribe to event bus, until I restarted all of them.

@ognjen-it
Copy link
Author

It's so strange. I will focus on solving this problem as soon as my work decreases a bit. But I don't grant I can resolve that :)

I faced the issue just like you when I migrated from nats to jetstream, I needed to restart all sensor deployments to catch new secret key for jetstream.
However, If someone face that same issue, here is the easiest way to restart all pods.

for i in `kubectl -n argo get deployments.apps -l controller=eventsource-controller | awk '{print $1}'`; do kubectl -n argo rollout restart deployment $i ; done
for i in `kubectl -n argo get deployments.apps -l controller=sensor-controller | awk '{print $1}'`; do kubectl -n argo rollout restart deployment $i ; done

@aaron-arellano
Copy link
Contributor

aaron-arellano commented Mar 4, 2025 •
edited
Loading

ok, thank you for confirming. our memory is low and I suspect this is a stale issue on our end cause a single event goes every couple of days like one event every 3-5 days instead of multiple events a day. With no events passing through the sensor the subscription is unable to re-subscribe.

Image Image

if you can get the fix in that would be great!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@la0bing @ognjen-it @aaron-arellano