Support for events generated by GCP Pub/Sub and GCS (via Pub/Sub or webhooks)

[vicaire]

We would like support for events triggered by GCP Pub/Sub and GCS to be added.

[magaldima]

Okay. Can you elaborate on how these signals would be configured? Do we need to add to the existing SensorSpec API? Webhooks and various Pub/Sub (streaming platforms) are already supported. Does GCP have it's own such platform?

[vicaire]

Hi magaldima@, here are some more details:

• Yes, GCP has its own pub/sub platform: https://cloud.google.com/pubsub/
• For GCS, the webhook support looks great! GCS can also forward notifications to Pub/Sub. Pub/Sub is probably a more important use case as it has the advantage of not requiring the user to expose a public IP address to receive the webhook notification.
• What is missing is then the ability to extract information from the Pub/Sub message or Webhook notification (the URI of the created object) and pass this as a parameter to a workflow. If the issue "Add filters for signals #26" solves the filtering problem in a generic enough way, that could be a solution to this issue.

For reference, the format of the GCS notification is straightforward json (https://cloud.google.com/storage/docs/object-change-notification).

[magaldima]

@vicaire take a look at: #49 . This is an effort to make building signals easier especially for stream (pub/sub) services. I think your second point may be able to be solved through a custom GCP Pub/Sub signal microservice. And yes, your third point essentially boils down to filters.

Let me know if you have time to build out a custom GCP Pub/Sub signal. I would be more than happy to help out.

[magaldima]

@vicaire since latest 0.5-beta1 release, you can now extract information/fields from a signal's event payload and pass it to a workflow. See the quickstart guide on an easy example to run.

As per a custom signal for the GCP pub/sub platform, I'll add this to the v1.0.0 milestone and work on this when I get some free time.

[vicaire]

Thanks magaldima@, that looks great.

[VaibhavPage]

#176

[vicaire]

Thank you! This is great.