Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Landing S3 trigger disable slack log & only log to cloudWatch #40

Closed
rivernews opened this issue Sep 30, 2022 · 1 comment
Closed

Landing S3 trigger disable slack log & only log to cloudWatch #40

rivernews opened this issue Sep 30, 2022 · 1 comment

Comments

@rivernews
Copy link
Owner

No description provided.

@rivernews
Copy link
Owner Author

rivernews commented Sep 30, 2022

In slack confirmed no new message showing up! While CloudWatch still logs:

image

rivernews added a commit that referenced this issue Sep 30, 2022
rivernews added a commit that referenced this issue Oct 2, 2022
* temp store all

* remove go_poc

* upgrade so project runs on M1

* Try S3 notification

* Fix prefix to include newssite alias

* Fix aws lambda PathError issue

* Save to metadata.json complete

* add untitled stories in metadata.json

* rename stories function to landing_metadata

* rename batch stories fetch tf to metadata

* Improved metadata access s3 event

* Metadata.json trigger computing env

* read parse metadata.json

* fetch a story POC
#24

* Sfn map parallism POC
#24

* randomize requests

* Refactor to allow individual tf modules
address #25 (comment)

* scaffold table

* draft table design

* create table

* Draining mechanism draft - identify all TODOs
#25 (comment)

* Draft for put landing page; identified TODOs
Issue: #25

* Complete tf surgery; Identify all TODOs in golang
For #25

* fix compile error; progress in metadata cronjob add query

* Ready to test

* Fix db field first char not lowercase
Tracked by #25 (comment)

* Fix permission of db index, S3 pull
Tracked by #25 (comment)

* All tests complete
Tracked by #25 (comment)

* Move landing PutItem out to s3 trigger lambda; ready for S3 batch move

* create reusable lambda module; optimize package size
#25 (comment)

* Fix golang build path

* Refactor to use our custom lambda module

* add landing s3 trigger

* rm golang module stories that are renamed

* Fix env var

* Fix permission for PutItem move from landing to s3 trigger

* Fix metadata s3 trigger not fired

* Fix s3 trigger not working - S3 notification can only have one resource

* Make it easier to test

* prod grade setting enabled

* In Sfn pin lambda version, so rolling deploy works better for lambda

* Display sfn map result / target stories count info in finalizer

* stop landing s3 trigger from sending slack logs
Fixes #40

* Let Sfn pin lambda version
Fixes #39

* improve log for metadata trigger

* improve cronjob log

* log cronjob event for better understanding of how it get triggered

* Disable cronjob to better debug
Fixes #43

* workaround to scale up our Sfn pipeline
Fix #44

* improve log for landing S3 trigger

* re-enable prod config plus cronjob
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant