Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

One time batch landings #37

Merged
merged 56 commits into from
Oct 2, 2022
Merged

One time batch landings #37

merged 56 commits into from
Oct 2, 2022

Conversation

rivernews
Copy link
Owner

rivernews added 30 commits August 21, 2022 23:13
@rivernews
temp store all
e5b9214
@rivernews
remove go_poc
5333981
@rivernews
upgrade so project runs on M1
89c666c
@rivernews
Try S3 notification
5c60040
@rivernews
Fix prefix to include newssite alias
d5d8186
@rivernews
Fix aws lambda PathError issue
fa266bf
@rivernews
Merge pull request #27 from rivernews/temp--s3-notif
24a13ad 
Try S3 notification
@rivernews
Save to metadata.json complete
0f7ab88
@rivernews
add untitled stories in metadata.json
b6851c5
@rivernews
rename stories function to landing_metadata
6c6e250
@rivernews
rename batch stories fetch tf to metadata
512f729
@rivernews
Improved metadata access s3 event
9eca283
@rivernews
Metadata.json trigger computing env
a62c33d
@rivernews
read parse metadata.json
bda216e
@rivernews
fetch a story POC
e29fdc8 
#24
@rivernews
Sfn map parallism POC
d2b025e 
#24
@rivernews
randomize requests
ba830cd
@rivernews
Merge pull request #29 from rivernews/fetch-stories
9ad201f 
Fetch stories (S3 event driven)
@rivernews
Refactor to allow individual tf modules
98b3bce 
address #25 (comment)
@rivernews
Merge pull request #30 from rivernews/individual-modules
8e8ef7f 
Refactor to allow individual tf modules

Tested deploy, pipeline worked as expected
@rivernews
scaffold table
578b06f
@rivernews
draft table design
58af865
@rivernews
create table
a0c103a
@rivernews
Merge pull request #31 from rivernews/provision-table
42d75ea 
Provision table
@rivernews
Merge pull request #32 from rivernews/table_for_metadata
7a2e9b5 
Table for metadata
@rivernews
Draining mechanism draft - identify all TODOs
eb61eef 
#25 (comment)
@rivernews
Draft for put landing page; identified TODOs
ec37c70 
Issue: #25
@rivernews
Complete tf surgery; Identify all TODOs in golang
d816fe6 
For #25
@rivernews
fix compile error; progress in metadata cronjob add query
2c8f841
@rivernews
Ready to test
6ad69b6
@rivernews
create reusable lambda module; optimize package size
3977565 
#25 (comment)
@rivernews
Fix golang build path
4b75811
@rivernews
Refactor to use our custom lambda module
68b4963
@rivernews
add landing s3 trigger
2643218
@rivernews
rm golang module stories that are renamed
17e652a
@rivernews
Fix env var
5df3517
@rivernews
Fix permission for PutItem move from landing to s3 trigger
1b79829
@rivernews
Fix metadata s3 trigger not fired
31abe5a
@rivernews
Fix s3 trigger not working - S3 notification can only have one resource
0917331
@rivernews
Make it easier to test
0258564
@rivernews
prod grade setting enabled
f19edcb
@rivernews
In Sfn pin lambda version, so rolling deploy works better for lambda
ef44117
@rivernews
Display sfn map result / target stories count info in finalizer
b5d71d3
@rivernews
stop landing s3 trigger from sending slack logs
9e5bbd6 
Fixes #40
@rivernews
Let Sfn pin lambda version
862ee64 
Fixes #39
@rivernews
improve log for metadata trigger
a8b7027
@rivernews
improve cronjob log
6d4a032
@rivernews
log cronjob event for better understanding of how it get triggered
f999c13
@rivernews
Disable cronjob to better debug
7ccc4d4 
Fixes #43
@rivernews
workaround to scale up our Sfn pipeline
4bd649f 
Fix #44
@rivernews
improve log for landing S3 trigger
f475441
@rivernews
re-enable prod config plus cronjob
4df9c3c
@rivernews rivernews mentioned this pull request Oct 2, 2022
Closed
@rivernews
Copy link
Owner Author

We'll resolve conflicts and merge with dev in #46

@rivernews rivernews closed this Oct 2, 2022
@rivernews rivernews reopened this Oct 2, 2022
@rivernews rivernews changed the base branch from dev to main October 2, 2022 08:05
@rivernews rivernews mentioned this pull request Oct 2, 2022
Closed
@rivernews rivernews changed the base branch from main to dev October 2, 2022 08:10
@rivernews rivernews merged commit 4f7720b into dev Oct 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@rivernews