NEOS-1438 Destination option to skip foreign key constraint errors in sync

[alishakawaguchi]

No description provided.

[linear]

NEOS-1438 add job destination to continue on fk constraint error

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
neosync-docs	⬜️ Ignored (Inspect)	Visit Preview		Sep 11, 2024 8:28pm

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Sep 11, 2024, 8:28 PM

[codecov]

Codecov Report

Attention: Patch coverage is 34.18803% with 77 lines in your changes missing coverage. Please review.

Project coverage is 36.58%. Comparing base (aa405fe) to head (7fb07b6).
Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
...ws/datasync/activities/gen-benthos-configs/sync.go	0.00%	26 Missing ⚠️
worker/pkg/benthos/sql/output_sql_insert.go	41.86%	24 Missing and 1 partial ⚠️
backend/sql/postgresql/models/models.go	36.84%	12 Missing ⚠️
worker/pkg/benthos/utils.go	0.00%	9 Missing ⚠️
worker/pkg/benthos/sql/output_sql_update.go	72.22%	4 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2665      +/-   ##
==========================================
+ Coverage   36.48%   36.58%   +0.09%     
==========================================
  Files         272      273       +1     
  Lines       25271    25441     +170     
==========================================
+ Hits         9221     9307      +86     
- Misses      14795    14872      +77     
- Partials     1255     1262       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[nickzelei]

LGTM, left a minor comment about a follow up we should do.

[nickzelei]

fmt.Sprintf("%s.%s", s.schema, s.table)
I know this gets fed into goqu and properly escaped, but what if the schema or table has a period in it? I'm guessing it doesnt know what to do. Anyways, idk if we are doing this anywhere but now that I'm hyper aware of special chars thought i'd point this out...

[nickzelei]

Maybe in a follow up we should update those BuildInsertQuery and BuildUpdateQuery functions to call out schema and table as specific parameters so that we can use goqu to build them like goqu.T(table).Schema(schema) so that it can handle the escaping for us.