Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(source): PostgreSQL/MySQL CDC source vs table relationship #2449

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

xiangjinwu
Copy link
Contributor

@xiangjinwu xiangjinwu commented Aug 12, 2024

Info

  • Description

    • Initially, CDC can only be created as table but not source.
    • After #1754, there are 2 options:
      • Create a table directly, same as before.
      • Create a shared source first, and then table based on the shared source.
    • After #2023, the first syntax is deprecated and the second syntax is recommended.
  • Fixes in this PR (using PostgreSQL as example but may apply to MySQL as well):

    • table or source is never the case. It is table(deprecated) or source + table (recommended).
    • Link to create table syntax reference is useless. That page does not contain FROM source_name TABLE syntax for shared PostgreSQL/MySQL CDC.
    • FROM source TABLE is updated to FROM source_name TABLE in the example, to keep consistent with CREATE SOURCE source_name above.
  • Notes

    • Once @StrikeW confirmed the content above is accurate, we may apply the same fix from PostgreSQL to MySQL, and from dev version to current or even older applicable versions.
  • Related code PR

    • [ Provide a link to the relevant code PR here, if applicable. ]
  • Related doc issue

    Resolves [ Provide a link to the relevant doc issue here, if applicable. ]

For reviewers

  • Preview

    • [ Paste the preview link to the updated page(s) here. Edit this item after the preview site is ready. To find the updated pages, scroll down to locate and open the Amplify preview link and select the dev version of the documentation. ]
  • Key points

    • [ Parts that may need revision or extra consideration. ]

Before merging

  • I have checked the doc site preview, and the updated parts look good.

  • I have acquired the approval from the owner (and optionally the reviewers) of the code PR and at least one tech writer (emile-00, hengm3467, & WanYixian).

@xiangjinwu
Copy link
Contributor Author

xiangjinwu commented Aug 12, 2024

Btw is the |table.name| Name of the table that you want to ingest data from. | parameter still useful? If the source is tied to an upstream table, there would be no point to share the source for multiple RisingWave tables, right?

Copy link

This pull request is automatically being deployed by Amplify Hosting (learn more).

Access this pull request here: https://pr-2449.d2fbku9n2b6wde.amplifyapp.com

Copy link
Contributor

@StrikeW StrikeW left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix.

Copy link
Member

@fuyufjh fuyufjh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM


### Syntax

Syntax for creating a CDC source.
Syntax for creating a shared CDC source.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better to clarify the logic first: need to create a source before creating table

Suggested change
Syntax for creating a shared CDC source.
Before creating a table, you need to first create a source to the upstream MySQL database using the following syntax. This source can be shared by multiple tables from the same MySQL database.

@@ -162,7 +162,7 @@ CREATE SOURCE [ IF NOT EXISTS ] source_name WITH (
);
```

Syntax for creating a CDC table. Note that a primary key is required and must be consistent with the upstream table.
Syntax for creating a CDC table based on the shared source. Note that a primary key is required and must be consistent with the upstream table.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better use a complete sentence.

Suggested change
Syntax for creating a CDC table based on the shared source. Note that a primary key is required and must be consistent with the upstream table.
Below is the syntax for creating a CDC table based on the shared source. Note that a primary key is required and must be consistent with the upstream table.

@@ -151,11 +151,11 @@ If you are running RisingWave locally from binaries and intend to use the native

## Create a table using the native CDC connector

To ensure all data changes are captured, you must create a table or source and specify primary keys. See the [`CREATE TABLE`](/sql/commands/sql-create-table.md) command for more details.
To ensure all data changes are captured, you must create a shared source first and create tables based on that source.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's ok as of now in this PR, but I feel it would be more useful if we have a topic or section about "shared sources" that we can now point to. There are a few RisingWave-specific concepts that we may want to elaborate somewhere. In addition, the "Sources" term in Glossary should be updated. cc: @st1page @WanYixian

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. The shared source is still in development by @xxchan and it is expected to release in v2.1. I think we will add the doc that time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants