[Feature][Catalog] Doris Catalog #5175

gnehil · 2023-07-28T09:26:46Z

Purpose of this pull request

config example:

sink {
  Doris {
    source_table_name = "jdbc"
    catalog {
      factory = "Doris"
    }
    fenodes = "127.0.0.1:8030"
    query-port = "9030"
    username = "root"
    password = ""
    table.identifier = "test_db.test_tbl"
    savemode = "KEEP_SCHEMA_AND_DATA"
  }
}

create table action is only triggered when savemode = "KEEP_SCHEMA_AND_DATA"

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

Hisoka-X · 2023-08-23T07:00:19Z

create {
    distribution-columns = "id"
    distribution-buckect = "10"
    properties {
        replication_allocation = "tag.location.default: 1"
    }
}

I think we need to basically unify the definition methods of different connectors for table create. Can you implement Doris's automatic table create function definition according to the table create template provided by StarRocks https://seatunnel.apache.org/docs/2.3.3/connector-v2/sink/StarRocks#save_mode_create_template ? Please.

...onnector-doris/src/main/java/org/apache/seatunnel/connectors/doris/catalog/DorisCatalog.java

Hisoka-X · 2023-08-23T09:10:16Z

Could you add some e2e for this new feature? Please.

gnehil · 2023-09-14T03:18:11Z

Could you add some e2e for this new feature? Please.

done

...onnector-doris/src/main/java/org/apache/seatunnel/connectors/doris/catalog/DorisCatalog.java

...nnector-doris/src/main/java/org/apache/seatunnel/connectors/doris/util/DorisCatalogUtil.java

...nnector-doris-e2e/src/test/java/org/apache/seatunnel/e2e/connector/doris/DorisCatalogIT.java

EricJoy2048 · 2023-09-14T11:25:56Z

Please fix code style by run sh tools/spotless/pre-commit.sh

gnehil · 2023-09-15T07:52:25Z

Please fix code style by run sh tools/spotless/pre-commit.sh

done

Hisoka-X · 2023-10-12T07:04:02Z

Could you update your fork repository? The CI didn't start because your dev branch no up to date.

gnehil · 2023-10-12T07:50:59Z

Could you update your fork repository? The CI didn't start because your dev branch no up to date.

It's done, please review again

Hisoka-X · 2023-10-12T07:53:19Z

Waiting CI pass. https://github.com/gnehil/seatunnel/actions/runs/6492799003/job/17632437784

Hisoka-X · 2023-10-12T12:19:45Z

The CI seem like failed. Could you investigate it?

...ris/src/test/java/org/apache/seatunnel/connectors/doris/catalog/DorisCatalogFactoryTest.java

...ctor-doris/src/test/java/org/apache/seatunnel/connectors/doris/catalog/DorisCatalogTest.java

EricJoy2048 · 2023-10-23T07:18:20Z

Is there any latest progress on this PR?

…AND_DATA

* doris catalog * fix create table * add license header * add license header * code style * do not exec create table when save mode is null * do not exec create table when save mode is not equals to KEEP_SCHEMA_AND_DATA * update create table * add doris catalog e2e test * add license header * add license header and fix style * static string sql template * spotless check * resolve api change * disable ut * merge test * code style * test jdbc driver * add be alive check * mod container * fix container * support drop table and fix create template * resolve conflicts --------- Co-authored-by: gnehil <adamlee489@gamil.com> Co-authored-by: gnehil <gnehil489@github>

gnehil changed the title ~~[feature] Doris Catalog~~ [Feature][Catalog] Doris Catalog Jul 28, 2023

gnehil marked this pull request as ready for review July 31, 2023 02:06

gnehil force-pushed the dev-doris-catalog branch from a769800 to 5de9966 Compare August 15, 2023 03:26

Hisoka-X added this to the 2.3.4 milestone Aug 23, 2023

Hisoka-X reviewed Aug 23, 2023

View reviewed changes

gnehil force-pushed the dev-doris-catalog branch from 34acbbe to 8619be2 Compare September 13, 2023 07:40

Hisoka-X reviewed Sep 14, 2023

View reviewed changes

...onnector-doris/src/main/java/org/apache/seatunnel/connectors/doris/catalog/DorisCatalog.java Outdated Show resolved Hide resolved

...nnector-doris/src/main/java/org/apache/seatunnel/connectors/doris/util/DorisCatalogUtil.java Outdated Show resolved Hide resolved

Hisoka-X reviewed Sep 14, 2023

View reviewed changes

...nnector-doris-e2e/src/test/java/org/apache/seatunnel/e2e/connector/doris/DorisCatalogIT.java Outdated Show resolved Hide resolved

EricJoy2048 closed this Oct 12, 2023

EricJoy2048 reopened this Oct 12, 2023

gnehil force-pushed the dev-doris-catalog branch from 0d6f079 to d367a2b Compare October 12, 2023 07:31

gnehil closed this Oct 12, 2023

gnehil reopened this Oct 12, 2023

Hisoka-X reviewed Oct 17, 2023

View reviewed changes

...ris/src/test/java/org/apache/seatunnel/connectors/doris/catalog/DorisCatalogFactoryTest.java Outdated Show resolved Hide resolved

...ctor-doris/src/test/java/org/apache/seatunnel/connectors/doris/catalog/DorisCatalogTest.java Outdated Show resolved Hide resolved

gnehil force-pushed the dev-doris-catalog branch from 3a07bb2 to c1b3a6b Compare November 2, 2023 09:43

gnehil force-pushed the dev-doris-catalog branch 2 times, most recently from 968eafc to fe56ae7 Compare November 15, 2023 03:30

gnehil force-pushed the dev-doris-catalog branch from fe56ae7 to a0d9d21 Compare November 21, 2023 02:50

gnehil added 2 commits November 30, 2023 15:04

doris catalog

9322661

fix create table

b8c013f

gnehil and others added 18 commits November 30, 2023 15:04

code style

0a701a9

do not exec create table when save mode is null

d3f519b

do not exec create table when save mode is not equals to KEEP_SCHEMA_…

74192f3

…AND_DATA

update create table

2631b81

add doris catalog e2e test

482d9ea

add license header

2e4e479

add license header and fix style

1eda1bb

static string sql template

fbbe5cc

spotless check

3b64118

resolve api change

8bddfc2

disable ut

fdd0bf3

merge test

28c0ad4

code style

0f8372a

test jdbc driver

f45958f

add be alive check

dd95a97

mod container

09d4eb3

fix container

66a5533

support drop table and fix create template

5501097

gnehil force-pushed the dev-doris-catalog branch from a0d9d21 to 41258ae Compare November 30, 2023 07:17

resolve conflicts

d5e80c4

gnehil force-pushed the dev-doris-catalog branch from 41258ae to d5e80c4 Compare November 30, 2023 09:27

EricJoy2048 approved these changes Dec 1, 2023

View reviewed changes

github-actions bot added approved reviewed labels Dec 1, 2023

gnehil requested a review from Hisoka-X December 4, 2023 06:36

CalvinKirs approved these changes Dec 9, 2023

View reviewed changes

CalvinKirs merged commit 1d3e335 into apache:dev Dec 9, 2023
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Catalog] Doris Catalog #5175

[Feature][Catalog] Doris Catalog #5175

gnehil commented Jul 28, 2023 •

edited

Loading

Hisoka-X commented Aug 23, 2023 •

edited

Loading

Hisoka-X commented Aug 23, 2023

gnehil commented Sep 14, 2023

EricJoy2048 commented Sep 14, 2023

gnehil commented Sep 15, 2023

Hisoka-X commented Oct 12, 2023

gnehil commented Oct 12, 2023

Hisoka-X commented Oct 12, 2023

Hisoka-X commented Oct 12, 2023

EricJoy2048 commented Oct 23, 2023

[Feature][Catalog] Doris Catalog #5175

[Feature][Catalog] Doris Catalog #5175

Conversation

gnehil commented Jul 28, 2023 • edited Loading

Purpose of this pull request

Check list

Hisoka-X commented Aug 23, 2023 • edited Loading

Hisoka-X commented Aug 23, 2023

gnehil commented Sep 14, 2023

EricJoy2048 commented Sep 14, 2023

gnehil commented Sep 15, 2023

Hisoka-X commented Oct 12, 2023

gnehil commented Oct 12, 2023

Hisoka-X commented Oct 12, 2023

Hisoka-X commented Oct 12, 2023

EricJoy2048 commented Oct 23, 2023

gnehil commented Jul 28, 2023 •

edited

Loading

Hisoka-X commented Aug 23, 2023 •

edited

Loading