Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(ingest): Adding support for Elasticsearch and Clickhouse #3227

Merged
merged 368 commits into from
Dec 18, 2021
Merged

feat(ingest): Adding support for Elasticsearch and Clickhouse #3227

merged 368 commits into from
Dec 18, 2021

Conversation

sudotty
Copy link

@sudotty sudotty commented Sep 13, 2021

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable)

@shirshanka
Copy link
Contributor

@sudotty thanks for the contribution! Could you add some integration tests using the same patterns as the hive / MySQL tests under (metadata-ingestion/tests/integration)

@shirshanka shirshanka changed the title Hb datahub feat(ingest): Adding support for Elasticsearch and Clickhouse Sep 16, 2021
gabe-lyons and others added 28 commits October 21, 2021 19:35
@gabe-lyons
feat(docs-site): remove ga and add peleton logo (datahub-project#3437)
deedc12
@gabe-lyons
feat(ingestion): support custom properties to be ingested via busines…
1fa49e2 
…s glossary yaml (datahub-project#3438)
@gabe-lyons
fix restli client filter method (datahub-project#3436)
36e5bca
@gabe-lyons
feat(lineage): improved lineage performance + simplified layout logic…
0bb8f68 
… + some easter eggs (datahub-project#3357)
@swaroopjagadish
docs: add metadata model diagram (datahub-project#3449)
aca93ca
@gabe-lyons
fix(graphql): fix errors on tag & term mutations (datahub-project#3448)
d3d7593
@gabe-lyons
fix(browse): disable breadcrumb links on non-browsable entities (data…
2f81454 
…hub-project#3447)
@remisalmon
fix(ingest): fix lookml derived tables parsing (datahub-project#3443)
91d20e4
@gabe-lyons
docs(site): some small changes for docs site (datahub-project#3444)
1fe181f
@hsheth2
perf(ingest): lazy load ingestion plugins (datahub-project#3430)
dccef30
@jeffmerrick
fix(site): fix docs website (datahub-project#3446)
d5a42e3
fix(backup-restore): fix backup restore jobs usage of system-metadata (…
87d344f 
…datahub-project#3445)
@kevinhu
fix(ingest): lineage for Airflow subdags (datahub-project#3351)
3fe5ef7
@maggiehays
docs: Update to Q3 2021 accomplishments (datahub-project#3420)
9b05a26
@varunbharill
fix(ingest): Add gcp logging dependency for bigquery source. (datahub…
7e27355 
…-project#3451)

fixes datahub-project#3450
@gabe-lyons
build(frontend): make unzip depend on yarnBuild (datahub-project#3452)
b4a4a83
@gabe-lyons
feat(react): add handy webpack analyzer command (datahub-project#3454)
d72ea01
@EnricoMi
test(CI): show test results on GitHub (datahub-project#3362)
b0435fd
@WaStCo
docs(transformers): add exemple of custom tag function (datahub-proje…
bd1a550 
…ct#3354)

Co-authored-by: STEUER COSTA <wagner.steuer-costa@lhind.dlh.de>
@DSchmidtDev
docs: add guide for using custom sources (datahub-project#3324)
43f1aeb
@AndreasTA-AW
feat(dbt-ingestion): add ability to skip specific models (datahub-pro…
561c04b 
…ject#3340)
@varunbharill
fix(ingest): Ensure mongodb source doesn't croak on large documents. (d…
8d8cdc4 
…atahub-project#3456)
@jjoyce0510
fix(mysql-setup): update default mysql collation to utf8mb4_bin (data…
664bbd3 
…hub-project#3459)
@rslanka
fix(ingestion): workaround for Python 3.8/3.9 mypy invalid syntax iss…
34a5d5c 
…ue with airflow 2.2.0 (datahub-project#3460)
@jjoyce0510
Fixing UI username display (datahub-project#3461)
a527114
@gabe-lyons
fix(react): fix up yarn test error reporting (datahub-project#3462)
3739420
@gabe-lyons
docs(frontend): remove confusing suggestion to manually create users (d…
9572f46 
…atahub-project#3465)
@maggiehays
docs: overhaul DataHub Features page (datahub-project#3439)
a62edb2
Dexter Lee and others added 26 commits December 13, 2021 18:18
feat(retention): Add retention to Local DB (datahub-project#3715)
8e4769f
@hsheth2
feat(ingest): cleanup deprecated datahub.integrations.airflow.* imp…
3b7fd24 
…orts (datahub-project#3732)
@jawadqu
feat(ingestion) : Add Metabase Source Connector (datahub-project#3602)
578590e 
Co-authored-by: Jawad Qureshi <jqureshi@petabloc.com>
@hsheth2
fix(ingest): count profiled tables separately in report (datahub-proj…
f24440e 
…ect#3731)
@anshbansal
feat(perf-test): changes for perf testing (datahub-project#3728)
a4ac79d
@gabe-lyons
ci(cypress): adding the foundation for cypress integration tests & so…
e9c87bc 
…me starter coverage for login, search & updates (datahub-project#3672)
@nsbala-tw @balabarath
fix(elastic): Fix for log4j CVE-2021-44228 vulnerability (datahub-pro…
89f6c47 
…ject#3733)

Co-authored-by: balabarath <bagopila@gmail.com>
@gabe-lyons
Revert "feat(graph): Make Dgraph a proper Neo4j alternative (datahub-…
3fd3313 
…project#3578)" (datahub-project#3740)
@jjoyce0510
fix(CI): regenerate Docker Quickstart (datahub-project#3741)
5c0a49d
@varunbharill
fix(ingest): changing datahub-graph to use underlying session connect…
5f80e7a 
…ion. (datahub-project#3743)
@jjoyce0510
fix(ingest): remove data platform isalpha check as it complains about…
40963b0 
… s3 (datahub-project#3742)
@gabe-lyons
feat(ingest): snowflake-usage add knob for direct objects accessed vs…
2e2ed34 
… base objects accessed (datahub-project#3744)
@varunbharill
fix(ingest): snowflake honor allow/deny pattern for lineage and usage. (
70d0688 
datahub-project#3748)
@jjoyce0510
refactor(auth): Remove base64 decoding of token service signing key (d…
1f53e0b 
…atahub-project#3747)
@hsheth2
test(ingest): fix pytest warning for class starting with Test (data…
adf9d2e 
…hub-project#3745)
@sgomezvillamor
feat: enables dbt metadata files to be loaded from URIs (datahub-proj…
c59c63e 
…ect#3739)
@treff7es
feat(ingest): skipping emitting metadata for duplicate tables from in…
1c1561c 
…gestion (datahub-project#3753)
@rslanka
feat(Stateful Ingestion): 1/3 Stateful ingestion server changes (data…
4b81ad6 
…hub-project#3749)
@jjoyce0510
fix(vuln): log4j vulnerability - bumping to 2.16.0 (datahub-project#3755
5b5135b 
)
@hsheth2
build(ingest): restrict latest mypy version (datahub-project#3756)
c6f3ddf
@merqurio
doc: Add IOMED as a DataHub adopter (datahub-project#3758)
21fbfbb
@MugdhaHardikar-GSLab
docs(ingest): spark-lineage update artifact name and version (datahub…
e6f8c1c 
…-project#3760)
@hsheth2
feat(ingest): profiling add upper bound on combined query size (datah…
df2cb94 
…ub-project#3762)
@jawadqu
feat(ingestion): Mode retry wait logic to avoid hitting Mode API rate…
b4eaa9e 
… limit (datahub-project#3761)
@rslanka @shirshanka
feat(Stateful Ingestion-2/3): Client side changes for checkpointing a…
bd69e73 
… source job state. (datahub-project#3763)

Co-authored-by: Shirshanka Das <shirshanka@apache.org>
@hsheth2
refactor(test): replace CliRunner with run_datahub_cmd method (da…
22cef5f 
…tahub-project#3746)
@wangqinghuan wangqinghuan deleted the hb-datahub branch December 17, 2021 08:57
@wangqinghuan wangqinghuan restored the hb-datahub branch December 17, 2021 08:58
@wangqinghuan wangqinghuan deleted the hb-datahub branch December 17, 2021 08:59
@shirshanka shirshanka merged commit 22cef5f into datahub-project:master Dec 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rslanka rslanka rslanka left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.