Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(elastricsearch): add description fiels #9693

Merged
merged 3 commits into from
Feb 5, 2024
Merged

docs(elastricsearch): add description fiels #9693

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
38b8c53
docs(elasticsearch): add description
dim-ops Jan 20, 2024
c5e6f65
fmt(elasticsearch): fmt
dim-ops Jan 23, 2024
a17d3ad
chore: imrpove collapse_urns description
Jan 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions metadata-ingestion/src/datahub/ingestion/source/elastic_search.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -299,9 +299,13 @@ class ElasticsearchSourceConfig(PlatformInstanceConfigMixin, EnvConfigMixin):

profiling: ElasticProfiling = Field(
default_factory=ElasticProfiling,
description="Configs to ingest data profiles from ElasticSearch.",
)
collapse_urns: CollapseUrns = Field(
default_factory=CollapseUrns,
description="""List of regex patterns to remove from the name of the URN. All of the indices before removal of URNs are considered as the same dataset. These are applied in order for each URN.
The main case where you would want to have multiple of these if the name where you are trying to remove suffix from have different formats.
e.g. ending with -YYYY-MM-DD as well as ending -epochtime would require you to have 2 regex patterns to remove the suffixes across all URNs.""",
)

def is_profiling_enabled(self) -> bool:
Expand Down
Loading