Skip to content

Commit

Permalink
Issue #17012: Multi User/Team Ownership (#17013)
Browse files Browse the repository at this point in the history
* Add multiple owners

* Multi Ownership

* Issue #17012: Multi User/Team Ownership

* Issue #17012: Multi User/Team Ownership

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 1

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 2

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 3

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 4

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 5

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 6

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 7

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 8

* Add Migrations for Owner Thread

* update ingestion for multi owner

* fix pytests

* fixed checkstyle

* Add Alert Name to Publishers (#17108)

* Add Alert Name to Publishers

* Fix Test

* Add Bound to Setuptools (#17105)

* Minor: fixed testSummaryGraph issue (#17115)

* feat: updated multi pipeline ui as per new mock (#17106)

* feat: updated multi pipeline ui as per new mock

* translation sync

* fixed failing unit test

* fixed playwright test

* fixed viewService click issue

* sorted pipeline based on test case length

* Added domo federated dataset support (#17061)

* fix usernames (#17122)

* Doc: Updated Doris & Redshift Docs (#17123)

Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>

* Fix #12677: Added Synapse Connector - docs and side docs (#17041)

* Fix #17098: Fixed case sensitive partition column name in Bigquery (#17104)

* Fixed case sensitive partiion col name bigquery

* update test

* #13876: change placement of comment and close button in task approval workflow (#17044)

* change placment of comment and close button in task approval workflow

* minor change

* playwright test for the close and comment function

* supported ref in activityFeedEditor

* fix playwright test

* added playwright test for data steward

* fix the test for the data streward user

* fix the close button not showing if task has no suggestions and icon fixes

* fix sonar issue

* change glossary and add suggestion button to dropdown button

* fix the glossary failure due to button change

* icon change for add tag and description

* fix glossary cypress failure due to button chnages

* changes as per comments

* MINOR: docs links fix (#17125)

* alation link fix

* dbt yaml config source link fix

* bigquery doc fix

* Explore tree feedbacks (#17078)

* fix explore design

* update switcher icon

* show menu when search query exists

* fix selection of active service

* fix type error

* fix tests

* fix tests

* fix tests

* MINOR: Databricks view TableType fix (#17124)

* Minor: fixed AUT test (#17128)

* Fix #16692: Override Lineage Support for View & Dashboard Lineage (#17064)

* #17065: fix the tags not rendering in selector after selection in edit tags task (#17107)

* fix the tags not rendering in selector after selection in edit tags taks

* added playwright test

* minor changes

* minor fix

* fix the tags not updating in edit and accept tag

* fix explore type changes for collate (#17131)

* MINOR: changed log level to debug (#17126)

* changed log level to debug

* fixed type

* changed type to optional

* Get feed and count data of soft deleted user (#17135)

* Doc: Adding OIDC Docs (#17139)

Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>

* Doc: Updating Profiler Workflow Docs URL (#17140)

Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>

* fix playwright and cypress (#17138)

* Minor: fixed edit modal issue for sql test case (#17132)

* Minor: fixed edit modal issue for sql test case

* fixed test

* Minor: Added whats new content for 1.4.6 release (#17148)

* MINOR [GEN-799]: add option to disable manual trigger using scheduleType (#17031)

* fix: raise for triggering system app

* added scheduleType ScheduledOrManual

* minor: remove "service" field from required properties in createAPIEndpoint schema (#17147)

* initial commit multi ownership

* update glossary and other entities

* update owners

* fix version pages

* fix tests

* Update entity_extension to move owner to array (#17200)

* fix tests

* fix api page errors

* fix owner label design

* locales

* fix owners in elastic search source

* fix types

* fix tests

* fix tests

* Updated CustomMetric owner to entityReferenceList. (#17211)

* Fix owners field in search mappings

* fix search aggregates

* fix inherited label

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 9

* Fix QUeries

* Fix Mysql Queries

* Typo

* fix tests

* fix tests

* fix tests

* fix advanced search constants

* fix service ingestion tests

* fix tests

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: k.nakagaki <141020064+nakaken-churadata@users.noreply.github.com>
Co-authored-by: Prajwal214 <167504578+Prajwal214@users.noreply.github.com>
Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: harshsoni2024 <64592571+harshsoni2024@users.noreply.github.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: Imri Paran <imri.paran@gmail.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Siddhant <86899184+Siddhanttimeline@users.noreply.github.com>
  • Loading branch information
19 people authored Jul 30, 2024
1 parent a39b59b commit fe107aa
Show file tree
Hide file tree
Showing 705 changed files with 4,141 additions and 4,119 deletions.
38 changes: 38 additions & 0 deletions bootstrap/sql/migrations/native/1.5.0/mysql/schemaChanges.sql
Original file line number Diff line number Diff line change
Expand Up @@ -148,3 +148,41 @@ JSON_EXTRACT(json, '$.sourceConfig.config.dbtConfigSource.dbtSecurityConfig.gcpC
JSON_EXTRACT(json, '$.sourceConfig.config.dbtConfigSource.dbtSecurityConfig.gcpConfig.externalType') OR
JSON_EXTRACT(json, '$.sourceConfig.config.dbtConfigSource.dbtSecurityConfig.gcpConfig.path')
) is NULL AND JSON_EXTRACT(json, '$.sourceConfig.config.dbtConfigSource.dbtSecurityConfig.gcpConfig') is not null;

-- Update Owner Field to Owners
DELETE from event_subscription_entity where name = 'ActivityFeedAlert';

-- Update thread_entity to move previousOwner and updatedOwner to array
UPDATE thread_entity
SET json = JSON_SET(
json,
'$.feedInfo.entitySpecificInfo.previousOwner',
JSON_ARRAY(
JSON_EXTRACT(json, '$.feedInfo.entitySpecificInfo.previousOwner')
)
)
WHERE JSON_CONTAINS_PATH(json, 'one', '$.feedInfo.entitySpecificInfo.previousOwner')
AND JSON_TYPE(JSON_EXTRACT(json, '$.feedInfo.entitySpecificInfo.previousOwner')) <> 'ARRAY';

UPDATE thread_entity
SET json = JSON_SET(
json,
'$.feedInfo.entitySpecificInfo.updatedOwner',
JSON_ARRAY(
JSON_EXTRACT(json, '$.feedInfo.entitySpecificInfo.updatedOwner')
)
)
WHERE JSON_CONTAINS_PATH(json, 'one', '$.feedInfo.entitySpecificInfo.updatedOwner')
AND JSON_TYPE(JSON_EXTRACT(json, '$.feedInfo.entitySpecificInfo.updatedOwner')) <> 'ARRAY';

-- Update entity_extension to move owner to array
UPDATE entity_extension
SET json = JSON_SET(
json,
'$.owner',
JSON_ARRAY(
JSON_EXTRACT(json, '$.owner')
)
)
WHERE JSON_CONTAINS_PATH(json, 'one', '$.owner')
AND JSON_TYPE(JSON_EXTRACT(json, '$.owner')) <> 'ARRAY';
35 changes: 35 additions & 0 deletions bootstrap/sql/migrations/native/1.5.0/postgres/schemaChanges.sql
Original file line number Diff line number Diff line change
Expand Up @@ -138,3 +138,38 @@ SET json = jsonb_set(
AND json#>>'{sourceConfig,config,dbtConfigSource,dbtSecurityConfig,gcpConfig,type}' IS NULL
AND json#>>'{sourceConfig,config,dbtConfigSource,dbtSecurityConfig,gcpConfig,externalType}' IS NULL
AND json#>>'{sourceConfig,config,dbtConfigSource,dbtSecurityConfig,gcpConfig,path}' IS NULL;

-- Update Owner Field to Owners
DELETE from event_subscription_entity where name = 'ActivityFeedAlert';

-- Update thread_entity to move previousOwner and updatedOwner to array
UPDATE thread_entity
SET json = jsonb_set(
json,
'{feedInfo,entitySpecificInfo,previousOwner}',
to_jsonb(ARRAY[json->'feedInfo'->'entitySpecificInfo'->'previousOwner'])
)
WHERE jsonb_path_exists(json, '$.feedInfo.entitySpecificInfo.previousOwner')
AND jsonb_path_query_first(json, '$.feedInfo.entitySpecificInfo.previousOwner ? (@ != null)') IS NOT null
AND jsonb_typeof(json->'feedInfo'->'entitySpecificInfo'->'updatedOwner') <> 'array';

UPDATE thread_entity
SET json = jsonb_set(
json,
'{feedInfo,entitySpecificInfo,updatedOwner}',
to_jsonb(ARRAY[json->'feedInfo'->'entitySpecificInfo'->'updatedOwner'])
)
WHERE jsonb_path_exists(json, '$.feedInfo.entitySpecificInfo.updatedOwner')
AND jsonb_path_query_first(json, '$.feedInfo.entitySpecificInfo.updatedOwner ? (@ != null)') IS NOT null
AND jsonb_typeof(json->'feedInfo'->'entitySpecificInfo'->'updatedOwner') <> 'array';

-- Update entity_extension to move owner to array
UPDATE entity_extension
SET json = jsonb_set(
json,
'{owner}',
to_jsonb(ARRAY[jsonb_path_query_first(json, '$.owner')])
)
WHERE jsonb_path_exists(json, '$.owner')
AND jsonb_path_query_first(json, '$.owner ? (@ != null)') IS NOT null
AND jsonb_typeof(json->'owner') <> 'array';
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,9 @@ def __init__(self, metadata: OpenMetadata):
def name(self) -> str:
return "Entity Report Processor"

def _get_team(self, owner: EntityReference) -> Optional[str]:
def _get_team( # pylint: disable=too-many-return-statements
self, owner: EntityReference
) -> Optional[str]:
"""Get the team from an entity. We'll use this info as well to
add info if an entity has an owner
Expand All @@ -91,10 +93,12 @@ def _get_team(self, owner: EntityReference) -> Optional[str]:
Returns:
Optional[str]
"""
if not owner:
if not owner or not owner.root:
return None

if isinstance(owner, EntityReferenceList):
if not owner.root:
return None
return owner.root[0].name

if owner.type == "team":
Expand Down Expand Up @@ -188,7 +192,7 @@ def refine(self, entity: Type[T]) -> None:
data_blob_for_entity = {}
try:
team = (
self._get_team(entity.owner)
self._get_team(entity.owners)
if not isinstance(entity, User)
else self._get_team(entity.teams) # type: ignore
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,9 @@ def _pre_hook_fn(self):
self.refine_entity_event = self._refine_entity_event()
next(self.refine_entity_event)

def _refine_entity_event(self) -> Generator[dict, WebAnalyticEventData, None]:
def _refine_entity_event( # pylint: disable=too-many-branches, too-many-statements
self,
) -> Generator[dict, WebAnalyticEventData, None]:
"""Coroutine to process entity web analytic event
Yields:
Expand Down Expand Up @@ -158,8 +160,11 @@ def _refine_entity_event(self) -> Generator[dict, WebAnalyticEventData, None]:
)

try:
owner = entity.owner.name if entity.owner else None
owner_id = str(entity.owner.id.root) if entity.owner else None
owner = None
owner_id = None
if entity.owners and len(entity.owners.root) > 0:
owner = entity.owners.root[0].name
owner_id = str(entity.owners.root[0].id.root)
except AttributeError as exc:
owner = None
owner_id = None
Expand Down
2 changes: 1 addition & 1 deletion ingestion/src/metadata/data_insight/source/metadata.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ def _iter(self, *_, **__) -> Iterable[Either[DataInsightRecord]]:

for data in (
producer.fetch_data(
fields=["owner", "tags"], entities_cache=self.entities_cache
fields=["owners", "tags"], entities_cache=self.entities_cache
)
or []
):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ def compare_and_create_test_cases(
if test_case_to_create.parameterValues
else None
),
owner=None,
owners=None,
computePassedFailedRowCount=test_case_to_create.computePassedFailedRowCount,
)
)
Expand Down
2 changes: 1 addition & 1 deletion ingestion/src/metadata/data_quality/source/test_suite.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ def _process_table_suite(self, table: Table) -> Iterable[Either[TableAndTests]]:
),
displayName=f"{self.source_config.entityFullyQualifiedName.root} Test Suite",
description="Test Suite created from YAML processor config file",
owner=None,
owners=None,
executableEntityReference=self.source_config.entityFullyQualifiedName.root,
)
yield Either(
Expand Down
4 changes: 2 additions & 2 deletions ingestion/src/metadata/ingestion/models/patch_request.py
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ class PatchedEntity(BaseModel):
"displayName": True,
"sourceUrl": True,
"description": True,
"owner": True,
"owners": True,
"tags": True,
"sourceHash": True,
# Table Entity Fields
Expand Down Expand Up @@ -143,7 +143,7 @@ class PatchedEntity(BaseModel):
"fileFormats": True,
}

RESTRICT_UPDATE_LIST = ["description", "tags", "owner", "displayName"]
RESTRICT_UPDATE_LIST = ["description", "tags", "owners", "displayName"]

ARRAY_ENTITY_FIELDS = ["columns", "tasks", "fields"]

Expand Down
9 changes: 5 additions & 4 deletions ingestion/src/metadata/ingestion/ometa/mixins/patch_mixin.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
from metadata.generated.schema.tests.testCase import TestCase, TestCaseParameterValue
from metadata.generated.schema.type.basic import EntityLink, Markdown
from metadata.generated.schema.type.entityReference import EntityReference
from metadata.generated.schema.type.entityReferenceList import EntityReferenceList
from metadata.generated.schema.type.lifeCycle import LifeCycle
from metadata.generated.schema.type.tagLabel import TagLabel
from metadata.ingestion.api.models import Entity
Expand Down Expand Up @@ -323,7 +324,7 @@ def patch_owner(
self,
entity: Type[T],
source: T,
owner: EntityReference = None,
owners: EntityReferenceList = None,
force: bool = False,
) -> Optional[T]:
"""
Expand All @@ -340,20 +341,20 @@ def patch_owner(
Updated Entity
"""
instance: Optional[T] = self._fetch_entity_if_exists(
entity=entity, entity_id=source.id, fields=["owner"]
entity=entity, entity_id=source.id, fields=["owners"]
)

if not instance:
return None

# Don't change existing data without force
if instance.owner and not force:
if instance.owners and instance.owners.root and not force:
# If a owner is already present and force is not passed,
# owner will not be overridden
return None

destination = deepcopy(instance)
destination.owner = owner
destination.owners = owners

return self.patch(entity=entity, source=instance, destination=destination)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ def add_task_to_pipeline(self, pipeline: Pipeline, *tasks: Task) -> Pipeline:
startDate=pipeline.startDate,
service=pipeline.service.fullyQualifiedName,
tasks=all_tasks,
owner=pipeline.owner,
owners=pipeline.owners,
tags=pipeline.tags,
)

Expand All @@ -113,7 +113,7 @@ def clean_pipeline_tasks(self, pipeline: Pipeline, task_ids: List[str]) -> Pipel
startDate=pipeline.startDate,
service=pipeline.service.fullyQualifiedName,
tasks=[task for task in pipeline.tasks if task.name in task_ids],
owner=pipeline.owner,
owners=pipeline.owners,
tags=pipeline.tags,
)

Expand Down
61 changes: 39 additions & 22 deletions ingestion/src/metadata/ingestion/ometa/mixins/user_mixin.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
from metadata.generated.schema.entity.teams.team import Team, TeamType
from metadata.generated.schema.entity.teams.user import User
from metadata.generated.schema.type.entityReference import EntityReference
from metadata.generated.schema.type.entityReferenceList import EntityReferenceList
from metadata.ingestion.api.common import T
from metadata.ingestion.ometa.client import REST
from metadata.utils.constants import ENTITY_REFERENCE_TYPE_MAP
Expand Down Expand Up @@ -118,30 +119,38 @@ def get_reference_by_email(
from_count: int = 0,
size: int = 1,
fields: Optional[list] = None,
) -> Optional[EntityReference]:
) -> Optional[EntityReferenceList]:
"""
Get a User or Team Entity Reference by searching by its mail
"""
maybe_user = self._search_by_email(
entity=User, email=email, from_count=from_count, size=size, fields=fields
)
if maybe_user:
return EntityReference(
id=maybe_user.id.root,
type=ENTITY_REFERENCE_TYPE_MAP[User.__name__],
name=maybe_user.name.root,
displayName=maybe_user.displayName,
return EntityReferenceList(
root=[
EntityReference(
id=maybe_user.id.root,
type=ENTITY_REFERENCE_TYPE_MAP[User.__name__],
name=maybe_user.name.root,
displayName=maybe_user.displayName,
)
]
)

maybe_team = self._search_by_email(
entity=Team, email=email, from_count=from_count, size=size, fields=fields
)
if maybe_team:
return EntityReference(
id=maybe_team.id.root,
type=ENTITY_REFERENCE_TYPE_MAP[Team.__name__],
name=maybe_team.name.root,
displayName=maybe_team.displayName,
return EntityReferenceList(
root=[
EntityReference(
id=maybe_team.id.root,
type=ENTITY_REFERENCE_TYPE_MAP[Team.__name__],
name=maybe_team.name.root,
displayName=maybe_team.displayName,
)
]
)

return None
Expand All @@ -154,19 +163,23 @@ def get_reference_by_name(
size: int = 1,
fields: Optional[list] = None,
is_owner: bool = False,
) -> Optional[EntityReference]:
) -> Optional[EntityReferenceList]:
"""
Get a User or Team Entity Reference by searching by its name
"""
maybe_user = self._search_by_name(
entity=User, name=name, from_count=from_count, size=size, fields=fields
)
if maybe_user:
return EntityReference(
id=maybe_user.id.root,
type=ENTITY_REFERENCE_TYPE_MAP[User.__name__],
name=maybe_user.name.root,
displayName=maybe_user.displayName,
return EntityReferenceList(
root=[
EntityReference(
id=maybe_user.id.root,
type=ENTITY_REFERENCE_TYPE_MAP[User.__name__],
name=maybe_user.name.root,
displayName=maybe_user.displayName,
)
]
)

maybe_team = self._search_by_name(
Expand All @@ -176,11 +189,15 @@ def get_reference_by_name(
# if is_owner is True, we only want to return the team if it is a group
if is_owner and maybe_team.teamType != TeamType.Group:
return None
return EntityReference(
id=maybe_team.id.root,
type=ENTITY_REFERENCE_TYPE_MAP[Team.__name__],
name=maybe_team.name.root,
displayName=maybe_team.displayName,
return EntityReferenceList(
root=[
EntityReference(
id=maybe_team.id.root,
type=ENTITY_REFERENCE_TYPE_MAP[Team.__name__],
name=maybe_team.name.root,
displayName=maybe_team.displayName,
)
]
)

return None
Original file line number Diff line number Diff line change
Expand Up @@ -434,7 +434,7 @@ def mark_datamodels_as_deleted(self) -> Iterable[Either[DeleteEntity]]:

def get_owner_ref( # pylint: disable=unused-argument, useless-return
self, dashboard_details
) -> Optional[EntityReference]:
) -> Optional[EntityReferenceList]:
"""
Method to process the dashboard owners
"""
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@
Markdown,
SourceUrl,
)
from metadata.generated.schema.type.entityReference import EntityReference
from metadata.generated.schema.type.entityReferenceList import EntityReferenceList
from metadata.ingestion.api.models import Either
from metadata.ingestion.api.steps import InvalidSourceException
from metadata.ingestion.ometa.ometa_api import OpenMetadata
Expand Down Expand Up @@ -103,7 +103,7 @@ def get_dashboard_details(self, dashboard: DomoDashboardDetails) -> dict:

def get_owner_ref(
self, dashboard_details: DomoDashboardDetails
) -> Optional[EntityReference]:
) -> Optional[EntityReferenceList]:
for owner in dashboard_details.owners or []:
try:
owner_details = self.client.domo.users_get(owner.id)
Expand Down Expand Up @@ -142,7 +142,7 @@ def yield_dashboard(
for chart in self.context.get().charts or []
],
service=self.context.get().dashboard_service,
owner=self.get_owner_ref(dashboard_details=dashboard_details),
owners=self.get_owner_ref(dashboard_details=dashboard_details),
)
yield Either(right=dashboard_request)
self.register_record(dashboard_request=dashboard_request)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ def yield_dashboard(
for chart in self.context.get().charts or []
],
service=self.context.get().dashboard_service,
owner=self.get_owner_ref(dashboard_details=dashboard_details),
owners=self.get_owner_ref(dashboard_details=dashboard_details),
)
yield dashboard_request
self.register_record(dashboard_request=dashboard_request)
Expand Down
Loading

0 comments on commit fe107aa

Please sign in to comment.