Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SDK - Components - Added ComponentStore search #3884

Conversation

Ark-kun
Copy link
Contributor

@Ark-kun Ark-kun commented Jun 1, 2020

ComponentStore(...).search searches for components by name in the configured component store. It prints the name and URL for components that match the given name.
Only components on GitHub are currently supported.

Example:

kfp.components.ComponentStore.default_store.search('xgboost')

>>> Xgboost train   https://raw.githubusercontent.com/.../components/XGBoost/Train/component.yaml
>>> Xgboost predict https://raw.githubusercontent.com/.../components/XGBoost/Predict/component.yaml

The default_store is configured with the main KFP component repo. The users can customize the store with their own repos and URL prefixes:

my_store = ComponentStore(
    url_search_prefixes=[
        'https://raw.githubusercontent.com/Ark-kun/kfp_components/master/components/',
        'https://raw.githubusercontent.com/<org>/<repo>/master/components/',
    ],
)
my_store.search('xgboost')

ComponentStore(...).search searches for components by name in the configured component store. It prints the name and URL for components that match the given name.
Only components on GitHub are currently supported.

Example:

```
kfp.components.ComponentStore.default_store.search('xgboost')

>>> Xgboost train   https://raw.githubusercontent.com/.../components/XGBoost/Train/component.yaml
>>> Xgboost predict https://raw.githubusercontent.com/.../components/XGBoost/Predict/component.yaml
```
@kubeflow-bot
Copy link

This change is Reviewable

@Ark-kun
Copy link
Contributor Author

Ark-kun commented Jun 1, 2020

/retest

Copy link

@numerology numerology left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Ark-kun !

Is that possible to add a test of this library, for example, by either mocking requests.get() or test against current KFP repo?

sdk/python/kfp/components/_component_store.py Outdated Show resolved Hide resolved
sdk/python/kfp/components/_component_store.py Outdated Show resolved Hide resolved
sdk/python/kfp/components/_component_store.py Outdated Show resolved Hide resolved
sdk/python/kfp/components/_component_store.py Outdated Show resolved Hide resolved
@Ark-kun
Copy link
Contributor Author

Ark-kun commented Jun 4, 2020

Is that possible to add a test of this library, for example, by either mocking requests.get() or test against current KFP repo?

It would be great to have. I'm experimenting with mocks and plan to add mock-based tests for ComponentStore in a future PR. Do you think it's OK?

@Ark-kun Ark-kun force-pushed the SDK---Components---Added-ComponentStore-search branch from ed2f04a to a095a6a Compare June 4, 2020 02:28
@Bobgy
Copy link
Contributor

Bobgy commented Jun 4, 2020

This is a great idea!
/lgtm
I'd like to understand more CUJs, like how third parties can share their store how to discover other stores?

@Ark-kun
Copy link
Contributor Author

Ark-kun commented Jun 4, 2020

I'd like to understand more CUJs, like how third parties can share their store how to discover other stores?

Currently the search only occurs in the configured repositories. Currently there are some issues with global search (it requires API token and it can only list 1000 results). The repo marker file will make the discovery more reliable, but only when users start adopting it.

The ComponentStore.default_store is configured with the main KFP component repo. The users can customize the store with their own repos and URL prefixes:

my_store = ComponentStore(
    url_search_prefixes=[
        'https://raw.githubusercontent.com/kubeflow/pipelines/master/components/',
        'https://raw.githubusercontent.com/Ark-kun/kfp_components/master/components/',
        'https://raw.githubusercontent.com/<org>/<repo>/master/components/',
    ],
)
my_store.search('xgboost')

@Ark-kun
Copy link
Contributor Author

Ark-kun commented Jun 4, 2020

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ark-kun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Ark-kun
Copy link
Contributor Author

Ark-kun commented Jun 4, 2020

/retest

4 similar comments
@Ark-kun
Copy link
Contributor Author

Ark-kun commented Jun 4, 2020

/retest

@Ark-kun
Copy link
Contributor Author

Ark-kun commented Jun 4, 2020

/retest

@Ark-kun
Copy link
Contributor Author

Ark-kun commented Jun 4, 2020

/retest

@Ark-kun
Copy link
Contributor Author

Ark-kun commented Jun 4, 2020

/retest

@numerology
Copy link

It would be great to have. I'm experimenting with mocks and plan to add mock-based tests for ComponentStore in a future PR. Do you think it's OK?

Sure. SGTM

@numerology
Copy link

/lgtm
/retest

@Ark-kun
Copy link
Contributor Author

Ark-kun commented Jun 4, 2020

/retest

@k8s-ci-robot k8s-ci-robot merged commit 1403b9b into kubeflow:master Jun 4, 2020
RedbackThomson pushed a commit to RedbackThomson/pipelines that referenced this pull request Jun 17, 2020
* SDK - Components - Added ComponentStore search

ComponentStore(...).search searches for components by name in the configured component store. It prints the name and URL for components that match the given name.
Only components on GitHub are currently supported.

Example:

```
kfp.components.ComponentStore.default_store.search('xgboost')

>>> Xgboost train   https://raw.githubusercontent.com/.../components/XGBoost/Train/component.yaml
>>> Xgboost predict https://raw.githubusercontent.com/.../components/XGBoost/Predict/component.yaml
```

* Implemented the review feedback

* Added retries
Jeffwan pushed a commit to Jeffwan/pipelines that referenced this pull request Dec 9, 2020
* SDK - Components - Added ComponentStore search

ComponentStore(...).search searches for components by name in the configured component store. It prints the name and URL for components that match the given name.
Only components on GitHub are currently supported.

Example:

```
kfp.components.ComponentStore.default_store.search('xgboost')

>>> Xgboost train   https://raw.githubusercontent.com/.../components/XGBoost/Train/component.yaml
>>> Xgboost predict https://raw.githubusercontent.com/.../components/XGBoost/Predict/component.yaml
```

* Implemented the review feedback

* Added retries
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants