Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge python container components with Lightweight components #1687

Closed
Ark-kun opened this issue Jul 26, 2019 · 0 comments · Fixed by #1889
Closed

Merge python container components with Lightweight components #1687

Ark-kun opened this issue Jul 26, 2019 · 0 comments · Fixed by #1889
Assignees
@Ark-kun
Labels
kind/feature priority/p0

Comments

@Ark-kun
Copy link
Contributor

Ark-kun commented Jul 26, 2019 •
edited
Loading

Python container components (build_python_component) lacks many features that Lightweight components (func_to_container_op) support:

  • Multiple outputs (already added)
  • Non-primitive types
  • Default values
  • Optional parameters
  • Dependency capture (code pickling)

There are some reports of user confusion since there are two ways to create component from a python function and they have slight implementation differences.

We should change kfp.compiler.build_python_component to use the same code generator as func_to_container_op. We should later merge the APIs and deprecate one of the functions so that there is only one way to make component from a python function.

Related issues: #1580
@Ark-kun Ark-kun self-assigned this Jul 26, 2019
@Ark-kun Ark-kun mentioned this issue Jul 29, 2019
Closed
@Ark-kun Ark-kun mentioned this issue Aug 19, 2019
Merged
magdalenakuhn17 pushed a commit to magdalenakuhn17/pipelines that referenced this issue Oct 22, 2023
@sukumargaonkar
Limit number of asyncio workers in custom transformers (kubeflow#1687)
c10e627 
For linux platforms, infer cpu-limit for the container from 'cfs_quota_us'
and 'cfs_quota_us' files. Use min(32, cpu_count + 4) to limit number of
max asyncio workers. Refer https://bugs.python.org/issue35279 for formula
source.

Python (and asyncio) is not container aware and thus ends up creating
asyncio workers proportional to cpu core count of the machine and not the
container. This spawns too many asyncio workers on high cpu core machines
running containers with low cpu-limit causing throttling.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Ark-kun Ark-kun

Labels
kind/feature priority/p0
Projects
None yet
Milestone
No milestone
1 participant
@Ark-kun