[Engprod] Stable python package management

[Bobgy]

We have done many hotfixes like

• [Fix] Pin avro==1.9.1 #3067
• Fix build failure #3035
• [Visualization] tfdv visualization throws an error #3079

Also e.g. things don't work when we upgrade base google/cloud-sdk image.

Can we set up a stable python package management strategy that pins all the packages we use?

Some options:

• https://github.com/pypa/pipenv
• https://github.com/jazzband/pip-tools

UPDATE: we have decided to go with pip-tools, because it's lightweight and just focus on the dependency pinning task we wanted.

/cc @numerology
/cc @Ark-kun
/area engprod

[Ark-kun]

I think we have two goals here:

• Test the SDK in a stable environment
• Test the SDK in a more recent environment that the user might have

Python's way of pinning dependencies is to freeze the packages to a file: pip3 freeze > requirements.txt which can then be used to reproduce the environment: pip3 install -r requirements.txt.

So, we should probably test both ways - test with pinned packages installed from requirements.txt and test with latest packages installed by running the setup.py.

We should probably use pinned versions for the image building.

[Bobgy]

Test the SDK in a more recent environment that the user might have

@Ark-kun Thanks, totally make sense to me. Can we handle this goal in post submit test instead, so it is not presubmit blocking?

[Ark-kun]

Can we handle this goal in post submit test instead, so it is not presubmit blocking?

Sounds good.

[Bobgy]

TODOs:

• backend and sample-test: backend: Pin python dependencies #3161
• metadata writer image
• visualization server
• Please add if there's other things I missed

[Ark-kun]

• SDK

[discordianfish]

Metadata writer image: #3408

[Ark-kun]

The visualization-server licenses seem to be broken after some package got updated.

[Bobgy]

@Ark-kun OK, I will try to do the same for visualization-server.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Bobgy]

/lifecycle frozen