Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make sure that our pickling process is robust against future changes #104

Open
yifanwu opened this issue Sep 7, 2021 · 1 comment
Open
Labels
good first issue Good for newcomers help wanted Extra attention is needed hygiene P2 Nice to have or need to have but in the future

Comments

@yifanwu
Copy link
Contributor

yifanwu commented Sep 7, 2021

Right now we rely a lot of pickled columns via SQLAlchemy (via type decorators).

Changes include:

  • Python minor version change
  • SQLAlchemy versions
  • Database versions
  • Database types
  • What machine things are ran on

It's probably important before we start supporting real customers to make sure that the pickling process is robust and backward compatible.

I think strings are more robust against these software changes but might be slower? Something to consider.

@yifanwu yifanwu added good first issue Good for newcomers help wanted Extra attention is needed hygiene P2 Nice to have or need to have but in the future labels Sep 7, 2021
@yifanwu yifanwu changed the title Make sure that our pickling process is robust against version changes (both Python minor versions and lib) Make sure that our pickling process is robust against future changes Sep 7, 2021
@dorx
Copy link
Contributor

dorx commented Sep 7, 2021

For some context on pickler choice: apache/airflow#7870

We should use cloudpickle if it's something that airflow et al. is using.

We can pass in our pickler of choice via:
image

We would then instantiate the column with something like:

Column(PickleType(pickler=cloudpickle.pickle), nullable=True)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers help wanted Extra attention is needed hygiene P2 Nice to have or need to have but in the future
Projects
None yet
Development

No branches or pull requests

2 participants