Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Redshift column alteration bug #709

Merged
merged 2 commits into from
Jul 22, 2022

Conversation

SorenSpicknall
Copy link
Collaborator

Under current behavior, if you try to copy or upsert a Parsons table to Redshift with alter_table set to True, it'll take the incoming column widths and send them to Redshift as alteration queries. This will fail when a varchar column coming in from the Parsons table exceeds Redshift's maximum column width of 65535.

It's desirable to use the alter_table argument in combination with truncatecolumns to enact a sequence as follows:

  1. Incoming Parsons table has values that are too wide for the current column width and too wide for Redshift varchar constraints
  2. Parsons resizes the destination columns to take up the maximum allowed width in Redshift, and then
  3. Parsons successfully pushes the table to Redshift, truncating the column values that exceed 65535 characters.

As it stands, step 2 fails in this sequence because an invalid column width is sent to Redshift within an alteration query. This caps the allowable length of that alteration query. With this change, copy and upsert calls with truncatecolumns=False would still behave as expected, but calls with that argument set to true and alter_table=True would exhibit the new behavior outlined above.

Copy link
Collaborator

@shaunagm shaunagm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me (with caveat that I can't do any tests here)

@SorenSpicknall SorenSpicknall merged commit 67957aa into main Jul 22, 2022
@SorenSpicknall SorenSpicknall deleted the soren-bugfix_redshift_alter_column_widths branch July 22, 2022 20:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants