Fix Redshift column alteration bug #709
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Under current behavior, if you try to copy or upsert a Parsons table to Redshift with alter_table set to True, it'll take the incoming column widths and send them to Redshift as alteration queries. This will fail when a varchar column coming in from the Parsons table exceeds Redshift's maximum column width of 65535.
It's desirable to use the alter_table argument in combination with truncatecolumns to enact a sequence as follows:
As it stands, step 2 fails in this sequence because an invalid column width is sent to Redshift within an alteration query. This caps the allowable length of that alteration query. With this change, copy and upsert calls with truncatecolumns=False would still behave as expected, but calls with that argument set to true and alter_table=True would exhibit the new behavior outlined above.