Ensure correct field order in INSERT and implement order_id sequence functionality #201
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…stor to aRdbmsDestinationAction
…to aAction and aRdbmsDestinationAction
…e endpoint plus associated tests
…date ETL config to use new endpoint
smgallo
added
bug
plessbd
reviewed
Aug 2, 2017
| @@ -485,7 +496,7 @@ function ($s) { | |||
| $destColumnList | |||
| ); | |||
| $updateColumns = implode(',', $updateColumnList); | |||
| $sql = "INSERT INTO $qualifiedDestTableName ($destColumns) " . $this->sourceQueryString | |||
| $sql = "INSERT INTO $qualifiedDestTableName\n($destColumns)\n" . $this->sourceQueryString | |||
There was a problem hiding this comment.
the mix of interpolation and not here is messing with my reading of this...
plessbd
approved these changes
Aug 2, 2017
ryanrath
pushed a commit
to ryanrath/xdmod
that referenced
this pull request
Sep 18, 2017
…functionality (ubccr#201) * Ensure correct field order in INSERT. Implement order_id as sequence.
chakrabortyr
pushed a commit
to chakrabortyr/xdmod
that referenced
this pull request
Oct 17, 2017
…functionality (ubccr#201) * Ensure correct field order in INSERT. Implement order_id as sequence.
ryanrath
pushed a commit
to ryanrath/xdmod
that referenced
this pull request
Jan 15, 2021
…urce - Updated the url for the TreeGrid to point at `/etl/pipelines/actions` - Upated the `getActionsforPipelines` method chain so that it produces output suitable for use in an ExtJS TreeGrid. - The change @ ubccr#201 was needed due to `get_object_vars` returning a keyed array `<property> => <value>`. - ubccr#289: This function preps the output for use use in an EXtJS TreeGrid. - ubccr#453: This was simplify / declutter the display of the action name from it's fully qualified form `xdmod.pipeline.action` to `action` as the nodes immediately preceeding the action node will already be displaying the full pipeline name.
ryanrath
pushed a commit
to ryanrath/xdmod
that referenced
this pull request
Jan 15, 2021
…urce - Updated the url for the TreeGrid to point at `/etl/pipelines/actions` - Upated the `getActionsforPipelines` method chain so that it produces output suitable for use in an ExtJS TreeGrid. - The change @ ubccr#201 was needed due to `get_object_vars` returning a keyed array `<property> => <value>`. - ubccr#289: This function preps the output for use use in an EXtJS TreeGrid. - ubccr#453: This was simplify / declutter the display of the action name from it's fully qualified form `xdmod.pipeline.action` to `action` as the nodes immediately preceeding the action node will already be displaying the full pipeline name.
ryanrath
pushed a commit
to ryanrath/xdmod
that referenced
this pull request
Feb 25, 2021
…urce - Updated the url for the TreeGrid to point at `/etl/pipelines/actions` - Upated the `getActionsforPipelines` method chain so that it produces output suitable for use in an ExtJS TreeGrid. - The change @ ubccr#201 was needed due to `get_object_vars` returning a keyed array `<property> => <value>`. - ubccr#289: This function preps the output for use use in an EXtJS TreeGrid. - ubccr#453: This was simplify / declutter the display of the action name from it's fully qualified form `xdmod.pipeline.action` to `action` as the nodes immediately preceeding the action node will already be displaying the full pipeline name.
ryanrath
pushed a commit
to ryanrath/xdmod
that referenced
this pull request
Mar 31, 2021
…urce - Updated the url for the TreeGrid to point at `/etl/pipelines/actions` - Upated the `getActionsforPipelines` method chain so that it produces output suitable for use in an ExtJS TreeGrid. - The change @ ubccr#201 was needed due to `get_object_vars` returning a keyed array `<property> => <value>`. - ubccr#289: This function preps the output for use use in an EXtJS TreeGrid. - ubccr#453: This was simplify / declutter the display of the action name from it's fully qualified form `xdmod.pipeline.action` to `action` as the nodes immediately preceeding the action node will already be displaying the full pipeline name.
ryanrath
pushed a commit
to ryanrath/xdmod
that referenced
this pull request
Mar 31, 2021
…urce - Updated the url for the TreeGrid to point at `/etl/pipelines/actions` - Upated the `getActionsforPipelines` method chain so that it produces output suitable for use in an ExtJS TreeGrid. - The change @ ubccr#201 was needed due to `get_object_vars` returning a keyed array `<property> => <value>`. - ubccr#289: This function preps the output for use use in an EXtJS TreeGrid. - ubccr#453: This was simplify / declutter the display of the action name from it's fully qualified form `xdmod.pipeline.action` to `action` as the nodes immediately preceeding the action node will already be displaying the full pipeline name.
ryanrath
pushed a commit
to ryanrath/xdmod
that referenced
this pull request
Jul 2, 2021
…urce - Updated the url for the TreeGrid to point at `/etl/pipelines/actions` - Upated the `getActionsforPipelines` method chain so that it produces output suitable for use in an ExtJS TreeGrid. - The change @ ubccr#201 was needed due to `get_object_vars` returning a keyed array `<property> => <value>`. - ubccr#289: This function preps the output for use use in an EXtJS TreeGrid. - ubccr#453: This was simplify / declutter the display of the action name from it's fully qualified form `xdmod.pipeline.action` to `action` as the nodes immediately preceeding the action node will already be displaying the full pipeline name.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ensure correct field order in INSERT and implement order_id sequence functionality for historical purposes.
Description
The field list for the INSERT must be in the same order as the fields returned by the query or we will get field mismatches. Use the source record fields (generated from the source query in initialize()) as the correct order. Since the destination field map may have been user-specified we cannot guarantee the order.
Some historical ingestors use an order_id and treat it similar to an auto-increment field, setting its value starting at 0 and incrementing for each record. It is not clear if this is used anywhere, but the functionality is maintained here for compatibility. Note that this does not work when adding fields into a table (e.g.,only when the table is truncated) and will not work properly in cases where the order is relative to a key. For example, if a key is resource_id and the order should be maintained for each unique resource_id we cannot use this method. To not overwrite existing data, only set the order_id if the source field exists and is NULL.
Motivation and Context
Fixes a bug where the INSERT INTO...SELECT field order for same-database inserts is not the same as the order of the fields in the SELECT.
Tests performed
Ran tests and performed sample ingestion.
Types of changes
Checklist: