Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support ETL '$include' directive #785

Merged
merged 10 commits into from
Feb 2, 2019
Merged

Support ETL '$include' directive #785

merged 10 commits into from
Feb 2, 2019

Conversation

smgallo
Copy link
Contributor

@smgallo smgallo commented Jan 25, 2019

Description

Support specification of an ETL $include directive that will allow the contents of a file to be included directly in a JSON configuration file as a JSON-encoded string. This works similarly to the RFC-6901 JSON Pointers that we already implement and allows variables to be used in the file reference.

For example, given the include file etl_sql.d/query.sql:

SELECT
  name,
  description
FROM jobs j
WHERE 1;

And the configuration file

{
    "sql": {
        "$include": "etl_sql.d/query.sql"
    }
}

Generate the following JSON:

{
    "sql": "SELECT\n  name,\n  description\nFROM jobs j\nWHERE 1;\n
}

Motivation and Context

Improve readability of included complex SQL statements and general reuse.

Tests performed

Manually tested style prior to pushing. I think that using push --force causes the "continuous-integration/travis-ci/push" tests to fail.

smgallo@bobbiedarin:xdmod [14:59:29]$ ~/src/xdmod-qa/travis/build.sh -s                                                                                    (etl-include-directive ⬇ ⬆ ◼)
Running only style tests
X11 forwarding request failed on channel 0
Comparing HEAD to upstream/xdmod8.1 (165d38cd0e1e13fa12c641e28586f5f229c9330a)
travis_fold:start:syntax
Running syntax tests...
travis_fold:end:syntax
Syntax tests succeeded!

travis_fold:start:style
Running style tests...
travis_fold:end:style
Style tests succeeded!

travis_fold:start:extra
Running extra tests...
travis_fold:end:extra
Extra tests succeeded!

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project as found in the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@smgallo smgallo added enhancement Enhancement of the functionality of an existing feature Category:ETL Extract Transform Load labels Jan 25, 2019
@smgallo smgallo added this to the 8.1.0 milestone Jan 25, 2019
@@ -0,0 +1 @@
{"reference":{"name":"resource_allocations","engine":"MyISAM","columns":[{"name":"resource","type":"varchar(40)","nullable":true}]},"include":"\"SELECT\\n name,\\n description\\nFROM modw.jobs j\\nWHERE 1;\\n\"","reference_with_subreference":{"subreference":[{"object1":"value"},{"object2":"value"}]}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wish this was pretty printed, but I wont hold it up cause of that ;)

@smgallo smgallo merged commit 3601376 into xdmod8.1 Feb 2, 2019
@smgallo smgallo deleted the etl-include-directive branch February 2, 2019 02:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Category:ETL Extract Transform Load enhancement Enhancement of the functionality of an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants