We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Currently, Pramen sources are either file, table, or query based: input.path=, input.table=, input.sql=.
input.path=
input.table=
input.sql=
But some sources do not fit this notion, and require different set of options to specify.
For instance, it could be data file / control file pairs.
Add the ability for a source to specify more complicated source paths than just one of the above list.
tables = [ { input.data.file.1 = "s3://full/path/to/data.file1" input.control.file.1 = "s3://full/path/to/control.file1" input.data.file.2 = "s3://full/path/to/data.file2" input.control.file.2 = "s3://full/path/to/control.file2" input.data.file.3 = "s3://full/path/to/data.file3" input.control.file.3 = "s3://full/path/to/control.file3" ... } ]
Extend za.co.absa.pramen.api.Query trait with Custom case class that takes a Map[String, String] like this for the above example:
za.co.absa.pramen.api.Query
Custom
Map[String, String]
Map ( "data.file.1" -> "s3://full/path/to/data.file1", "control.file.1" -> "s3://full/path/to/control.file1", "data.file.2" -> "s3://full/path/to/data.file2", "control.file.2" -> "s3://full/path/to/control.file2", ...
The text was updated successfully, but these errors were encountered:
#182 Add support for custom queries for data sources.
04a55f5
These are queries that cannot fit the standard set of `table`, `sql`, `path`.
efca387
0d9d891
a274435
yruslan
No branches or pull requests
Background
Currently, Pramen sources are either file, table, or query based:
input.path=
,input.table=
,input.sql=
.But some sources do not fit this notion, and require different set of options to specify.
For instance, it could be data file / control file pairs.
Feature
Add the ability for a source to specify more complicated source paths than just one of the above list.
Example
Proposed Solution
Extend
za.co.absa.pramen.api.Query
trait withCustom
case class that takes aMap[String, String]
like this for the above example:The text was updated successfully, but these errors were encountered: