Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for sources that do not fit common query pattern #182

Closed
yruslan opened this issue Apr 3, 2023 · 0 comments
Closed

Add support for sources that do not fit common query pattern #182

yruslan opened this issue Apr 3, 2023 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@yruslan
Copy link
Collaborator

yruslan commented Apr 3, 2023

Background

Currently, Pramen sources are either file, table, or query based: input.path=, input.table=, input.sql=.

But some sources do not fit this notion, and require different set of options to specify.

For instance, it could be data file / control file pairs.

Feature

Add the ability for a source to specify more complicated source paths than just one of the above list.

Example

  tables = [
    {
        input.data.file.1 = "s3://full/path/to/data.file1"
        input.control.file.1 = "s3://full/path/to/control.file1"

        input.data.file.2 = "s3://full/path/to/data.file2"
        input.control.file.2 = "s3://full/path/to/control.file2"

        input.data.file.3 = "s3://full/path/to/data.file3"
        input.control.file.3 = "s3://full/path/to/control.file3"
        ...
    }
  ]

Proposed Solution

Extend za.co.absa.pramen.api.Query trait with Custom case class that takes a Map[String, String] like this for the above example:

Map (
 "data.file.1" -> "s3://full/path/to/data.file1",
 "control.file.1" -> "s3://full/path/to/control.file1",
 "data.file.2" -> "s3://full/path/to/data.file2",
 "control.file.2" -> "s3://full/path/to/control.file2",
 ...
@yruslan yruslan added the enhancement New feature or request label Apr 3, 2023
@yruslan yruslan self-assigned this Apr 3, 2023
yruslan added a commit that referenced this issue Apr 3, 2023
These are queries that cannot fit the standard set of `table`, `sql`, `path`.
yruslan added a commit that referenced this issue Apr 3, 2023
These are queries that cannot fit the standard set of `table`, `sql`, `path`.
yruslan added a commit that referenced this issue Apr 3, 2023
These are queries that cannot fit the standard set of `table`, `sql`, `path`.
yruslan added a commit that referenced this issue Apr 5, 2023
These are queries that cannot fit the standard set of `table`, `sql`, `path`.
@yruslan yruslan closed this as completed Apr 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant