Updating Schema reading procedures and refactoring #78
Conversation
Hsankesara
commented
Jan 16, 2024
•
Hsankesara
commented
- Updated data reading procedure
- refactor to make all modules more spark dependent
- Added and updated tests + upgraded spark version to 3.5.0
Hsankesara
force-pushed
the
updating_schema_inference
branch
from
ca53ac0 to
f2d8faa
Compare
Added custom data reading module that can be used independently
config.yaml
Outdated
| @@ -4,7 +4,7 @@ project: | |||
| version: mock_version | |||
|
|
|||
| input: | |||
| data_type: local # couldbe mock, local, sftp, s3 | |||
| data_type: mock # couldbe mock, local, sftp, s3 | |||
There was a problem hiding this comment.
would this be better as data_source or source_type data type is more specific to the data, this I think relates more to the source of the data
radarpipeline/io/downloader.py
Outdated
There was a problem hiding this comment.
While SFTP (though rsync might be useful for restart function) and S3 (implemented?) probably cover a lot of cases, I don't want to really support every method here as we can't support the long tail of the distribution. It should probably be the user's responsibility to provide a way to expose the remote data with network mounts, local copies, etc.
There was a problem hiding this comment.
Agreed, I think SFTP and S3 would cover the majority of cases. Anything else would need to be sorted by the user. I haven't implemented S3 yet but it's on my TODO list.
| logger = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| class CustomDataReader(): |
There was a problem hiding this comment.
What is the distinction here between ingestion and reader?
There was a problem hiding this comment.
Nothing, I think ingestion is the filename. I tried to keep the function names which are exposed to the user simple and straight. That's why I named it that way.
