Add write to output stream #1468
Conversation
snuyanzin
force-pushed
the
file
branch
3 times, most recently
from
6879c73 to
72268ce
Compare
|
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## main #1468 +/- ##
============================================
+ Coverage 92.26% 92.32% +0.05%
- Complexity 3243 3251 +8
============================================
Files 324 324
Lines 6377 6387 +10
Branches 622 623 +1
============================================
+ Hits 5884 5897 +13
+ Misses 343 340 -3
Partials 150 150 ☔ View full report in Codecov by Sentry. |
| @@ -111,13 +113,13 @@ public String generate(Schema<IN, ?> schema, int limit) { | |||
| @Override | |||
| public String getStartStream(Schema<IN, ?> schema) { | |||
| StringBuilder sb = new StringBuilder(); | |||
| generateHeader(null, sb); | |||
| generateHeader(schema, sb, false); | |||
There was a problem hiding this comment.
suddenly it was an uncovered bug leading to NPE 🙈
| return null; | ||
| } | ||
|
|
||
| @Override | ||
| public String getEndStream() { | ||
| throw new UnsupportedOperationException(); | ||
| return null; |
There was a problem hiding this comment.
Wouldn't it be better to return empty string?
There was a problem hiding this comment.
that's where I'm still not sure, empty string also might be ok here
There was a problem hiding this comment.
didn't get it, it has been already changed in previous commit
f30ad74
Or what exactly do you mean?
There was a problem hiding this comment.
Sorry for some reason I was only seeing one changed. (Maybe I looked at a single commit vs the whole change set 🤷♂️)
There was a problem hiding this comment.
no problem, thanks for the feedback
This is awesome! In the examples I show I've implemented a simple writer. This feature will make a lot of difference in writing large amounts of data. |
This is the next step after #1176 (some formats are already supporting streaming).
Now it would make sense to add ability to write data to output streams (in tests there is an example with files).
Currently supported for json, sql, csv, xml
This means for those formats it is possible to write huge amount of data into files or somewhere else (since streaming is happening in row basis keep in mind that the case when one row of data is not able to be kept in memory is not supported yet)
fyi: @eliasnogueira , I noticed you've mentioned something similar to huge files writing in one of the talks