-
Notifications
You must be signed in to change notification settings - Fork 752
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(source): add ndjson source for streaming load #4561
Conversation
This pull request is being automatically deployed with Vercel (learn more). 🔍 Inspect: https://vercel.com/databend/databend/C2CdjAHtRkZrF4gyA4XSPdkmDbN3 [Deployment for e3b4166 canceled] |
Thanks for the contribution! Please review the labels and make any necessary changes. |
Cool, almost good to me, stateless test failed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
self.buffer.clear(); | ||
if self | ||
.reader | ||
.read_line(&mut self.buffer) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here is a block IO in an async function, might block runtime thread.
In this PR, it would not cause serious problem, because we are using std::io::Curse
, but in the future, peopel may trying to use this in other situnations, and shot their foot.
Maybe we should replace R: std::io::BufRead
with R: tokio::io::BufRead
or R: futures::io::AsyncBufRead
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, good catch, I will create a fix for that.
I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/
Summary
Summary about this PR
Changelog
Related Issues
Fixes #4531
Test Plan
Unit Tests
Stateless Tests