feat: copy to support for lance #2342
Conversation
|
can we add some tests. Ideally, we should test the round trip to/from lance in a |
|
This is ready and has tests now (both SLT and python). I've apparently had the code for this correct in every case, except that the local write path pre-creates the directory for us, which the lance writer doesn't like. The lance API is kinda wack: and in the end doesn't really use our object store at all (and I think can't) not for technical reasons but for API reasons, so eventually we should be able to get that ironed out but requires sort of major surgery the lance types. |
| copy (select * from lance_tbl) to 'file://${TMP}' format lance; | ||
|
|
||
| query IT | ||
| select * from lance_scan('file://${TMP}') order by point.lat; | ||
| ---- | ||
| 0.2,1.8 {lat:42.1,long:-74.1} | ||
| 1.1,1.2 {lat:45.5,long:-122.7} |
There was a problem hiding this comment.
will add extra tests. no clue, lance's file parsing is a bit picky relative to other methods, but it should.
There was a problem hiding this comment.
I took out the file::// and it works fine, which is what we test with every data format; I believe these are still absolute paths (/tmp/etc)
There was a problem hiding this comment.
I think relative paths are very important from a usability perspective. I'd hate having to write an absolute path every time.
If both these don't work, I think we should open a follow up issue to get it working:
copy (select 1) to './path/to/table' format lance;
select * from lance_scan('./path/to/table');
|
sorry, one more note. Let's update the README support matrix. |
closes #2066