Support insert into statement in sqllogictest
#4496
Conversation
There was a problem hiding this comment.
I think this is great -- thank you @xudong963.
I suggest we try and avoid the changes to add interior mutability to MemTable if we can avoid it (I left a suggestion below) and I am happy to help make this happen if need be.
But all in all this is wonderful and I think we could merge this PR in as is and iterate on master as well
| @@ -958,6 +958,22 @@ impl SessionContext { | |||
| } | |||
| } | |||
|
|
|||
| /// Return a [`TabelProvider`] for the specified table. | |||
| pub fn table_provider<'a>( | |||
There was a problem hiding this comment.
👍 it might be nice to refactor fn table() to call this function now to avoid some duplication.
|
|
||
| /// DataFusion sql-logicaltest error | ||
| #[derive(Debug)] | ||
| pub enum DFSqlLogicTestError { |
| NotImplemented(String), | ||
| /// Error returned from DFSqlLogicTest inner | ||
| Internal(String), |
There was a problem hiding this comment.
I suggest we simply panic in the sqllogic runner in these cases so the location of the error is easier to see
| // Check if the sql is `insert` | ||
| if sql.trim_start().to_lowercase().starts_with("insert") { | ||
| // Process the insert statement | ||
| insert(ctx, sql).await?; | ||
| return Ok("".to_string()); | ||
| } |
There was a problem hiding this comment.
👍 I like this basic approach (special case the sql and route it to the test runner implementation).
One thing that might be worth doing is to actually try and parse the input into sql, to detect INSERT statements though I think string manipulation is fine too or we could do this later
// Handle any test only special case statements
let sql = sql.into();
match DFParser::parse_sql(&sql) {
Ok(Statement(Insert)) => {
//debug!("Parsed statement: {:#?}", stmt);
}
Err(_) => {
// ignore anything else, including errors -- they will be handled by the sql context below
}
};There was a problem hiding this comment.
Yes, more generic way, will fix in next PR.
| 11 20 | ||
|
|
||
| # Test insert into a undefined table | ||
| statement error |
| # under the License. | ||
|
|
||
| statement ok | ||
| CREATE TABLE users AS VALUES(1,2),(2,3); |
| // Todo: check columns match table schema | ||
| table_name = name.to_string(); | ||
| match &*source.body { | ||
| SetExpr::Values(values) => { |
| } | ||
|
|
||
| // Final, append the `RecordBatch` to memtable's batches | ||
| let mut table_batches = table_batches.write(); |
There was a problem hiding this comment.
Rather than changing the batches in the existing memtable, what would you think about creating a new memtable with the same name with the new values (rather than modifying the original one)
I think you might be able to avoid changes to SessionContext and MemTable entirely.
Something like this (untested)
// fetch existing batches
let mut existing_batches = ctx.table(table_name.as_str()).collect();
// append new batch
exsiting_batches.extend(insert_batches)
// Replace table provider provider
let new_provider = MemTable::try_new(batches[0].schema(), vec![batches]);
ctx.register_table(table_name, new_provider)There was a problem hiding this comment.
Yes, I have thought about the way you mentioned, (also need to delete original memtable).
-- But for performance reasons, I choose the current way: modifying the original one.
If you think the changes to add interior mutability to MemTable don't make sense, I can change it in the following ticket! (I don't have a strong bias for both ways)
There was a problem hiding this comment.
I think keeping MemTable as simple as possible is likely the best approach -- so for that reason I prefer to remove the interior mutability.
I can give it a shot if you agree -- I think the performance of copying record batches (for reasonably small data such as what is in the test) will be ok
There was a problem hiding this comment.
Ok, agree! Thanks @alamb . I'll refactor it later.
|
Benchmark runs are scheduled for baseline = df41267 and contender = b5b50ba. b5b50ba is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
Which issue does this PR close?
Closes #4397
Rationale for this change
Support initial
insert intostatement in sqllogictest framework which will help a lot migrate tests!What changes are included in this PR?
insert intostatement.insert intosqllogictest.Are these changes tested?
Yes
Are there any user-facing changes?
No