Large SQL files skipped without error on supabase db reset
#470
Labels
bug
Something isn't working
Comments
|
Debugging this right now. First find of significance:
Line 131 in 5dd3234 |
|
And the issue appears to be that cli/internal/utils/parser/token.go Lines 70 to 79 in 5dd3234 By adding a print to the end: for scanner.Scan() {
token := scanner.Text()
stats = append(stats, token)
}
fmt.Println(scanner.Err())
return statsI get the following output: |
|
Following up from documentation:
https://pkg.go.dev/bufio#Scanner EDIT I opted for increasing the buffer size of the scanner instead: https://stackoverflow.com/a/39864391/1137077 |
sweatybridge
pushed a commit
that referenced
this issue
Sep 24, 2022
For really long lines, it is possible for the scanner to run out of buffer space, resulting in ```go scanner.Err() // bufio.Scanner: token too long ``` This error was not checked however, causing a silent failure which was supressed. This commit adds a check for `scanner.Err()` and properly propagates any non-nil values.
sweatybridge
pushed a commit
that referenced
this issue
Sep 24, 2022
For really long lines, it is possible for the scanner to run out of buffer space, resulting in ```go scanner.Err() // bufio.Scanner: token too long ``` Following [this](https://stackoverflow.com/a/39864391/1137077) suggestion from Stack Overflow, this commit implements a workaround where the maximum buffer capacity is increased from 64K to 256K. Testing against the SQL file with geodata from #470 showed that already 128K was enough to resolve that issue. 256K is double that and - as a wise man once said - "ought to be enough for anybody".
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug report
Describe the bug
My database should contain geographical data for Sweden. This data is inserted using 327
INSERTstatements and the SQL file takes up ~4MB. When I attempt to run this file (usingsupabase db reset) as part of either my migrations (supabase/migrations/*.sql) or my initial seed (supabase/seed.sql) some data is skipped over and the reset operation is reported as a success.To Reproduce
Steps to reproduce the behavior, please provide code snippets or a repository:
supabase/migrations/supabase db resetlocationstableExpected behavior
locationstable should contain 327 records - the full SQL file should be executed.Screenshots
In this case, the actual count was 206 records:
System information
Additional info
INSERTs, I'm consistently getting 206 records inserted.INSERTstatements into multiple files, the number of inserted records changes.INSERT, I am able to have all 327 records inserted (though this is a very messy workaround).The text was updated successfully, but these errors were encountered: