Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql: v2.0.5: panic while executing 1 statements: UPSERT panics in AddRow #29436

Closed
tbg opened this issue Aug 31, 2018 · 7 comments · Fixed by #29487
Closed

sql: v2.0.5: panic while executing 1 statements: UPSERT panics in AddRow #29436

tbg opened this issue Aug 31, 2018 · 7 comments · Fixed by #29487
Assignees
Labels
A-sql-execution Relating to SQL execution. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. S-2-temp-unavailability Temp crashes or other availability problems. Can be worked around or resolved by restarting.

Comments

@tbg
Copy link
Member

tbg commented Aug 31, 2018

https://sentry.io/cockroach-labs/cockroachdb/issues/669605580/

*log.safeError: conn_executor.go:521: panic while executing 1 statements: INSERT INTO _(_, _, _, _, _, _, _, _, _, _, _, _) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12) ON CONFLICT (_, _, _) DO UPDATE SET _ = _._, _ = _._, _ = _._, _ = _._, _ = _._, _ = _._ RETURNING _: caused by <redacted>
  File "github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go", line 489, in func2
  File "github.com/cockroachdb/cockroach/pkg/sql/sqlbase/row_container.go", line 266, in AddRow
  File "github.com/cockroachdb/cockroach/pkg/sql/tablewriter.go", line 514, in finalize
  File "github.com/cockroachdb/cockroach/pkg/sql/upsert.go", line 320, in internalNext
  File "github.com/cockroachdb/cockroach/pkg/sql/upsert.go", line 280, in startExec
...
(16 additional frame(s) were not displayed)

conn_executor.go:521: panic while executing 1 statements: INSERT INTO _(_, _, _, _, _, _, _, _, _, _, _, _) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12) ON CONFLICT (_, _, _) DO UPDATE SET _ = _._, _ = _._, _ = _._, _ = _._, _ = _._, _ = _._ RETURNING _: caused by <redacted>
@tbg tbg changed the title sql: panic while executing 1 statements: INSERT INTO _(_, _, _, _, _, _, _, _, _... sql: v2.0.5: panic while executing 1 statements: INSERT panics in AddRow Aug 31, 2018
@tbg tbg added S-2-temp-unavailability Temp crashes or other availability problems. Can be worked around or resolved by restarting. A-sql-execution Relating to SQL execution. labels Aug 31, 2018
@knz knz added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Sep 1, 2018
@jordanlewis
Copy link
Member

This is caused by the explicit panic in this statement:

// AddRow attempts to insert a new row in the RowContainer. The row slice is not
// used directly: the Datum values inside the Datums are copied to internal storage.
// Returns an error if the allocation was denied by the MemoryMonitor.
func (c *RowContainer) AddRow(ctx context.Context, row tree.Datums) (tree.Datums, error) {
	if len(row) != c.numCols {
		panic(fmt.Sprintf("invalid row length %d, expected %d", len(row), c.numCols))
	}
	if c.numCols == 0 {
		c.numRows++
		return nil, nil
	}

I think the issue here is that upsert returning doesn't take into account mutation columns. Here's where the row container is initialized:

		tu.rowsUpserted = sqlbase.NewRowContainer(
			evalCtx.Mon.MakeBoundAccount(),
			sqlbase.ColTypeInfoFromColDescs(tableDesc.Columns),
			tu.insertRows.Len(),
		)

As you can see, the col type info is created directly from tabledesc.Columns. That field includes all of the public columns of the table descriptor, but not the ones that are in the write and delete only state. Since upsert returning needs to write to columns that are in that state, when it tries to save the columns to return into the row container, there's a mismatch in the number of columns it wrote and the number of columns the row container was created with, and 💥.

I'll try to make a repro, but @vivekmenezes does this sound right?

@vivekmenezes
Copy link
Contributor

vivekmenezes commented Sep 2, 2018 via email

@jordanlewis
Copy link
Member

Hey I think this actually explains the issue in #29374 and others as well. The code that checks for NULL values in non-nullable cols for insert suffers from this exact same problem...

@jordanlewis
Copy link
Member

Hm, no, my theory is wrong because users can't insert into a fresh column until its done backfilling completely.

@jordanlewis
Copy link
Member

But I did confirm that UPSERT RETURNING on a table with a mutation does crash the server, so there's that!

@jordanlewis
Copy link
Member

Hmm, you don't even need RETURNING to crash the server. Any UPSERT on a table with mutation columns will crash it. But check out the error message we get!

The way I provoked this was to create a table, add a non-nullable column with a default, kill the server, restart it so it has to wait for its schema change lease to time out, then play around in the sql shell. The panic here was just on UPSERT INTO a VALUES(0,0) where there was a column d being added.

panic: Non-nullable column "a:d" with no value! Index scanned was "primary" with the index key columns (a) and the values (-1) [recovered]
        panic: panic while executing 1 statements: UPSERT INTO _ VALUES (_, _); caused by Non-nullable column "a:d" with no value! Index scanned was "primary" with the index key columns (a) and the values (-1)

goroutine 296 [running]:
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).closeWrapper(0xc422d0e600, 0x694ac40, 0xc42557ab80, 0x5fbe8c0, 0xc427097740)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:675 +0x36f
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1(0xc422d0e600, 0x694ac40, 0xc42557ab80)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:414 +0x61
panic(0x5fbe8c0, 0xc427097740)
        /usr/local/Cellar/go/1.10.2/libexec/src/runtime/panic.go:502 +0x229
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.(*RowFetcher).finalizeRow(0xc428287290, 0x694ad00, 0xc42827eae0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/rowfetcher.go:1261 +0x9d9
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.(*RowFetcher).NextRow(0xc428287290, 0x694ad00, 0xc42827eae0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/rowfetcher.go:1013 +0x183
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.(*RowFetcher).NextRowDecoded(0xc428287290, 0x694ad00, 0xc42827eae0, 0xc425a27e00, 0xc42827ed80, 0x1, 0x1, 0x0, 0x0, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/rowfetcher.go:1027 +0x5a
github.com/cockroachdb/cockroach/pkg/sql.(*tableUpserter).fetchExisting(0xc428286a80, 0x694ad00, 0xc42827eae0, 0x0, 0x0, 0xc425aaf500, 0xc422d0e8f8, 0x63fe336, 0xc426040f70, 0x515eea1, ...)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/tablewriter_upsert.go:825 +0x3df
github.com/cockroachdb/cockroach/pkg/sql.(*tableUpserter).atBatchEnd(0xc428286a80, 0x694ad00, 0xc42827eae0, 0xc425aaf500, 0xc422d0ea38, 0xc42827eb00)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/tablewriter_upsert.go:311 +0x69
github.com/cockroachdb/cockroach/pkg/sql.(*upsertNode).BatchedNext(0xc425748690, 0x694ad00, 0xc42827eae0, 0xc425aaf500, 0xc422d0ea38, 0x61c3d00, 0xc426041001, 0xb4a0bb0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/upsert.go:320 +0x299
github.com/cockroachdb/cockroach/pkg/sql.(*rowCountNode).startExec(0xc428487540, 0x694ad00, 0xc42827eae0, 0xc425aaf500, 0xc422d0ea38, 0x1, 0xc426041138)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_batch.go:173 +0xd0
github.com/cockroachdb/cockroach/pkg/sql.startExec.func2(0x63fe94e, 0x5, 0x693e180, 0xc428487540, 0xc425599828, 0xc426041148)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go:568 +0x90
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1(0xc42827d6d0, 0x63fe94e, 0x5, 0x693e180, 0xc428487540)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:127 +0x5f
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal(0xc42827d6d0, 0x693e180, 0xc428487540, 0x63fe94e, 0x5)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:552 +0x378
github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit(0xc42827d6d0, 0x693e180, 0xc428487540, 0xc426041d28, 0x4014009)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:94 +0x8f
github.com/cockroachdb/cockroach/pkg/sql.walkPlan(0x694ad00, 0xc42827eae0, 0x693e180, 0xc428487540, 0x0, 0x6552cf8, 0x0, 0x0, 0xc42827ec00, 0xc426041e98, ...)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:58 +0x193
github.com/cockroachdb/cockroach/pkg/sql.startExec(0x694ad00, 0xc42827eae0, 0xc425aaf500, 0xc422d0ea38, 0x693e180, 0xc428487540, 0x0, 0x1)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go:573 +0x10e
github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start(0xc4282b7200, 0x694ad00, 0xc42827eae0, 0xc428273a58, 0x5fbdbc0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:127 +0xab
github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Run(0xc4282b7200, 0x694ad00, 0xc42827eae0, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:264 +0x56
github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).StartSync(0xc425aaf880, 0x694ad00, 0xc42827eae0, 0x6552578, 0xc42622a380, 0x69286c0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:594 +0x191
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc420846700, 0xc4284532c0, 0xc425a27e00, 0xc426042808, 0xc42848aa00, 0xc422d0ead0, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:245 +0x879
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc420846700, 0x694ad00, 0xc42827e420, 0xc422d0ead0, 0xc4284532c0, 0xc425a27e00, 0x693e180, 0xc428487540, 0xc42848aa00)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:735 +0x24c
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc422d0e600, 0x694ad00, 0xc42827e420, 0xc422d0ea38, 0x2, 0x9963380, 0xc427e9ca20, 0x0, 0x0, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:966 +0x2d8
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc422d0e600, 0x694ad00, 0xc42827e420, 0x694dd00, 0xc42827d130, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:808 +0x688
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc422d0e600, 0x694ad00, 0xc42827e420, 0x694dd00, 0xc42827d130, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:401 +0xb82
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc422d0e600, 0x694ad00, 0xc42827e420, 0x694dd00, 0xc42827d130, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:95 +0x358
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc422d0e600, 0x694ac40, 0xc42557ab80, 0xc4203c41e0, 0x0, 0x0)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1079 +0x2186
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc420529680, 0x694ac40, 0xc42557ab80, 0x0, 0x0, 0xc42071b039, 0x4, 0xc42071b02a, 0x9, 0x692c600, ...)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:416 +0x1bb
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl.func3(0xc420529680, 0x694ac40, 0xc42557ab80, 0xc420341180, 0x5400, 0x15000, 0xc4207a5ec0, 0xc4203c41e0, 0xc4203c41d0, 0xc420427b50, ...)
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:267 +0x122
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl
        /Users/jordan/go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:266 +0xf04

@jordanlewis
Copy link
Member

Ah, UPSERT also doesn't evaluate and insert DEFAULT values of schema change columns, so you can in fact get the panic from #29374 without too much trouble. I've reproduced it in TestOperationsWithColumnMutation.

@jordanlewis jordanlewis changed the title sql: v2.0.5: panic while executing 1 statements: INSERT panics in AddRow sql: v2.0.5: panic while executing 1 statements: UPSERT panics in AddRow Sep 3, 2018
craig bot pushed a commit that referenced this issue Sep 4, 2018
29487: sql: fix UPSERT RETURNING in presence of schema changes r=jordanlewis a=jordanlewis

Fixes #29436.

Release note (bug fix): fix crash caused by certain kinds of UPSERT
RETURNING statements on tables with active schema changes.

Co-authored-by: Jordan Lewis <jordanthelewis@gmail.com>
@craig craig bot closed this as completed in #29487 Sep 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-sql-execution Relating to SQL execution. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. S-2-temp-unavailability Temp crashes or other availability problems. Can be worked around or resolved by restarting.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants