Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [duplicate key value violates unique constraint] #129909

Closed
github-actions bot opened this issue Aug 29, 2024 · 10 comments · Fixed by #130361
Closed

pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [duplicate key value violates unique constraint] #129909

github-actions bot opened this issue Aug 29, 2024 · 10 comments · Fixed by #130361
Assignees
@fqazi
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@github-actions
Copy link

github-actions bot commented Aug 29, 2024 •
edited by cockroach-jira-scripts
Loading

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 4756b021edfdd477427f4ec34bf83f837f26a855:

   {
    "query": "SELECT EXISTS (SELECT * FROM [SHOW REGIONS FROM DATABASE])",
    "result": false
   },
   {
    "query": "WITH descriptors AS (SELECT crdb_internal.pb_to_json('cockroach.sql.sqlbase.Descriptor', descriptor)-\u003e'table' AS d FROM system.descriptor WHERE id = $1::REGCLASS) SELECT EXISTS (SELECT 1 FROM descriptors WHERE ((d-\u003e'localityConfig')-\u003e'regionalByRow') IS NOT NULL)",
    "queryArgs": [
     "schema_w0_23.table_w1_70"
    ],
    "result": false
   }
  ],
  "previousStatements": [
   "ALTER TABLE schema_w0_23.table_w1_70 ADD CONSTRAINT \"table_w1_70_col70_w1_80_unique\" UNIQUE (col70_w1_80)"
  ]
 }
}
    schema_change_external_test.go:134: 
        	Error Trace:	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:134
        	Error:      	Received unexpected error:
        	            	***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: transaction committed but schema change aborted with error: (23505): duplicate key value violates unique constraint "table_w1_70_col70_w1_80_unique" (SQLSTATE 23505)
        	            	(1) forced error mark
        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:658
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:123
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (4) ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error
        	            	Wraps: (5) ERROR: transaction committed but schema change aborted with error: (23505): duplicate key value violates unique constraint "table_w1_70_col70_w1_80_unique" (SQLSTATE 23505)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 99
Total Schema Statements Succeeded = 99
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 72
Total Transactions Rolled Back = 51
Total Transactions Executed = 123
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/test.outputs/logTestWorkload4057762091
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload4057762091
--- FAIL: TestWorkload (22.10s)

Parameters:

  • attempt=1
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-41764
@github-actions github-actions bot added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Aug 29, 2024
@cockroach-teamcity
Copy link
Member

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 5b1702fc72dba825e94c60ac581f7f2bdc04d1fb:

   {
    "query": "SELECT EXISTS (SELECT * FROM [SHOW REGIONS FROM DATABASE])",
    "result": false
   },
   {
    "query": "WITH descriptors AS (SELECT crdb_internal.pb_to_json('cockroach.sql.sqlbase.Descriptor', descriptor)-\u003e'table' AS d FROM system.descriptor WHERE id = $1::REGCLASS) SELECT EXISTS (SELECT 1 FROM descriptors WHERE ((d-\u003e'localityConfig')-\u003e'regionalByRow') IS NOT NULL)",
    "queryArgs": [
     "public.table_w0_76"
    ],
    "result": false
   }
  ],
  "previousStatements": [
   "ALTER TABLE public.table_w0_76 ADD CONSTRAINT \"table_w0_76_col26_w1_33_unique\" UNIQUE (col26_w1_33)"
  ]
 }
}
    schema_change_external_test.go:134: 
        	Error Trace:	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:134
        	Error:      	Received unexpected error:
        	            	***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: transaction committed but schema change aborted with error: (23505): duplicate key value violates unique constraint "table_w0_76_col26_w1_33_unique" (SQLSTATE 23505)
        	            	(1) forced error mark
        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:658
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:123
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (4) ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error
        	            	Wraps: (5) ERROR: transaction committed but schema change aborted with error: (23505): duplicate key value violates unique constraint "table_w0_76_col26_w1_33_unique" (SQLSTATE 23505)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 52
Total Schema Statements Succeeded = 52
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 32
Total Transactions Rolled Back = 28
Total Transactions Executed = 60
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_15_of_25/test.outputs/logTestWorkload1831917458
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload1831917458
--- FAIL: TestWorkload (13.59s)

Parameters:

  • attempt=1
  • run=15
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@rafiss rafiss changed the title pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed [duplicate key value violates unique constraint] Aug 30, 2024
@rafiss rafiss removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Aug 30, 2024
@rafiss
Copy link
Collaborator

rafiss commented Aug 30, 2024

similar to #129366 cc @fqazi

@rafiss rafiss added the P-2 Issues/test failures with a fix SLA of 3 months label Aug 30, 2024
@cockroach-teamcity cockroach-teamcity mentioned this issue Aug 31, 2024
Closed
@cockroach-teamcity
Copy link
Member

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

   {
    "query": "SELECT EXISTS (SELECT * FROM schema_w1_28.table_w0_64)",
    "result": false
   },
   {
    "query": "WITH descriptors AS (SELECT crdb_internal.pb_to_json('cockroach.sql.sqlbase.Descriptor', descriptor)-\u003e'table' AS d FROM system.descriptor WHERE id = $1::REGCLASS) SELECT EXISTS (SELECT mut FROM (SELECT json_array_elements(d-\u003e'mutations') AS mut FROM descriptors) WHERE (mut-\u003e'primaryKeySwap') IS NOT NULL)",
    "queryArgs": [
     "schema_w1_28.table_w0_64"
    ],
    "result": false
   }
  ],
  "previousStatements": [
   "ALTER TABLE schema_w1_28.table_w0_64 ADD COLUMN col_w0_64_w0_70 VARCHAR NOT NULL"
  ]
 }
}
    schema_change_external_test.go:134: 
        	Error Trace:	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:134
        	Error:      	Received unexpected error:
        	            	***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: transaction committed but schema change aborted with error: (23502): validation of column "col_w0_64_w0_70" NOT NULL failed on row: ".co.l64_w0_65"='i', "col/64_w0_66"=0.28295178868727344, col6_4_w0_67=3802, col64_w0_68=3220, col_w0_64_w0_70=NULL (SQLSTATE 23502)
        	            	(1) forced error mark
        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:658
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:123
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (4) ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error
        	            	Wraps: (5) ERROR: transaction committed but schema change aborted with error: (23502): validation of column "col_w0_64_w0_70" NOT NULL failed on row: ".co.l64_w0_65"='i', "col/64_w0_66"=0.28295178868727344, col6_4_w0_67=3802, col64_w0_68=3220, col_w0_64_w0_70=NULL (SQLSTATE 23502)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 40
Total Schema Statements Succeeded = 40
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 24
Total Transactions Rolled Back = 24
Total Transactions Executed = 48
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_15_of_25/test.outputs/logTestWorkload2769255836
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload2769255836
--- FAIL: TestWorkload (10.49s)

Parameters:

  • attempt=1
  • run=15
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193
        	            	  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202
        	            	  |   | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1543
        	            	  |   | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1744
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:708
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:819
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:170
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:819
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:819
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:137
        	            	  |   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:135
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:819
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:718
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:903
        	            	  | Wraps: (2) single entry value with no default column id
        	            	  | Error types: (1) *withstack.withStack (2) *errutil.leafError (SQLSTATE XXUUU)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.withPrefix (7) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 125
Total Schema Statements Succeeded = 125
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 82
Total Transactions Rolled Back = 75
Total Transactions Executed = 157
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_19_of_25/test.outputs/logTestWorkload1968110049
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload1968110049
--- FAIL: TestWorkload (24.50s)

Parameters:

  • attempt=1
  • run=19
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ f341fb5d738192dbd9ea55e3fd7ed9d52407f6f5:

   {
    "query": "WITH descriptors AS (SELECT crdb_internal.pb_to_json('cockroach.sql.sqlbase.Descriptor', descriptor)-\u003e'table' AS d FROM system.descriptor WHERE id = $1::REGCLASS) SELECT EXISTS (SELECT mut FROM (SELECT json_array_elements(d-\u003e'mutations') AS mut FROM descriptors) WHERE (mut-\u003e'primaryKeySwap') IS NOT NULL)",
    "queryArgs": [
     "public.table_w1_3"
    ],
    "result": false
   },
   {
    "query": "SELECT EXISTS (SELECT * FROM [SHOW REGIONS FROM DATABASE])",
    "result": false
   }
  ],
  "previousStatements": [
   "CREATE UNIQUE INDEX index_w1_3_w0_169 ON public.table_w1_3 (c_ol3_w1_6) STORING (\"col3 _w1_5\", col_w1_3_w0_8, col3_w1_4)"
  ]
 }
}
    schema_change_external_test.go:134: 
        	Error Trace:	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:134
        	Error:      	Received unexpected error:
        	            	***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: transaction committed but schema change aborted with error: (23505): failed to ingest index entries during backfill: duplicate key value violates unique constraint "index_w1_3_w0_169" (SQLSTATE 23505)
        	            	(1) forced error mark
        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:658
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:123
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (4) ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error
        	            	Wraps: (5) ERROR: transaction committed but schema change aborted with error: (23505): failed to ingest index entries during backfill: duplicate key value violates unique constraint "index_w1_3_w0_169" (SQLSTATE 23505)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 107
Total Schema Statements Succeeded = 107
Total Schema Statement Expected Failures = 1
Total Transactions Committed = 77
Total Transactions Rolled Back = 67
Total Transactions Executed = 144
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/0/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_15_of_25/test.outputs/logTestWorkload4142447864
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload4142447864
--- FAIL: TestWorkload (22.75s)

Parameters:

  • attempt=1
  • run=15
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 950a090603bffa9216f01b03aeeb7dd093c6be64:

   {
    "query": "SELECT EXISTS (SELECT * FROM [SHOW REGIONS FROM DATABASE])",
    "result": false
   },
   {
    "query": "WITH descriptors AS (SELECT crdb_internal.pb_to_json('cockroach.sql.sqlbase.Descriptor', descriptor)-\u003e'table' AS d FROM system.descriptor WHERE id = $1::REGCLASS) SELECT EXISTS (SELECT 1 FROM descriptors WHERE ((d-\u003e'localityConfig')-\u003e'regionalByRow') IS NOT NULL)",
    "queryArgs": [
     "public.table_w1_51"
    ],
    "result": false
   }
  ],
  "previousStatements": [
   "ALTER TABLE public.table_w1_51 ADD CONSTRAINT \"table_w1_51_col59_w0_64_unique\" UNIQUE (col59_w0_64)"
  ]
 }
}
    schema_change_external_test.go:134: 
        	Error Trace:	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:134
        	Error:      	Received unexpected error:
        	            	***UNEXPECTED COMMIT ERROR; Received an unexpected commit error: ERROR: transaction committed but schema change aborted with error: (23505): failed to ingest index entries during backfill: duplicate key value violates unique constraint "table_w1_51_col59_w0_64_unique" (SQLSTATE 23505)
        	            	(1) forced error mark
        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:658
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:123
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (4) ***UNEXPECTED COMMIT ERROR; Received an unexpected commit error
        	            	Wraps: (5) ERROR: transaction committed but schema change aborted with error: (23505): failed to ingest index entries during backfill: duplicate key value violates unique constraint "table_w1_51_col59_w0_64_unique" (SQLSTATE 23505)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 66
Total Schema Statements Succeeded = 66
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 42
Total Transactions Rolled Back = 35
Total Transactions Executed = 77
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_1_of_25/test.outputs/logTestWorkload3110114137
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload3110114137
--- FAIL: TestWorkload (14.95s)

Parameters:

  • attempt=1
  • run=1
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ fa9c0528fc0d06be1b4cfc534ec0501448111fbe:

    "query": "SELECT EXISTS (SELECT table_name FROM information_schema.views WHERE (table_schema = $1) AND (table_name = $2))",
    "queryArgs": [
     "public",
     "view_w1_126"
    ],
    "result": false
   }
  ],
  "previousStatements": [
   "CREATE VIEW public.view_w1_126 AS SELECT public.table_w0_111.crdb_internal_mvcc_timestamp, public.table_w1_86.\"c%v\fol86_w1_90\", public.table_w1_86.col86_w1_87, public.table_w1_86.\"col8\\\\xf06_w1_99\", public.table_w1_86.col86_w1_97, public.table_w1_86.col8😲6_w1_96, public.table_w1_86.col86_w1_102, public.table_w1_86.col86_w1_94, public.table_w1_86.\"%pcol86_w1_88\", public.table_w1_86.\"c|ol86_w1_89\", public.table_w1_86.\"col86\\v_w1_91\", public.table_w1_86.\"'col86_w1_93\", public.table_w1_86.cͪol86_w1_95, public.table_w1_86.col86_w1_98, public.table_w1_86.\"\\fcol8\\v6 _w1_100\", public.table_w1_86.col86_w1_92, public.table_w1_86.col86_w1_101 FROM public.table_w0_111, public.table_w1_86"
  ]
 }
}
    schema_change_external_test.go:134: 
        	Error Trace:	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:134
        	Error:      	Received unexpected error:
        	            	***UNEXPECTED ERROR; Received an unexpected execution error.: ERROR: relation "view_w1_126" (138): column name "crdb_internal_mvcc_timestamp" conflicts with a system column name (SQLSTATE 42701)
        	            	(1) forced error mark
        	            	  | "fatal error when running txn"
        	            	  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
        	            	Wraps: (2)
        	            	Wraps: (3) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*opStmt).executeStmt
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/operation_generator.go:3139
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).runInTxn
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:499
        	            	  | github.com/cockroachdb/cockroach/pkg/workload/schemachange.(*schemaChangeWorker).run
        	            	  | 	github.com/cockroachdb/cockroach/pkg/workload/schemachange/schemachange.go:587
        	            	  | pkg/ccl/testccl/workload/schemachange/schemachange_test_test.TestWorkload.TestWorkload.func4.func6
        	            	  | 	pkg/ccl/testccl/workload/schemachange/schemachange_test_test/pkg/ccl/testccl/workload/schemachange/schema_change_external_test.go:123
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (4) ***UNEXPECTED ERROR; Received an unexpected execution error.
        	            	Wraps: (5) ERROR: relation "view_w1_126" (138): column name "crdb_internal_mvcc_timestamp" conflicts with a system column name (SQLSTATE 42701)
        	            	Error types: (1) *markers.withMark (2) *schemachange.ErrorState (3) *withstack.withStack (4) *errutil.withPrefix (5) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 85
Total Schema Statements Succeeded = 85
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 59
Total Transactions Rolled Back = 48
Total Transactions Executed = 107
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/3/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_19_of_25/test.outputs/logTestWorkload2300880375
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload2300880375
--- FAIL: TestWorkload (28.26s)

Parameters:

  • attempt=1
  • run=19
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Sep 7, 2024
Open
@cockroach-teamcity
Copy link
Member

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 00edd9c46e7f3638846d0dc182a9dd50a0aa9cbc:

        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:152
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Next
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/stats.go:126
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).nextAdapter
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:246
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:152
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).next
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:250
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:282
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:320
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:928
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1990
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1704
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1707
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2427
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1974
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1181
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:146
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3436
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:145
        	            	  |   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:251
        	            	  | Wraps: (2) single entry value with no default column id
        	            	  | Error types: (1) *withstack.withStack (2) *errutil.leafError (SQLSTATE XX000)
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 125
Total Schema Statements Succeeded = 125
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 82
Total Transactions Rolled Back = 59
Total Transactions Executed = 141
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_14_of_25/test.outputs/logTestWorkload1273033940
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload1273033940
--- FAIL: TestWorkload (26.89s)

Parameters:

  • attempt=1
  • run=14
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@fqazi
Copy link
Collaborator

fqazi commented Sep 9, 2024

Fixed via: #130351

@fqazi fqazi closed this as completed Sep 9, 2024
@fqazi fqazi reopened this Sep 9, 2024
@fqazi fqazi mentioned this issue Sep 9, 2024
Merged
@cockroach-teamcity
Copy link
Member

pkg/ccl/testccl/workload/schemachange/schemachange_test.TestWorkload failed on master @ 7571549d8772a087ab4577d1e770e02582ba5877:

        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193
        	            	  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202
        	            	  |   | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1635
        	            	  |   | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1836
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:718
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:829
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:170
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:829
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:829
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:137
        	            	  |   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:135
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:829
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:728
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228
        	            	  |   | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
        	            	  |   | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:913
        	            	  | Wraps: (2) single entry value with no default column id
        	            	  | Error types: (1) *withstack.withStack (2) *errutil.leafError (SQLSTATE XXUUU)
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pgconn.PgError
        	Test:       	TestWorkload
Schema Workload Stats
Total Schema Statements Executed = 39
Total Schema Statements Succeeded = 39
Total Schema Statement Expected Failures = 0
Total Transactions Committed = 27
Total Transactions Rolled Back = 19
Total Transactions Executed = 46
    schema_change_external_test.go:99: backup, tracing data, and system table dumps in /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/testlogs/pkg/ccl/testccl/workload/schemachange/schemachange_test/run_5_of_25/test.outputs/logTestWorkload650788707
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestWorkload650788707
--- FAIL: TestWorkload (15.74s)

Parameters:

  • attempt=1
  • run=5
  • shard=1
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

craig bot pushed a commit that referenced this issue Sep 10, 2024
@fqazi
Merge #130361
1e9b4bc 
130361: workload/schemachange: add unique constraint hits duplicate key errors r=fqazi a=fqazi

Previously, the ALTER TABLE ADD UNIQUE CONSTRAINT operation could encounter duplicate key errors. This would occur because our scan query was not exhaustive enough. To address this and to help stabilize the workload, we will allow unique violations to be potential commit errors.

Fixes: #129909

Release note: None

Co-authored-by: Faizan Qazi <faizan@cockroachlabs.com>
@craig craig bot closed this as completed in 84d1d18 Sep 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fqazi fqazi

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

workload/schemachange: add unique constraint hits duplicate key errors fqazi/cockroach
3 participants
@rafiss @fqazi @cockroach-teamcity