Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql: TestBackfillWithProtectedTS failed #139814

Open
cockroach-teamcity opened this issue Jan 25, 2025 · 23 comments
Open

sql: TestBackfillWithProtectedTS failed #139814

cockroach-teamcity opened this issue Jan 25, 2025 · 23 comments
Assignees
@rafiss
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-1 Issues/test failures with a fix SLA of 1 month T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jan 25, 2025 •
edited by cockroach-jira-scripts
Loading

sql.TestBackfillWithProtectedTS failed on master @ 12a979cd5f2ac1d37502a1353596a1d77751ab5d:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS3546642436
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS3546642436
--- FAIL: TestBackfillWithProtectedTS (500.31s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1737812360.241467438,0 must be after replica GC threshold 1737812367.650912440,0 (r78: /Table/107{-/2})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: could not read jobs table: context canceled
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1737812360.241467438,0 must be after replica GC threshold 1737812367.650912440,0 (r78: /Table/107{-/2})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1737812360.241467438,0 must be after replica GC threshold 1737812367.650912440,0 (r78: /Table/107{-/2})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (439.84s)

Parameters:

  • attempt=1
  • run=6
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-46832
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Jan 25, 2025
This was referenced Jan 26, 2025
Open
Closed
@rafiss
Copy link
Collaborator

rafiss commented Jan 27, 2025

this includes #139774 which was supposed to fix this. i'll investigate more.

@rafiss rafiss self-assigned this Jan 27, 2025
@rafiss rafiss added P-1 Issues/test failures with a fix SLA of 1 month and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jan 27, 2025
@github-actions github-actions bot mentioned this issue Jan 27, 2025
Open
@cockroach-teamcity cockroach-teamcity mentioned this issue Jan 27, 2025
Open
@github-actions GitHub Actions
Copy link

sql.TestBackfillWithProtectedTS failed on master @ de704e269f52c6cd19f2fd72fcb44105c5e44a3e:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS2000592110
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS2000592110
--- FAIL: TestBackfillWithProtectedTS (484.03s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738086754.320884245,0 must be after replica GC threshold 1738086783.581844998,0 (r79: /{Table/107/3-Max})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: expected error was not hit
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738086754.320884245,0 must be after replica GC threshold 1738086783.581844998,0 (r79: /{Table/107/3-Max})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738086754.320884245,0 must be after replica GC threshold 1738086783.581844998,0 (r79: /{Table/107/3-Max})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (434.30s)

Parameters:

  • attempt=1
  • run=1
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Jan 28, 2025
Open
@cockroach-teamcity
Copy link
Member Author

sql.TestBackfillWithProtectedTS failed with artifacts on master @ aba435dba18db3094a191166212c951e25b5be35:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/ebab9da743cf8d3e9fbccfacc75c4814/logTestBackfillWithProtectedTS1516674431
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/ebab9da743cf8d3e9fbccfacc75c4814/logTestBackfillWithProtectedTS1516674431
--- FAIL: TestBackfillWithProtectedTS (473.07s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738100967.884760635,0 must be after replica GC threshold 1738101006.109462831,0 (r79: /{Table/107/3-Max})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: expected error was not hit
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738100967.884760635,0 must be after replica GC threshold 1738101006.109462831,0 (r79: /{Table/107/3-Max})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_arm64.s:1222
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738100967.884760635,0 must be after replica GC threshold 1738101006.109462831,0 (r79: /{Table/107/3-Max})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (442.19s)
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@github-actions GitHub Actions
Copy link

sql.TestBackfillWithProtectedTS failed on master @ 630c342dc5550f55e2303e7937d56c4837006544:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS1115719195
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS1115719195
--- FAIL: TestBackfillWithProtectedTS (482.52s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738107116.534334494,0 must be after replica GC threshold 1738107151.493435744,0 (r79: /{Table/107/3-Max})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: expected error was not hit
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738107116.534334494,0 must be after replica GC threshold 1738107151.493435744,0 (r79: /{Table/107/3-Max})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738107116.534334494,0 must be after replica GC threshold 1738107151.493435744,0 (r79: /{Table/107/3-Max})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (442.33s)

Parameters:

  • attempt=1
  • run=1
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@github-actions GitHub Actions
Copy link

sql.TestBackfillWithProtectedTS failed on master @ f6a6226821bbf5999381baa8659b6d7d93b3a526:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS949771655
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS949771655
--- FAIL: TestBackfillWithProtectedTS (472.76s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738120902.157271380,0 must be after replica GC threshold 1738120912.571689878,0 (r79: /{Table/107/3-Max})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: expected error was not hit
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738120902.157271380,0 must be after replica GC threshold 1738120912.571689878,0 (r79: /{Table/107/3-Max})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738120902.157271380,0 must be after replica GC threshold 1738120912.571689878,0 (r79: /{Table/107/3-Max})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (419.98s)

Parameters:

  • attempt=1
  • run=1
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Jan 29, 2025
Open
@cockroach-teamcity
Copy link
Member Author

sql.TestBackfillWithProtectedTS failed on master @ f6a6226821bbf5999381baa8659b6d7d93b3a526:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS3656757062
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS3656757062
--- FAIL: TestBackfillWithProtectedTS (498.17s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738133331.204352610,0 must be after replica GC threshold 1738133343.817533903,0 (r78: /Table/107{-/2})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: could not read jobs table: context canceled
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738133331.204352610,0 must be after replica GC threshold 1738133343.817533903,0 (r78: /Table/107{-/2})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738133331.204352610,0 must be after replica GC threshold 1738133343.817533903,0 (r78: /Table/107{-/2})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (442.84s)

Parameters:

  • attempt=1
  • run=23
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@github-actions GitHub Actions
Copy link

sql.TestBackfillWithProtectedTS failed on master @ 8655f5db5fe8ff856d90f71211a3158f60f1769f:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS3887558240
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS3887558240
--- FAIL: TestBackfillWithProtectedTS (499.09s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738205590.177857198,0 must be after replica GC threshold 1738205595.478597858,0 (r77: /Table/107{-/2})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: could not read jobs table: context canceled
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738205590.177857198,0 must be after replica GC threshold 1738205595.478597858,0 (r77: /Table/107{-/2})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738205590.177857198,0 must be after replica GC threshold 1738205595.478597858,0 (r77: /Table/107{-/2})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (438.62s)

Parameters:

  • attempt=1
  • run=1
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Jan 30, 2025
Closed
@cockroach-teamcity
Copy link
Member Author

sql.TestBackfillWithProtectedTS failed on master @ 8655f5db5fe8ff856d90f71211a3158f60f1769f:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS1110555499
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS1110555499
--- FAIL: TestBackfillWithProtectedTS (488.49s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738259733.504503667,0 must be after replica GC threshold 1738259740.400389608,0 (r77: /Table/107{-/2})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: could not read jobs table: context canceled
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738259733.504503667,0 must be after replica GC threshold 1738259740.400389608,0 (r77: /Table/107{-/2})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738259733.504503667,0 must be after replica GC threshold 1738259740.400389608,0 (r77: /Table/107{-/2})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (423.10s)

Parameters:

  • attempt=1
  • run=25
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@github-actions GitHub Actions
Copy link

sql.TestBackfillWithProtectedTS failed on master @ 5b931856d005214057f0703890ce3a77cdc21f30:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS3086749923
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS3086749923
--- FAIL: TestBackfillWithProtectedTS (510.29s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738286909.729878255,0 must be after replica GC threshold 1738286923.634941141,0 (r77: /Table/107{-/2})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: could not read jobs table: context canceled
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738286909.729878255,0 must be after replica GC threshold 1738286923.634941141,0 (r77: /Table/107{-/2})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738286909.729878255,0 must be after replica GC threshold 1738286923.634941141,0 (r77: /Table/107{-/2})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (459.33s)

Parameters:

  • attempt=1
  • run=1
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@rafiss rafiss mentioned this issue Jan 31, 2025
Closed
@github-actions GitHub Actions
Copy link

sql.TestBackfillWithProtectedTS failed on master @ 6a0a02510e1fc1b3cd6e7899d13e3f1a436a6e0c:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS810890404
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS810890404
--- FAIL: TestBackfillWithProtectedTS (481.69s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738352823.644230530,2 must be after replica GC threshold 1738352825.116785911,0 (r77: /Table/107{-/2})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: could not read jobs table: context canceled
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738352823.644230530,2 must be after replica GC threshold 1738352825.116785911,0 (r77: /Table/107{-/2})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738352823.644230530,2 must be after replica GC threshold 1738352825.116785911,0 (r77: /Table/107{-/2})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (414.12s)

Parameters:

  • attempt=1
  • run=1
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql.TestBackfillWithProtectedTS failed on master @ 507c68f8abb0c9a3f527c66cb42129c2cb096e7d:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS345770608
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS345770608
--- FAIL: TestBackfillWithProtectedTS (494.33s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738563801.384158058,0 must be after replica GC threshold 1738563819.259859579,0 (r78: /Table/107{-/2})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: could not read jobs table: context canceled
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738563801.384158058,0 must be after replica GC threshold 1738563819.259859579,0 (r78: /Table/107{-/2})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738563801.384158058,0 must be after replica GC threshold 1738563819.259859579,0 (r78: /Table/107{-/2})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (429.21s)

Parameters:

  • attempt=1
  • run=23
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 3, 2025

sql.TestBackfillWithProtectedTS failed on master @ 2f1ba08be9ededd11c8aa46e9bded07766f5d80a:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS102464772
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS102464772
--- FAIL: TestBackfillWithProtectedTS (476.42s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738624842.803643165,2 must be after replica GC threshold 1738624845.162120969,0 (r77: /Table/107{-/2})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: could not read jobs table: context canceled
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738624842.803643165,2 must be after replica GC threshold 1738624845.162120969,0 (r77: /Table/107{-/2})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738624842.803643165,2 must be after replica GC threshold 1738624845.162120969,0 (r77: /Table/107{-/2})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (425.28s)

Parameters:

  • attempt=1
  • run=1
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql.TestBackfillWithProtectedTS failed with artifacts on master @ 803ecba3e3e48195f256dff9aaea9975c1f0844b:

=== RUN   TestBackfillWithProtectedTS
I250204 18:53:48.450735 1 (gostd) rand.go:256  [-] 1  random seed: 9046371300302816388
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/ebab9da743cf8d3e9fbccfacc75c4814/logTestBackfillWithProtectedTS2777219903
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/ebab9da743cf8d3e9fbccfacc75c4814/logTestBackfillWithProtectedTS2777219903
--- FAIL: TestBackfillWithProtectedTS (473.50s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738695293.557825841,0 must be after replica GC threshold 1738695302.812017014,0 (r79: /{Table/107/3-Max})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: expected error was not hit
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738695293.557825841,0 must be after replica GC threshold 1738695302.812017014,0 (r79: /{Table/107/3-Max})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_arm64.s:1222
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738695293.557825841,0 must be after replica GC threshold 1738695302.812017014,0 (r79: /{Table/107/3-Max})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (416.53s)
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql.TestBackfillWithProtectedTS failed on master @ 82b1a2b174dcedfd23b0a3a40c6069143f4b038f:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS2697217821
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS2697217821
--- FAIL: TestBackfillWithProtectedTS (491.31s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738701097.357997094,0 must be after replica GC threshold 1738701132.289629257,0 (r77: /Table/107{-/2})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: could not read jobs table: context canceled
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738701097.357997094,0 must be after replica GC threshold 1738701132.289629257,0 (r77: /Table/107{-/2})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738701097.357997094,0 must be after replica GC threshold 1738701132.289629257,0 (r77: /Table/107{-/2})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (446.31s)

Parameters:

  • attempt=1
  • run=2
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 5, 2025

sql.TestBackfillWithProtectedTS failed on master @ bd25b0eba38bbc5c9878338b760bf1f9d5550d65:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS4253887434
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS4253887434
--- FAIL: TestBackfillWithProtectedTS (497.25s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738720778.543230503,0 must be after replica GC threshold 1738720817.182356227,0 (r77: /Table/107{-/2})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: could not read jobs table: context canceled
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738720778.543230503,0 must be after replica GC threshold 1738720817.182356227,0 (r77: /Table/107{-/2})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738720778.543230503,0 must be after replica GC threshold 1738720817.182356227,0 (r77: /Table/107{-/2})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (458.42s)

Parameters:

  • attempt=1
  • run=1
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

sql.TestBackfillWithProtectedTS failed on master @ d276833ff08ef11d7894b7b0124f1883d90a3377:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS3881173975
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS3881173975
--- FAIL: TestBackfillWithProtectedTS (479.63s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738736794.079181834,0 must be after replica GC threshold 1738736795.878992897,0 (r81: /Table/107/{1/1044205956791959553-2})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: could not read jobs table: context canceled
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738736794.079181834,0 must be after replica GC threshold 1738736795.878992897,0 (r81: /Table/107/{1/1044205956791959553-2})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738736794.079181834,0 must be after replica GC threshold 1738736795.878992897,0 (r81: /Table/107/{1/1044205956791959553-2})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (431.11s)

Parameters:

  • attempt=1
  • run=17
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 6, 2025

sql.TestBackfillWithProtectedTS failed on master @ 871fb9ff40888d77373b2da06c991e7f13d82789:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS2989156968
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS2989156968
--- FAIL: TestBackfillWithProtectedTS (538.25s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738828775.063686945,0 must be after replica GC threshold 1738828800.343003902,0 (r79: /{Table/107/3-Max})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: expected error was not hit
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738828775.063686945,0 must be after replica GC threshold 1738828800.343003902,0 (r79: /{Table/107/3-Max})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738828775.063686945,0 must be after replica GC threshold 1738828800.343003902,0 (r79: /{Table/107/3-Max})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (500.62s)

Parameters:

  • attempt=1
  • run=1
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 7, 2025

sql.TestBackfillWithProtectedTS failed on master @ c0b0b6878b382d493889187c94fa85c8d0702203:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS2795359375
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS2795359375
--- FAIL: TestBackfillWithProtectedTS (486.28s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738888570.367273440,0 must be after replica GC threshold 1738888601.357932824,0 (r79: /{Table/107/3-Max})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: expected error was not hit
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738888570.367273440,0 must be after replica GC threshold 1738888601.357932824,0 (r79: /{Table/107/3-Max})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738888570.367273440,0 must be after replica GC threshold 1738888601.357932824,0 (r79: /{Table/107/3-Max})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (448.82s)

Parameters:

  • attempt=1
  • run=1
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 7, 2025

sql.TestBackfillWithProtectedTS failed on master @ 0563ba6941e3c5409aac567f9f806c2f91303d7f:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS1652261176
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS1652261176
--- FAIL: TestBackfillWithProtectedTS (502.65s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738941780.095197840,0 must be after replica GC threshold 1738941816.885044205,0 (r79: /{Table/107/3-Max})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: expected error was not hit
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738941780.095197840,0 must be after replica GC threshold 1738941816.885044205,0 (r79: /{Table/107/3-Max})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738941780.095197840,0 must be after replica GC threshold 1738941816.885044205,0 (r79: /{Table/107/3-Max})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (449.64s)

Parameters:

  • attempt=1
  • run=1
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Feb 7, 2025
Open
@cockroach-teamcity
Copy link
Member Author

sql.TestBackfillWithProtectedTS failed on master @ 13d8c5191614530b50802f77081f15d2ce2ca705:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS1511645903
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS1511645903
--- FAIL: TestBackfillWithProtectedTS (492.20s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738960285.200871388,0 must be after replica GC threshold 1738960313.842443276,0 (r79: /{Table/107/3-Max})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: expected error was not hit
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738960285.200871388,0 must be after replica GC threshold 1738960313.842443276,0 (r79: /{Table/107/3-Max})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738960285.200871388,0 must be after replica GC threshold 1738960313.842443276,0 (r79: /{Table/107/3-Max})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (450.69s)

Parameters:

  • attempt=1
  • run=9
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 7, 2025

sql.TestBackfillWithProtectedTS failed on master @ 0effe73566771abc833f913507e760301b4ae976:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS3923157757
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS3923157757
--- FAIL: TestBackfillWithProtectedTS (465.42s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738968778.867122110,0 must be after replica GC threshold 1738968780.762657743,0 (r79: /{Table/107/3-Max})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: expected error was not hit
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738968778.867122110,0 must be after replica GC threshold 1738968780.762657743,0 (r79: /{Table/107/3-Max})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738968778.867122110,0 must be after replica GC threshold 1738968780.762657743,0 (r79: /{Table/107/3-Max})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (398.66s)

Parameters:

  • attempt=1
  • run=1
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 8, 2025

sql.TestBackfillWithProtectedTS failed on master @ dbb0baa860dc8f5edb6fdee6fecadda17cf5e867:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS2898991880
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS2898991880
--- FAIL: TestBackfillWithProtectedTS (489.63s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738976331.128787526,0 must be after replica GC threshold 1738976342.946427870,0 (r77: /Table/107{-/2})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: could not read jobs table: context canceled
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738976331.128787526,0 must be after replica GC threshold 1738976342.946427870,0 (r77: /Table/107{-/2})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738976331.128787526,0 must be after replica GC threshold 1738976342.946427870,0 (r77: /Table/107{-/2})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (438.39s)

Parameters:

  • attempt=1
  • run=1
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 8, 2025

sql.TestBackfillWithProtectedTS failed on master @ bbaa2e50b2fe789527aac09b99fa5eee432e7695:

=== RUN   TestBackfillWithProtectedTS
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestBackfillWithProtectedTS1832755074
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:141: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    backfill_protected_timestamp_test.go:526: -- test log scope end --
test logs left over in: outputs.zip/logTestBackfillWithProtectedTS1832755074
--- FAIL: TestBackfillWithProtectedTS (505.82s)
=== RUN   TestBackfillWithProtectedTS/create_index
    backfill_protected_timestamp_test.go:477: running backfill with PTS not setup early enough
    backfill_protected_timestamp_test.go:425: i=0 begin running operations in the middle of backfill
    backfill_protected_timestamp_test.go:429: i=0 finished running operations in the middle of backfill. retErr: failed to DELETE and INSERT: pq: batch timestamp 1738981807.425371308,0 must be after replica GC threshold 1738981825.501559955,0 (r77: /Table/107{-/2})
    backfill_protected_timestamp_test.go:474: backfill goroutine failed: could not read jobs table: context canceled
    backfill_protected_timestamp_test.go:515: 
        	Error Trace:	pkg/sql/backfill_protected_timestamp_test.go:515
        	Error:      	Received unexpected error:
        	            	failed to DELETE and INSERT: pq: batch timestamp 1738981807.425371308,0 must be after replica GC threshold 1738981825.501559955,0 (r77: /Table/107{-/2})
        	            	(1) attached stack trace
        	            	  -- stack trace:
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2.1
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:438
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.2
        	            	  | 	pkg/sql/backfill_protected_timestamp_test.go:463
        	            	  | github.com/cockroachdb/cockroach/pkg/sql_test.TestBackfillWithProtectedTS.func7.Group.GoCtx.4
        	            	  | 	pkg/util/ctxgroup/ctxgroup.go:189
        	            	  | golang.org/x/sync/errgroup.(*Group).Go.func1
        	            	  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
        	            	  | runtime.goexit
        	            	  | 	src/runtime/asm_amd64.s:1695
        	            	Wraps: (2) failed to DELETE and INSERT
        	            	Wraps: (3) pq: batch timestamp 1738981807.425371308,0 must be after replica GC threshold 1738981825.501559955,0 (r77: /Table/107{-/2})
        	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *pq.Error
        	Test:       	TestBackfillWithProtectedTS/create_index
    --- FAIL: TestBackfillWithProtectedTS/create_index (450.95s)

Parameters:

  • attempt=1
  • run=1
  • shard=14
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Feb 8, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rafiss rafiss

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-1 Issues/test failures with a fix SLA of 1 month T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

sql: deflake TestBackfillWithProtectedTS rafiss/cockroach
2 participants
@rafiss @cockroach-teamcity