cli: TestHalfOnlineLossOfQuorumRecovery failed

[cockroach-teamcity]

cli.TestHalfOnlineLossOfQuorumRecovery failed on master @ 93ad913106b6f0f6ec98bc2cfa788ff6d8085bd4:

=== RUN   TestHalfOnlineLossOfQuorumRecovery
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestHalfOnlineLossOfQuorumRecovery557329092
    test_log_scope.go:81: use -show-logs to present logs inline
[debug recover make-plan --confirm=y --certs-dir=test_certs --host=127.0.0.1:34717 --plan=/var/lib/engflow/worker/work/3/exec/_tmp/2dcc0a708d6e811d98e64cdd1ab12ba9/TestHalfOnlineLossOfQuorumRecovery916069754/recovery-plan.json]
    debug_recover_loss_of_quorum_test.go:535: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/cli/debug_recover_loss_of_quorum_test.go:535
        	Error:      	"debug recover make-plan --confirm=y --certs-dir=test_certs --host=127.0.0.1:34717 --plan=/var/lib/engflow/worker/work/3/exec/_tmp/2dcc0a708d6e811d98e64cdd1ab12ba9/TestHalfOnlineLossOfQuorumRecovery916069754/recovery-plan.json\nERROR: failed to retrieve replica info from cluster: rpc error: code = Unknown desc = failed retrieving replicas from node n1 during fan-out: recv msg error: grpc: key /Local/RangeID/1/r/RangeGCThreshold does not have /Local/Range prefix [code 2/Unknown]\n" does not contain "- node n1"
        	Test:       	TestHalfOnlineLossOfQuorumRecovery
        	Messages:   	planner didn't provide correct apply instructions
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestHalfOnlineLossOfQuorumRecovery557329092
--- FAIL: TestHalfOnlineLossOfQuorumRecovery (31.38s)

Parameters:

• attempt=1
• race=true
• run=2
• shard=14

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-38861

[cockroach-teamcity]

cli.TestHalfOnlineLossOfQuorumRecovery failed on master @ d146ecff6f687e438706cf63591cafca60cc116d:

=== RUN   TestHalfOnlineLossOfQuorumRecovery
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestHalfOnlineLossOfQuorumRecovery3317157959
    test_log_scope.go:81: use -show-logs to present logs inline
[debug recover make-plan --confirm=y --certs-dir=test_certs --host=127.0.0.1:43875 --plan=/var/lib/engflow/worker/work/2/exec/_tmp/2dcc0a708d6e811d98e64cdd1ab12ba9/TestHalfOnlineLossOfQuorumRecovery1639678948/recovery-plan.json]
    debug_recover_loss_of_quorum_test.go:535: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/cli/debug_recover_loss_of_quorum_test.go:535
        	Error:      	"debug recover make-plan --confirm=y --certs-dir=test_certs --host=127.0.0.1:43875 --plan=/var/lib/engflow/worker/work/2/exec/_tmp/2dcc0a708d6e811d98e64cdd1ab12ba9/TestHalfOnlineLossOfQuorumRecovery1639678948/recovery-plan.json\nERROR: failed to retrieve replica info from cluster: rpc error: code = Unknown desc = failed retrieving replicas from node n1 during fan-out: recv msg error: grpc: key /Local/RangeID/1/r/RangeGCThreshold does not have /Local/Range prefix [code 2/Unknown]\n" does not contain "- node n1"
        	Test:       	TestHalfOnlineLossOfQuorumRecovery
        	Messages:   	planner didn't provide correct apply instructions
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestHalfOnlineLossOfQuorumRecovery3317157959
--- FAIL: TestHalfOnlineLossOfQuorumRecovery (46.72s)

Parameters:

• attempt=1
• race=true
• run=2
• shard=14

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

cli.TestHalfOnlineLossOfQuorumRecovery failed on master @ 0ea8b622c5354a891cfb867d8d872a0746ba77b3:

=== RUN   TestHalfOnlineLossOfQuorumRecovery
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestHalfOnlineLossOfQuorumRecovery4224013447
    test_log_scope.go:81: use -show-logs to present logs inline
[debug recover make-plan --confirm=y --certs-dir=test_certs --host=127.0.0.1:37559 --plan=/var/lib/engflow/worker/work/0/exec/_tmp/2dcc0a708d6e811d98e64cdd1ab12ba9/TestHalfOnlineLossOfQuorumRecovery3625772260/recovery-plan.json]
    debug_recover_loss_of_quorum_test.go:535: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/cli/debug_recover_loss_of_quorum_test.go:535
        	Error:      	"debug recover make-plan --confirm=y --certs-dir=test_certs --host=127.0.0.1:37559 --plan=/var/lib/engflow/worker/work/0/exec/_tmp/2dcc0a708d6e811d98e64cdd1ab12ba9/TestHalfOnlineLossOfQuorumRecovery3625772260/recovery-plan.json\nERROR: failed to retrieve replica info from cluster: rpc error: code = Unknown desc = failed retrieving replicas from node n1 during fan-out: recv msg error: grpc: key /Local/RangeID/1/r/RangeGCThreshold does not have /Local/Range prefix [code 2/Unknown]\n" does not contain "- node n1"
        	Test:       	TestHalfOnlineLossOfQuorumRecovery
        	Messages:   	planner didn't provide correct apply instructions
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestHalfOnlineLossOfQuorumRecovery4224013447
--- FAIL: TestHalfOnlineLossOfQuorumRecovery (32.06s)

Parameters:

• attempt=1
• race=true
• run=2
• shard=14

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

[rickystewart]

Maybe this should be skipped under race?

[cockroach-teamcity]

cli.TestHalfOnlineLossOfQuorumRecovery failed on master @ e5f5597b1c1570fff03ee6dfd4a1e8be65c0e008:

=== RUN   TestHalfOnlineLossOfQuorumRecovery
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestHalfOnlineLossOfQuorumRecovery1927781118
    test_log_scope.go:81: use -show-logs to present logs inline
[debug recover make-plan --confirm=y --certs-dir=test_certs --host=127.0.0.1:40383 --plan=/var/lib/engflow/worker/work/2/exec/_tmp/627c07b75a1b49827ad9adf3e9f30827/TestHalfOnlineLossOfQuorumRecovery4013921476/recovery-plan.json]
    debug_recover_loss_of_quorum_test.go:535: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/cli/debug_recover_loss_of_quorum_test.go:535
        	Error:      	"debug recover make-plan --confirm=y --certs-dir=test_certs --host=127.0.0.1:40383 --plan=/var/lib/engflow/worker/work/2/exec/_tmp/627c07b75a1b49827ad9adf3e9f30827/TestHalfOnlineLossOfQuorumRecovery4013921476/recovery-plan.json\nERROR: failed to retrieve replica info from cluster: rpc error: code = Unknown desc = failed retrieving replicas from node n1 during fan-out: recv msg error: grpc: key /Local/RangeID/1/r/RangeGCThreshold does not have /Local/Range prefix [code 2/Unknown]\n" does not contain "- node n1"
        	Test:       	TestHalfOnlineLossOfQuorumRecovery
        	Messages:   	planner didn't provide correct apply instructions
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestHalfOnlineLossOfQuorumRecovery1927781118
--- FAIL: TestHalfOnlineLossOfQuorumRecovery (32.19s)

Parameters:

• attempt=1
• race=true
• run=3
• shard=14

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!