Sentry: replica_proposal.go:252: log.Fatal: sequence identical for different leases, prevLease=repl=(n3,s3):2 seq=3 start=1716193792.589095066,0 exp=1716193801.803216642,0 pro=1716193795.803216642,0, ...

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5378300281/?referrer=webhooks_plugin

Panic Message:

replica_proposal.go:252: log.Fatal: sequence identical for different leases, prevLease=repl=(n3,s3):2 seq=3 start=1716193792.589095066,0 exp=1716193801.803216642,0 pro=1716193795.803216642,0, newLease=repl=(n2,s2):3 seq=3 start=1716193556.741357355,0 exp=1716193562.741282574,0 pro=1716193556.741282574,0
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).leasePostApplyLocked
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:252
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).applySnapshot
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go:668
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:934
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:465
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:343
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaftSnapshotRequest
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:406
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1091
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:211
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:322
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:208
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:382
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:263
  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.StreamServerInterceptor.func1
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:165
  | google.golang.org/grpc.getChainStreamHandler.func1
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1478
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:293
  | google.golang.org/grpc.getChainStreamHandler.func1
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1478
  | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:167
  | google.golang.org/grpc.getChainStreamHandler.func1
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1478
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:260
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:322
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:259
  | google.golang.org/grpc.chainStreamInterceptors.func1
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1469
  | google.golang.org/grpc.(*Server).processStreamingRPC
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1627
  | google.golang.org/grpc.(*Server).handleStream
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:1708
  | google.golang.org/grpc.(*Server).serveStreams.func1.2
  | 	google.golang.org/grpc/external/org_golang_google_grpc/server.go:965
  | runtime.goexit
  | 	GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) log.Fatal: sequence identical for different leases, prevLease=repl=(n3,s3):2 seq=3 start=1716193792.589095066,0 exp=1716193801.803216642,0 pro=1716193795.803216642,0, newLease=repl=(n2,s2):3 seq=3 start=1716193556.741357355,0 exp=1716193562.741282574,0 pro=1716193556.741282574,0
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: sequence identical for different leases, prevLease=repl=(n3,s3):2 seq=3 start=1716193792.589095066,0 exp=1716193801.803216642,0 pro=1716193795.803216642,0, newLease=repl=(n2,s2):3 seq=3 start=1716193556.741357355,0 exp=1716193562.741282574,0 pro=1716193556.741282574,0
replica_proposal.go:252: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/asm_amd64.s#L1593-L1595
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L964-L966
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1707-L1709
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1626-L1628
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1468-L1470
https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/rpc/pkg/rpc/context.go#L258-L260

cockroach/pkg/util/stop/stopper.go

Lines 321 to 323 in fceb4cf

	return f(ctx)
	}

https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/rpc/pkg/rpc/context.go#L259-L261
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1477-L1479
https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/rpc/pkg/rpc/auth.go#L166-L168
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1477-L1479
https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/rpc/pkg/rpc/context.go#L292-L294
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1477-L1479

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 164 to 166 in fceb4cf

	}
	err = handler(srv, ss)
	if err != nil {

https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go#L262-L264
https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go#L381-L383
https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L207-L209

cockroach/pkg/util/stop/stopper.go

Lines 321 to 323 in fceb4cf

	return f(ctx)
	}

https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L210-L212
https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go#L1090-L1092
https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L405-L407
https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L342-L344
https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L464-L466
https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L933-L935
https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go#L667-L669
https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go#L251-L253

GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.2 at line 965
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1708
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processStreamingRPC at line 1627
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainStreamInterceptors.func1 at line 1469
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func2 at line 259
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 322
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func2.1 at line 260
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1478
pkg/rpc/pkg/rpc/auth.go in pkg/rpc.kvAuth.streamInterceptor at line 167
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1478
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func4 at line 293
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.getChainStreamHandler.func1 at line 1478
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.StreamServerInterceptor.func1 at line 165
pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go in pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler at line 263
pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).RaftSnapshot at line 382
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot at line 208
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 322
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleSnapshot.func1 at line 211
pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go in pkg/kv/kvserver.(*Store).receiveSnapshot at line 1091
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest at line 406
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).withReplicaForRequest at line 343
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRaftSnapshotRequest.func1 at line 465
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 934
pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go in pkg/kv/kvserver.(*Replica).applySnapshot at line 668
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go in pkg/kv/kvserver.(*Replica).leasePostApplyLocked at line 252

Tags

Tag	Value
Command	server
Environment	v23.1.21
Go Version	go1.19.13
Platform	linux amd64
Distribution	CCL
Cockroach Release	v23.1.21
Cockroach SHA	fceb4cf
# of CPUs	8
# of Goroutines	432

Jira issue: CRDB-38876

[yuzefovich]

unactionable