The number of unprocessed events in channel polkadot-network-bridge exceeded 100000.

[ioannist]

Related to:
paritytech/polkadot#6904

Moonbeam fullnode 0.33 terminates when under heavy query load (for historical blocks) for 10-20 minutes.

Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:32 [Relaychain] The number of unprocessed events in channel `polkadot-network-bridge` exceeded 100000.
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]: The channel was created at:
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    0: sc_network::service::out_events::channel
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    1: <alloc::sync::Arc<sc_network::service::NetworkService<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::OpaqueExtrinsic>,primitive_types::H256>> as polkadot_network_bridge::network::Network>::event_stream
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    2: polkadot_overseer::spawn
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    3: polkadot_overseer::OverseerBuilder<polkadot_overseer::Init<S>,polkadot_overseer::Init<CandidateValidation>,polkadot_overseer::Init<PvfChecker>,polkadot_overseer::Init<CandidateBacking>,polkadot_overseer::Init<StatementDistribution>,polkadot_overseer::Init<AvailabilityDistribution>,polkadot_overseer::Init<AvailabilityRecovery>,polkadot_overseer::Init<BitfieldSigning>,polkadot_overseer::Init<BitfieldDistribution>,polkadot_overseer::Init<Provisioner>,polkadot_overseer::Init<RuntimeApi>,polkadot_overseer::Init<AvailabilityStore>,polkadot_overseer::Init<NetworkBridgeRx>,polkadot_overseer::Init<NetworkBridgeTx>,polkadot_overseer::Init<ChainApi>,polkadot_overseer::Init<CollationGeneration>,polkadot_overseer::Init<CollatorProtocol>,polkadot_overseer::Init<ApprovalDistribution>,polkadot_overseer::Init<ApprovalVoting>,polkadot_overseer::Init<GossipSupport>,polkadot_overseer::Init<DisputeCoordinator>,polkadot_overseer::Init<DisputeDistribution>,polkadot_overseer::Init<ChainSelection>,polkadot_overseer::Init<std::collections::hash::map::HashMap<primitive_types::H256,alloc::vec::Vec<futures_channel::oneshot::Sender<core::result::Result<(),polkadot_node_subsystem_types::errors::SubsystemError>>>>>,polkadot_overseer::Init<std::collections::hash::map::HashMap<primitive_types::H256,alloc::sync::Arc<polkadot_node_jaeger::spans::Span>>>,polkadot_overseer::Init<std::collections::hash::map::HashMap<primitive_types::H256,u32>>,polkadot_overseer::Init<SupportsParachains>,polkadot_overseer::Init<lru::LruCache<primitive_types::H256,()>>,polkadot_overseer::Init<polkadot_overseer::metrics::Metrics>>::build_with_connector
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    4: <polkadot_service::overseer::RealOverseerGen as polkadot_service::overseer::OverseerGen>::generate
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    5: polkadot_service::new_full
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    6: moonbeam_service::build_relay_chain_interface::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    7: moonbeam_service::start_node_impl::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    8: moonbeam_cli::command::run::{{closure}}::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    9: sc_cli::runner::Runner<C>::run_node_until_exit
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   10: moonbeam_cli::command::run
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   11: moonbeam::main
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   12: std::sys_common::backtrace::__rust_begin_short_backtrace
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   13: std::rt::lang_start::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   14: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:              at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/ops/function.rs:284:13
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   15: std::panicking::try::do_call
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:              at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panicking.rs:500:40
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   16: std::panicking::try
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:              at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panicking.rs:464:19
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   17: std::panic::catch_unwind
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:              at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panic.rs:142:14
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   18: std::rt::lang_start_internal::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:              at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/rt.rs:148:48
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   19: std::panicking::try::do_call
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:              at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panicking.rs:500:40
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   20: std::panicking::try
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:              at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panicking.rs:464:19
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   21: std::panic::catch_unwind
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:              at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panic.rs:142:14
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   22: std::rt::lang_start_internal
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:              at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/rt.rs:148:20
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   23: main
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   24: <unknown>
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   25: __libc_start_main
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   26: _start
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:  The last event was sent from:
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    0: sc_network::service::out_events::OutChannels::send::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    1: sc_network::service::out_events::OutChannels::send
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    2: sc_network::service::NetworkWorker<B,H>::handle_swarm_event
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    3: sc_network::service::NetworkWorker<B,H>::next_action::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    4: <core::pin::Pin<P> as core::future::future::Future>::poll
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    5: sc_service::build_network_future::{{closure}}::{{closure}}::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    6: sc_service::builder::build_network::{{closure}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    7: tokio::runtime::task::raw::poll
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    8: std::sys_common::backtrace::__rust_begin_short_backtrace
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:    9: core::ops::function::FnOnce::call_once{{vtable.shim}}
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   10: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:              at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/alloc/src/boxed.rs:1985:9
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   11: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:              at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/alloc/src/boxed.rs:1985:9
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   12: std::sys::unix::thread::Thread::new::thread_start
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:              at ./rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/sys/unix/thread.rs:108:17
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   13: <unknown>
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:   14: <unknown>
Oct 30 12:27:32 stakebaby-chalandri moonbeam[327057]:
Oct 30 12:27:33 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:33 [Relaychain] ✨ Imported #17948289 (0xba0d…3746)
Oct 30 12:27:36 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:36 [Relaychain] 💤 Idle (14 peers), best: #17948289 (0xba0d…3746), finalized #17948286 (0x115e…8a1a), ⬇ 663.0kiB/s ⬆ 9.8kiB/s
Oct 30 12:27:36 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:36 [🌗] ⚙️  Preparing  0.0 bps, target=#4759845 (3 peers), best: #4759823 (0xf164…3409), finalized #4759803 (0x4f41…067e), ⬇ 5.8kiB/s ⬆ 0.7kiB/s
Oct 30 12:27:41 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:41 [Relaychain] 💤 Idle (18 peers), best: #17948289 (0xba0d…3746), finalized #17948287 (0x6224…01c5), ⬇ 116.7kiB/s ⬆ 7.3kiB/s
Oct 30 12:27:41 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:41 [🌗] ⚙️  Preparing  0.0 bps, target=#4759845 (3 peers), best: #4759823 (0xf164…3409), finalized #4759803 (0x4f41…067e), ⬇ 9.4kiB/s ⬆ 0.1kiB/s
Oct 30 12:27:42 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:42 [Relaychain] cannot query the runtime API version: Api called for an unknown Block: State already discarded for 0x844abf4aa5868e67ba4aba2d97571d55976183ddc10da872c4de94abd1773126
Oct 30 12:27:42 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:42 [Relaychain] ✨ Imported #17948290 (0x0e02…9936)
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] error receiving message from subsystem context: Generated(Context("Signal channel is terminated and empty.")) err=Generated(Context("Signal channel is terminated and empty."))
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] subsystem exited with error subsystem="availability-recovery-subsystem" err=FromOrigin { origin: "availability-recovery", source: Generated(Context("Signal channel is terminated and empty.")) }
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] subsystem exited with error subsystem="provisioner-subsystem" err=FromOrigin { origin: "provisioner", source: OverseerExited(Generated(Context("Signal channel is terminated and empty."))) }
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] subsystem exited with error subsystem="network-bridge-rx-subsystem" err=FromOrigin { origin: "network-bridge", source: SubsystemError(Generated(Context("Signal channel is terminated and empty."))) }
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] subsystem exited with error subsystem="network-bridge-tx-subsystem" err=FromOrigin { origin: "network-bridge", source: SubsystemError(Generated(Context("Signal channel is terminated and empty."))) }
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] err=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] Overseer exited with error err=Generated(SubsystemStalled("approval-distribution-subsystem"))
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] subsystem exited with error subsystem="chain-api-subsystem" err=FromOrigin { origin: "chain-api", source: Generated(Context("Signal channel is terminated and empty.")) }
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] subsystem exited with error subsystem="bitfield-signing-subsystem" err=FromOrigin { origin: "bitfield-signing", source: Generated(Context("Signal channel is terminated and empty.")) }
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] Essential task `overseer` failed. Shutting down service.
Oct 30 12:27:45 stakebaby-chalandri moonbeam[327057]: 2023-10-30 12:27:45 [Relaychain] subsystem exited with error subsystem="candidate-validation-subsystem" err=FromOrigin { origin: "candidate-validation", source: Generated(Context("Signal channel is terminated and empty.")) }
Oct 30 12:27:57 stakebaby-chalandri moonbeam[327057]: Error: Service(Other("Essential task failed."))
Oct 30 12:27:57 stakebaby-chalandri systemd[1]: moonbeam.service: Main process exited, code=exited, status=1/FAILURE
Oct 30 12:27:57 stakebaby-chalandri systemd[1]: moonbeam.service: Failed with result 'exit-code'.

Runtime settings:

ExecStart=/var/lib/moonbeam-data/moonbeam-skylake \
     --execution wasm \
     --wasm-execution compiled \
     --state-pruning=archive \
     --port 30343 \
     --rpc-port 9943 \
     --db-cache 4000 \
     --base-path /ssd-pool/chains/moonbeam \
     --chain moonbeam \
     --max-runtime-instances 256 \
     --trie-cache-size 573741824 \
     --rpc-max-request-size 100 \
     --rpc-max-response-size 100 \
     --rpc-max-connections 300 \
     --in-peers 3 \
     --out-peers 3 \
     --max-parallel-downloads 2 \
     --no-hardware-benchmarks \
     --no-grandpa \
     --no-telemetry \
     --no-prometheus \