Stateless node bad performance

[LesnyRumcajs]

Issue summary

We've been running a stateless node over the weekend (May 18th - May 20th). The results are not satisfactory, given the overall connectivity and resource usage.

1. Connectivity looks poor looking at the bootstrapper dashboard. - Forest node is /dns4/bootstrap-mainnet-0.chainsafe-fil.io/tcp/34000/p2p/12D3KooWKKkCZbcigsWTEu1cgNetNbZJqeNtysRtFpq7DTqw3eqH

2. Resource usage is high:
   

We expected the resource usage for a stateless node to be around ~500 MiB at most, but it's higher. Looking at the trend, there may be a memory leak somewhere. The CPU usage is also high - higher than in full-node Lotus.

3. Lots of warnings in logs (which may contribute to resource usage, but also indicate a genuine issue on the p2p layer of Forest)

2024-05-20 10:16:04.127	
2024-05-20T08:16:04.127165Z  WARN forest_filecoin::libp2p_bitswap::request_manager: sending on a closed channel
2024-05-20 10:16:04.126	
2024-05-20T08:16:04.126292Z  WARN forest_filecoin::libp2p_bitswap::request_manager: sending on a closed channel
2024-05-20 10:15:40.039	
2024-05-20T08:15:40.039723Z  WARN libp2p_request_response::handler: Dropping inbound stream because we are at capacity
2024-05-20 10:15:35.344	
2024-05-20T08:15:35.344432Z  WARN forest_filecoin::libp2p::service: Ping timeout: 12D3KooWJsKmHDxvbH5h22Qy8k3hFVHdQ3hnodJsPWqocFnoNhMF
2024-05-20 10:15:34.770	
2024-05-20T08:15:34.770194Z  WARN forest_filecoin::libp2p::service: Ping timeout: 12D3KooWPrxMyMKyb9Zn6WwWjtwFucYJBsCMYHpvLuhutktjDfRK
2024-05-20 10:14:53.679	
2024-05-20T08:14:53.678135Z  WARN forest_filecoin::libp2p::service: Error sending chain exchange response: ChainExchangeResponse { status: Success, message: "Success", chain: [TipsetBundle { blocks: [CachingBlockHeader { uncached: RawBlockHeader { miner_address: Address(Address("f01152731")), ticket: Some(Ticket { vrfproof: VRFProof([142, 82, 255, 20, 79, 214, 175, 230, 1, 116, 137, 57, 106, 108, 102, 251, 145, 146, 238, 12, 197, 108, 84, 4, 159, 76, 234, 163, 169, 206, 171, 75, 73, 104, 13, 226, 47, 146, 197, 206, 83, 252, 59, 222, 123, 85, 105, 38, 9, 26, 0, 28, 241, 156, 227, 44, 137, 129, 181, 237, 34, 238, 179, 137, 90, 194, 64, 225, 53, 18, 112, 45, 206, 216, 6, 101, 175, 134, 47, 18, 100, 21, 184, 20, 150, 22, 158, 238, 230, 246, 135, 251, 111, 133, 87, 190]) }), election_proof: Some(ElectionProof { win_count: 6, vrfproof: VRFProof([130, 120, 180, 164, 54, 186, 141, 190, 44, 128, 159, 191, 110, 74, 45, 234, 102, 181, 54, 233, 114, 102, 205, 218, 201, 119, 5, 220, 156, 214, 20, 234, 225, 197, 136, 149, 39, 43, 12, 8, 204, 199, 44, 246, 26, 45, 36, 215, 24, 240, 21, 228, 33, 97, 228, 128, 236, 62, 161, 177, 30, 75, 50, 200, 135, 53, 159, 200, 63, 66, 13, 45, 163, 46, 219, 51, 170, 209, 254, 45, 227, 153, 2, 150, 95, 106, 120, 122, 167, 123, 54, 5, 132, 36, 80, 246]) }), beacon_entries: [BeaconEntry { round: 6953272, signature: [140, 173, 230, 123, 43, 91, 94, 102, 3, 193, 113, 223, 71, 86, 252, 15, 143, 78, 70, 44, 84, 11, 160, 44, 79, 180, 98, 133, 184, 184, 196, 171, 172, 65, 36, 75, 24, 16, 163, 69, 32, 97, 240, 199, 146, 213, 21, 229] }], winning_post_proof: [PoStProof(PoStProof { post_proof: StackedDRGWinning32GiBV1, proof_bytes: [153, 21, 67, 198, 198, 163, 237, 241, 79, 63, 189, 12, 175, 151, 42, 17, 41, 29, 17, 136, 251, 7, 183, 182, 210, 12, 1, 176, 193, 198, 65, 46, 136, 38, 95, 30, 7, 100, 181, 72, 108, 18, 52, 30, 59, 57, 106, 210, 146, 155, 202, 227, 118, 196, 5, 55, 167, 83, 117, 232, 108, 170, 144, 222, 44, 110, 13, 236, 109, 39, 60, 228, 232, 59, 50, 151, 32, 49, 20, 248, 253, 75, 173, 115, 6, 158, 5, 102, 202, 227, 5, 196, 152, 33, 103, 170, 21, 178, 182, 98, 119, 18, 65, 71, 151, 104, 111, 148, 216, 183, 176, 201, 176, 104, 36, 52, 100, 53, 199, 126, 53, 53, 55, 184, 248, 37, 198, 192, 96, 169, 244, 148, 247, 48, 42, 140, 85, 176, 43, 186, 86, 90, 7, 58, 172, 224, 66, 146, 211, 147, 116, 22, 190, 28, 12, 16, 144, 36, 155, 110, 210, 14, 170, 117, 29, 148, 149, 209, 177, 198, 202, 188, 129, 25, 46, 95, 47, 63, 91, 72, 67, 236, 51, 109, 147, 165, 14, 202, 251, 82, 132, 56] })], parents: TipsetKey(SmallCidNonEmptyVec(NonEmpty { head: Inline(CidV1DagCborBlake2b256 { digest: [120, 163, 74, 105, 231, 132, 10, 205, 248, 163, 189, 65, 137, 238, 240, 184, 244, 45, 144, 236, 21, 38, 51, 209, 20, 253, 204, 140, 177, 219, 94, 166] }), tail: [] })), weight: 91004666363, epoch: 3845227, state_root: Cid(bafy2bzacebchxoezb5ymoc5l7vw32b4ntpax62wp23agydo6ceoye6yrbxlgc), message_receipts: Cid(bafy2bzacebpwg6ky6x5vdvcqcdkotucqxxp4yfm2kda56cfow7kysvyjkocga), messages: Cid(bafy2bzacecenvc3esfsssqdndyetcracrnnpjlb3rqeopbiqshhsfat7ykbiy), bls_aggregate: Some(Signature { sig_type: Bls, bytes: [183, 250, 98, 134, 174, 46, 129, 7, 46, 203, 163, 24, 12, 219, 224, 226, 19, 220, 137, 24, 78, 151, 10, 182, 238, 200, 72, 82, 45, 129, 219, 56, 174, 239, 18, 0, 218, 12, 147, 3, 130, 24, 120, 92, 7, 230, 119, 243, 2, 199, 116, 95, 126, 28, 105, 98, 178, 32, 145, 80, 216, 136, 215, 137, 54, 198, 44, 167, 85, 11, 21, 186, 171, 81, 161, 69, 145, 97, 184, 77, 242, 75, 46, 191, 225, 36, 223, 79, 202, 119, 92, 191, 199, 202, 210, 243] }), timestamp: 1713663210, signature: Some(Signature { sig_type: Bls, bytes: [135, 61, 51, 133, 229, 204, 51, 83, 57, 101, 209, 200, 112, 80, 168, 92, 168, 42, 55, 97, 24, 172, 42, 114, 13, 40, 198, 254, 250, 14, 143, 167, 238, 168, 161, 115, 18, 81, 2, 220, 211, 176, 214, 130, 13, 155, 166, 215, 9, 4, 168, 92, 119, 120, 51, 240, 82, 133, 181, 193, 130, 191, 28, 139, 142, 207, 157, 52, 191, 133, 13, 214, 26, 252, 85, 243, 2, 190, 15, 98, 0, 188, 4, 16, 40, 181, 128, 58, 234, 43, 84, 78, 189, 110, 99, 208] }), fork_signal: 0, parent_base_fee: TokenAmount(0.0000000000000001) }, cid: OnceCell(Cid(bafy2bzaceci5b5j5hpiiptkjy6dzvd32pu5hbfcg36li76a7m5kjail5qov6o)), has_ever_been_verified_against_any_signature: false }], messages: Some(CompactedMessages { bls_msgs: [Message { version: 0, from: Address(Address("f3q76alposs2hxxg2oavhhk6yxscyk2vuvr7pr7ql6eutlosa7vtgqihqg4uvgschw77sjltu2l3gfjoyyaeia")), to: Address(Address("f02097936")), sequence: 6953, value: TokenAmount(0.0), method_num: 5, params: RawBytes { 85068182004081820d58c0b50ece5857c9ee6b2b4d8c8d42c354aa3d3965d07e23c9bd3f3cf428ac68b1b7b81504e4fbdc4936e16cc8920755dcecade46df13ab7fff985e26e82081fc727f92b72b5fb46e1b6215585a72ca1bfe47847d90be66e9dffbe3469a30858af54123059f2efedb7d4a466090b499c995472d9b21a4ddafca04bd36d1037490ed20b76cfc05166132181847b69804a8d2fa874834c43f5a86e5759b0af2ce831967bc0912ebc02da903facf235f9f2ef33a00a54a5462aa24773905e1b414f69b51a003aac525820868f10a3c896b98a72f033a28ee8f1819a5f0f8d33e94669990d01c1cd7fce3e }, gas_limit: 26936375, gas_fee_cap: TokenAmount(0.000000185622601407), gas_premium: TokenAmount(0.000000000000100534) }, Message { version: 0, from: Address(Address("f3umn5xe4tmeum7gkomfdpnmznb6zqw6odccyg726j4uxffljysarpxm4eensbvezi5poo3k4whlfoy7gf3xka")), to: Address(Address("f05")), sequence: 4626, value: TokenAmount(0.0), method_num: 4, params: RawBytes { 8192828bd82a5828000181e20392202054a6a59d428868f115fa9306b5b2928b599da3f9c9ccbd92a1c5865b1c765a0f1b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b626166796265696671366d706e6e78616e333366707972716c796b34647a70746332656569736132367633666f7276647375346d737a6b616b6a341a003b13291a0042fc2940480019922c0c974c1440584201b9b842e92e839e9d92944e508966775f97778f8d58f7ef0573417eacaadcb70234fc936845998edd2e2f8d2ab79cc9f9f630e85d627749739ff874c8a7850b9401828bd82a5828000181e20392202033b025b39908f9937f541a196884dd9d850004074bc4e91296973b9357d3a30f1b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b6261667962656965716470797974746e6c786f35727073706f6f66636f337334756c716d65656c6871786c7074746779326c7267676234343277751a003b13291a0042fc2940480019922c0c974c144058420109a7150c6932582e6b93a8154cab270a3df27f44aaf0b3b69f67a3dadceae8d1302afdba2d49a25b50c8749e618d6b8010af6c469b976db679eca638944f008f01828bd82a5828000181e203922020e97b645e2f884e1dfe1076369b201fcb3642b018386632cdd4e770971021430e1b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b6261667962656962707769696a63776c7934687679716c69747374617279336a6862356f3637356d687175746c6770646a726f666f6f6a6a7a35691a003b13291a0042fc2940480019922c0c974c1440584201c6f2b0dd85c9337a04805789648e5feddfb7f193c70ca27fe5218e996ee3df703629997e5b26e91be32a3f507862faa8167cc1d3ca386b723641e083bad84c3500828bd82a5828000181e20392202055dfec03da6c7e1bd001eef727d93ebff00183d7805eeb6baecc3c818779540e1b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b62616679626569633462646467716264706e6977776232333467756179327964337174613436726b68347a707a69726c6b736b67683368687a62341a003b13291a0042fc2940480019922c0c974c14405842011c1ac623d1abf9179f557cd9c819c416478752d0f1188384ce731d1f18cb09ce65fdfadd5e46592a7381a7fabe408eaf154a3e3b7accf45e5509cab0f818af8200828bd82a5828000181e203922020e93e62b28791692983246702eb76206be9f8c83aacc29ddd45306a1604276c271b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b62616679626569686e3432767471693362776a6b6f33627667797a6b6d7166663676376573797835796c78707275326f667a6b746372376c6866341a003b13291a0042fc2940480019922c0c974c144058420148bc7dfe0a493c7fd8ffc940a556ee16eadc95a50cfd3ecb2a0bff884e039127208c2707a4c97abe42a8bdeca0342feba471221a105306e44ac116c54459cbd200828bd82a5828000181e203922020c5d51b299bb93734db4c101de070a317704ea126b2ae998373a61906b2c94b2d1b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b626166796265696134696a353270743375616378676a68636d6e766b64336477686d613235333773766c346c356d676d6d6669796c366d656e32751a003b13291a0042fc2940480019922c0c974c144058420157726e24723ed3ac9fcebca941f8a40277db0e365c8fb6b0bb314a6f13f787ac489126edd33bf55b81eebcc4e3ac4c84b04da28198d585e11989421b4579ce9f01828bd82a5828000181e203922020a23e81c9cf5fea21663221b0f9586e5e634e7f05e03081a3301c6cdb7ae5ec371b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b6261667962656966736636376a756e7a716c6d706b7868326a7a736579767970616b657a636435786977707676797a747164356a6d6a73667967341a003b13291a0042fc2940480019922c0c974c1440584201e599b266656ab81bd7c0cf5a3ac397a3b352379d82070c2c5a7ab1da2a664b9617f8cbb42dce2f5031fe6e26ed91f11262cd7436bd04b37162be624ccb1a3c2b01828bd82a5828000181e20392202013766a5bcb959aa8ac5b22822fc4f7d9c79b457a9501b3edf440f79e673e66081b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b6261667962656962646b64363675376d6d617673636476343363726266356a686e366b66366e6f79667963336f34676869796862616737347475651a003b13291a0042fc2940480019922c0c974c14405842010ef9e00b1c22daa44eca93d3d6a80d51f20ab38d70fc006e70be518b99b37e9a6c46a080ec3e42a7213555a53c9f820f4972acec3d823dd02c4aa5eca4adecc001828bd82a5828000181e2039220204bea9a8c45a2588271148aacbf2560f5fa662567e44407551fee51fcc67057301b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b62616679626569627071736e6b667878717771617a667367346f7a667535337a667033756564696b683336376679696a33666967757a336a35636d1a003b13291a0042fc2940480019922c0c974c14405842012824925d194f15ff986df0527aa7ad69cee046ff15b0a1d12fe3afcc2a06c2bb7bdf39b17ef413583807a5ff84c92937795a3ef165f2185a7c0beb84c4f4660801828bd82a5828000181e203922020bddf4b55fee739966feb5c856a469bc1a686c0863017defede0e26c8e4d48f1f1b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b62616679626569616974666c6d7a656b367171656734626b686a696f7163673677716576686836776d6c79723378346133356c796f62356b6368791a003b13291a0042fc2940480019922c0c974c144058420142a7fb32292be26089e296ca905e42a34648ecfac08eb6047fc1a403f5e62ec14c4f71b7836b589af639ac63a59c4bb0ca63a10802a549644a5b09af2ba1631c01828bd82a5828000181e2039220200bbc8bdd10c70e294862600f043c237ca80e043c11c1a5252c911b36069f002b1b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b626166796265696733796f6769766a6a336666766868346272697736726c727164707474366164356c7a366e7767666f76653363787a3472786f341a003b13291a0042fc2940480019922b15b74038405842014c7cf74b709018ee3eb95864d0f37088b3f1dc83a3d689605554abbab55ef161317abad2e9918f2484d1ee6168ef5c2dda96b75a4fbc9ebdd1d6fd2d48e027dd01828bd82a5828000181e2039220200a76de48c9c6517ff032e743fa919b7e6b44a8480a52291baeb3f2b427d2ee261b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b62616679626569646f74713332686d33657a33666177677a766e7234666f32746a74706d6935333362787976686f7a7a656c6561663736776135711a003b13291a0042fc2940480019922b15b74038405842014440487ff525126ac46dcad4b45688cc0b56a76f4ada46b24c2af925d2a0fbcb68c71c407d3d7abf530e79c9883b5a5ff95c078c33646fa28d3ff8e44746b96300828bd82a5828000181e203922020ebdbc0567bbaace94a650d7e8386c34a50ed5f5d7a32743554354c4ae47f07291b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b626166796265696136696e367778346a6c6d726b373577646f74793670626468703276336c326235633464666a6d767836676368366a6a653733611a003b13291a0042fc2940480019922b15b74038405842011377553180af636cbc5c6f2273c07d347ab346b3be87012cc3809a51fd358f762b6b333c4e39e3a104312b3f34f5bb443acc79b79cb17de45db1bd07150b1dbd00828bd82a5828000181e20392202018ebbb2550b0448103f274ac4417f43a1efa5ad114a20387061d1d74246e10151b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b62616679626569663534707673326b67713470616c66726c6e677734746c786666627568626c696e69346c3332693365676e3570797a62726873651a003b13291a0042fc2940480019922b15b74038405842013f55a534440e191d1981b49ed6ed0d6a9be998960f4434ead301f818587521196a66c54c5b35a880f29a5e38472e443716ea0a6224e538d5e80b38e65b95d63200828bd82a5828000181e2039220201b201b7f50ccf1c171fee62c1a1bfe410fdc77a61d736d1088e4933f2d36a8291b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b62616679626569687732656371706533793435706d71626568356370747a34333373767a33686f776a646a776f6d786f78336f66716c72657479341a003b13291a0042fc2940480019922b15b7403840584201b37393af71d6644c75a8683625dff9344b369083a823d716e08ea8d4ff5a12081089eba596d9b097df0bb5c0f0587229edbf4d471805cee1ded249140e8e17af01828bd82a5828000181e203922020d517bff9321300969b393ed7c764561bc7a215fcac539937043b5f206bf33b331b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b62616679626569687a367271676f366e79627470376569713434616c636e6a723268776c643669366a753672363732346668666f73696a6a3466791a003b13291a0042fc2940480019922b15b740384058420125fa7fbea36f97bdb7eaab6db06f4057783893e11721fd23795074ad9cfb453a1324ef51f81ce34c817c6125efc9eef89c3e7d2849b4cf0733b2c7d8e2433f9d01828bd82a5828000181e20392202054186ab49c5778e50ac505f166f23b404bd2f31ab8975f9673cd8cb73c65e42a1b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b62616679626569633432667a3332766f74326972627169666a7a7769636e687a3564656c6d356b766936637262356a7033346936716d76737234341a003b13291a0042fc2940480019922b15b7403840584201e18ab1860c320d29f2260a733805a078a3b4e35f6813a61378e1f8c241ffe77a26fad6d60ccd38218c809cc867a612d828417c4b6c2c6f4ab6a633db807fec3300828bd82a5828000181e203922020c926be413c240d8f31038c2508e08baa0b5f1919a64739fc6cf3c7099fe0ff011b0000000800000000f55501bf5f6c9b16203d56861c2aaafd4f0fa9bf06de2645008ce1ac01783b626166796265696767726669736e6e6e6b637076773261786e686862686b76647a7536756e633437717878326c6b65673261327a7a3434613672751a003b13291a0042fc2940480019922b15b7403840584201f62b136228793e50085bb02232f0f972141b3d0716763aa4b63b90d82ff686150296f1ed3bbc97a460637fd339035ff446142e62afb8ac6d6ace762e288c0a0601 }, gas_limit: 2811210696, gas_fee_cap: TokenAmount(0.000000000000101524), gas_premium: TokenAmount(0.00000000000010047) }, Message { version: 0, from: Address(Address("f3skkaztfhx3hevr2homjqd5llda3padga6lcnoequc4kflokyhn3ugvrorrsfexo523arudhsrua2nuis7wva")), to: Address(Address("f02035252")), sequence: 118083, value: TokenAmount(0.0), method_num: 7, params: RawBytes { 8219d8f35907808457ecf4ce6fe8a290682cce10a41438c29995c0284ac08a72e64e96e771146a15e2ed73c9689e7421ce4995d35754d1ad74eb4f8182bf1b46e9c70861d926157bbb58ba958b56491cac78456722e9b3bef2a470ee38c93b8f0d414d512e0d5616f84bbeaf5e2e05ea7b76c8ff25daa129802be21ca19f3aa588d59ee98fa6d6dd9868700eb64e36eb67bd4b0140ece686b03c92b36a490dafd8bc7b58a129b995722403daad9263d9e662af25aeeaaa47ce3fb18d7178215c0abb67c18e435c9254a9d8611d94cf738d52bffebc52585621c4ae642d23a8eeca53749f1426565cade3f5db7034dac9a32d97c89c5bc2809920724f7b33e78ad919449d8b51a6161b0fe4dd0ff4c3981ef22c6774665a9dd72b8cca33f7c97affdfac76c389590aec8cce309ca212b2f9bebfddced6f0927ea19bc00e9f2754a17b7c7844082de46ca70f42c969a57b4fcb44f12745028a3b2adbc93043269c52bfb9e57738a1ea85adf29f649f226daf9363c57b2a3c3dcbdc063c8141feecab17b77627db808b4de90e0182dd23c2c8c85e39a790e2532c67ef1a8443cf9d0b0711a77f5e49fd194c9983d5feddeafd64e992334855af18555806aa87332b3647e5a3bf6579b95af1658d72f6c47c965b0e22e8909c1faf2c589cd4a4d99306712e0e82d82407820599743d22bd487a9bd62595d4cdd353d4c6cae4c0415e27b718aa914a57fd383b961f4e8170c588a6891b72d20fa723e45975bfde5fd9b22d1a5eb89412d0caae297d1cdbd07fca50ea5e1e7cf8763506f51556fffdd5f333dac96d771e8df1c82e6b6ecb1ccc7120e104c8c7d2fc55f70de263e7af580c2e12f480c310c3c51e3732dc044ddd29fba6e2381c2280282190eaa96873c81b0f9f7e032e1cf6c758d96216eb28b16ab09875d09fd4e9fa38c0f1c5eaf51929715669e07f4104fdd1abba1a33b715e68f982febc5feb538c99d9557fdf89d76f3e74f33a6ef9568e8e3be25554191afae23303d2019915f3d750f32a6ff734fdda0126c312abfcdfb79046e1195d7b0a49d56a8dc15e3aa52de0da8eea020fd8f7a7ac26a92a17f0da8920cc79bbdbf034e913d204ee67919c0d77a8b4cefc47d351c78e7be2138c20a0746a2adc0ea34ad952e4cfb92c0ca2762399a270d22aebebe13ef3991e5ab84ca49cc648a46e16006f21c38845282720b6c176a63d5a219a580db6615542f7e8b64fffef75d4bbaf8b277ef26bb8a5097a394c0e845430b8166b5721a0bfdea7b7d3ec527d9f9242c2e6833a7945009676dd1b51e0f224f311af53c794e72c23c9003061455d041cbf9fd83cf5cb371066cda59109d692e9c0545e7b755720de6f6346cc939251b4f34ce6d4d331e6845938300074200f938274f8a5010d56d7d49ac7734fa0861c69e4084a8b6b21aae8b62424c46d8c6e59f6cdfaa53497e99509d5e575d343afec22c3936d7d42aed62cb7a90525807d642a5bb0391ba444589312024-05-20T08:14:53.679061747Z e7e4708e215af4d1df75ebc781f56bc187eac1b1282872c691a035e36ca485d88b3b48a12e7482c68ab6e717917ee84a8325936a170ba071f2fde56e3751769e73b5a0f9edb612efddfcccaf64a7d4ffb2bff83eb14877ba4f9864855dad89e3356d1d09da44c8027d462fb070124fc8df40834fe01ea25bfa47ef3cd4bdebac83d76369568d5580788312dfde421a3a4254a955aeeb14cd281e9ad4e2b43ea7d650b9ae96a2036ef9243edff49f2360d6ee689050917fd1ff05e00c362d7b785fc5dfb465b593d3ad2a4054e904c7ab49f927e0794266743dafbb7bcdc84e89412ed205f247f66591a1ce73330211e78abf0232e2861d70e74ab40cdda4fccab849389864ca8af26bcddb48afe2508c6cdf55e40972c8cd4a88351bd09e177e076008e72f610c6f66105eb4a8666c1c2042c2ca58b29e691bd53d00640ea131b79940c8cfd6a2115d92cfe8728e16d9e633347fb4e84342b0e5b664fd55eb32c83b35b322897c413b9b99b404a8219eb829922b8c32368e0f0eb1fe14c17e7616cc1cbfa239d7fa7e193263430fd3e9d42882a7d0bbc805722efcbed7f933abab6dcb349f772bd42f938140160ddd1cef1d40404586a27fb726b3b587f747037a3f35733eba094dd5772933ce1d5b3df629b4044af602690ab3460df49dff105bcce36c02555e3b118564c1871fbdcde143bacac93dec3ead2048a6ec6e78857a32aa9a0531d9d2e98c4ea1354ee8d5014c3091e15178c79358bedb87aee9e474809ae478e34c1e6679a1269a6bd0275c660bbb9bae21e82a187c36f58e58197c5efa09c50f9f905ecbf3dee158f04b47c2d60b17f6ea2a5157fe342e162a457bca25323bcc67af1d954991a4aea537d47acd815d20cb83e694a33d92ab180b8aba8cc27f2617aace6169c5a82d0081fe99d3ae5aad3509ffab3d37b7df334e428f1666ae0b0b1aad5a054bb2f3ebe438e4520c922bf8c9d8cd3cddc5ff63085dfc49aae7d2bc3b78ad0fe6961aa550b0bdec71b49edb1fc92de17a2bb098296eed6059ed2370b095a5f01f5602636ab8815c6e25dec71e6d145ca6f2f16930328a4595401ee5756a47314032c8a6b8b8a5123d39696c52ebec7b88dca9db4f88b02d6984b5090478a0f55f1f6d12c85147ac5865c468171d038dab67c35822d033b874ee40a94d1db46c5786b566f29d9630a5f48070376f }, gas_limit: 76926430, gas_fee_cap: TokenAmount(0.000000000000101455), gas_premium: TokenAmount(0.000000000000100401) }, Message { version: 0, from: Address(Address("f3vh5r25crlvzsmooqhdir3qkpmucnemgd4fqqhzsubrv76dyw2flhyabce6glv3koyjpnsbf3bpr5fazvfoeq")), to: Address(Address("f0746416")), sequence: 7330, value: TokenAmount(0.0), method_num: 5, params: RawBytes { 85048182004081820d58c0ad8b33fad4866ede29eabf228d40ca533b626eec62d3440c6727fd811602f243033ec9b8fac316c4ee51c34638c04cadb9f940b8aa6465bb01e42b960af169a0e97e3483aea9f6820f31dfd6db9b073f1d373df450c1a89bfed249a4cc23ff3b05b40e28d0bf6900d2e722789b9d4c9648e220cc62cd680761c5230c54635172b110f58a11bfa668f7b2b96b9785eca78929b1109ce9101a750063d40ab35e005c172d15658be49490587e429420c3184dd65443621f0792ae99826903f67cf61a003aac525820868f10a3c896b98a72f033a28ee8f1819a5f0f8d33e94669990d01c1cd7fce3e }, gas_limit: 31447923, gas_fee_cap: TokenAmount(0.000000158993012034), gas_premium: TokenAmount(0.000000000000100218) }], bls_msg_includes: [[0, 1, 2, 3]], secp_msgs: [], secp_msg_includes: [[]] }) }] }
2024-05-20 10:14:10.306	
2024-05-20T08:14:10.306682Z  WARN forest_filecoin::libp2p_bitswap::request_manager: sending on a closed channel
2024-05-20 10:14:10.306	
2024-05-20T08:14:10.306209Z  WARN forest_filecoin::libp2p_bitswap::request_manager: sending on a closed channel
2024-05-20 10:14:10.297	
2024-05-20T08:14:10.295770Z  WARN forest_filecoin::libp2p_bitswap::request_manager: sending on a closed channel

The Forest used is v0.17.2. @samuelarogbonlo could you please add more details on the deployment, i.e., flags and configuration used? Thanks.

Other information and links

[samuelarogbonlo]

The config.toml

[client]
#Controls whether the keystore is encrypted
encrypt_keystore = false
data_dir = "/home/forest/forest_data"
enable_rpc = true
enable_metrics_endpoint = true
metrics_address = "127.0.0.1:6116"
rpc_address = "0.0.0.0:2345"

[network]
listening_multiaddrs = ["/ip4/0.0.0.0/tcp/34000"]
#Kademlia option enforce p2p connections.
kademlia = true
#Amount of peers the node should maintain a connection with
#Helps in syncing ( moderate value to not be exceeded)
target_peer_count = 5000

[samuelarogbonlo]

Also the node is started with the following flags

--chain mainnet --config config.toml --stateless

[samuelarogbonlo]

Host node performance.