{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":135468185,"defaultBranch":"master","name":"rust-fil-proofs","ownerLogin":"filecoin-project","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2018-05-30T16:12:53.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/22014611?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1715975738.0","currentOid":""},"activityList":{"items":[{"before":"50d381fd0011f385fae27b42aca0f35869c82feb","after":"0dfef1d8416c8240400fc188e90f0e55fa6b90ae","ref":"refs/heads/master","pushedAt":"2024-05-17T19:55:38.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"cryptonemo","name":"nemo","path":"/cryptonemo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/55720287?s=80&v=4"},"commit":{"message":"chore: Release","shortMessageHtmlLink":"chore: Release"}},{"before":"51c7a79d9058a45988290b4da7217bec05750cc8","after":"50d381fd0011f385fae27b42aca0f35869c82feb","ref":"refs/heads/master","pushedAt":"2024-05-17T19:50:29.000Z","pushType":"push","commitsCount":3,"pusher":{"login":"cryptonemo","name":"nemo","path":"/cryptonemo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/55720287?s=80&v=4"},"commit":{"message":"chore: Release","shortMessageHtmlLink":"chore: Release"}},{"before":null,"after":"b0cb55fcbfa95dff892e09e96b72ccddd8de8e52","ref":"refs/heads/code-improvements","pushedAt":"2024-05-17T17:13:19.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"vmx","name":"Volker Mische","path":"/vmx","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/42839?s=80&v=4"},"commit":{"message":"tests: simplify tests a bit\n\nIntead of always passing in the same PoRep ID, just hard-code it. Also\nmake it use a constant instead of a magic value like in the other tests.","shortMessageHtmlLink":"tests: simplify tests a bit"}},{"before":"2a9b81db3532ba832fff9b9d103b68f6b5e49bf1","after":"51c7a79d9058a45988290b4da7217bec05750cc8","ref":"refs/heads/master","pushedAt":"2024-05-17T16:25:26.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"cryptonemo","name":"nemo","path":"/cryptonemo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/55720287?s=80&v=4"},"commit":{"message":"feat: expose an API suitable for NI-PoRep proof aggregation (#1757)\n\n* feat: expose an API suitable for NI-PoRep proof aggregation\r\n* feat: add tests to ensure NI-PoRep is tested w and w/o aggregation\r\n* feat: enforce FIP-0090 bounds on NI-PoRep aggregation counts","shortMessageHtmlLink":"feat: expose an API suitable for NI-PoRep proof aggregation (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2300764428\" data-permission-text=\"Title is private\" data-url=\"https://github.com/filecoin-project/rust-fil-proofs/issues/1757\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/filecoin-project/rust-fil-proofs/pull/1757/hovercard\" href=\"https://github.com/filecoin-project/rust-fil-proofs/pull/1757\">#1757</a>)"}},{"before":"5281e3c9f0b12dd234beea46d3a6eb8c3351c836","after":"46281c28eb7097bfeb4f084da1ef01e93f9a7387","ref":"refs/heads/ni-porep-fix-agg","pushedAt":"2024-05-16T23:47:10.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"cryptonemo","name":"nemo","path":"/cryptonemo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/55720287?s=80&v=4"},"commit":{"message":"feat: enforce FIP-0090 bounds on NI-PoRep aggregation counts","shortMessageHtmlLink":"feat: enforce FIP-0090 bounds on NI-PoRep aggregation counts"}},{"before":"be146432f4795650edb1f9e9332aa5b7a77b30fd","after":"5281e3c9f0b12dd234beea46d3a6eb8c3351c836","ref":"refs/heads/ni-porep-fix-agg","pushedAt":"2024-05-16T22:06:24.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"cryptonemo","name":"nemo","path":"/cryptonemo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/55720287?s=80&v=4"},"commit":{"message":"fix: update bound enforcement","shortMessageHtmlLink":"fix: update bound enforcement"}},{"before":"3e04757f717ed87b1023982259c2c313ef8f8924","after":"be146432f4795650edb1f9e9332aa5b7a77b30fd","ref":"refs/heads/ni-porep-fix-agg","pushedAt":"2024-05-16T20:01:14.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"cryptonemo","name":"nemo","path":"/cryptonemo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/55720287?s=80&v=4"},"commit":{"message":"feat: enforce FIP-0090 bounds on NI-PoRep aggregation counts","shortMessageHtmlLink":"feat: enforce FIP-0090 bounds on NI-PoRep aggregation counts"}},{"before":null,"after":"3e04757f717ed87b1023982259c2c313ef8f8924","ref":"refs/heads/ni-porep-fix-agg","pushedAt":"2024-05-16T15:50:13.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"cryptonemo","name":"nemo","path":"/cryptonemo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/55720287?s=80&v=4"},"commit":{"message":"feat: expose an API suitable for NI-PoRep proof aggregation\nfeat: add tests to ensure NI-PoRep is tested w and w/o aggregation","shortMessageHtmlLink":"feat: expose an API suitable for NI-PoRep proof aggregation"}},{"before":"19e508905ea9916b11e6e53ec37d0882995333fb","after":"2a9b81db3532ba832fff9b9d103b68f6b5e49bf1","ref":"refs/heads/master","pushedAt":"2024-05-15T20:40:51.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"DrPeterVanNostrand","name":null,"path":"/DrPeterVanNostrand","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9577225?s=80&v=4"},"commit":{"message":"fix: increase ni-porep min challenges for test sectors (#1756)\n\n- NI-PoRep's minimum challenge count should be 12.8x interactive PoRep's minimum challenge count (then rounded up to the nearest whole number of challenges) for both testing and production sector sizes.","shortMessageHtmlLink":"fix: increase ni-porep min challenges for test sectors (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2298371521\" data-permission-text=\"Title is private\" data-url=\"https://github.com/filecoin-project/rust-fil-proofs/issues/1756\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/filecoin-project/rust-fil-proofs/pull/1756/hovercard\" href=\"https://github.com/filecoin-project/rust-fil-proofs/pull/1756\">#1756</a>)"}},{"before":null,"after":"866455237206fb0d76ca61e25026e85ce367b368","ref":"refs/heads/ni-porep-test-sectors-challenge-count","pushedAt":"2024-05-15T16:27:32.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"DrPeterVanNostrand","name":null,"path":"/DrPeterVanNostrand","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9577225?s=80&v=4"},"commit":{"message":"fix: increase ni-porep min challenges for test sectors","shortMessageHtmlLink":"fix: increase ni-porep min challenges for test sectors"}},{"before":"0c920258740a79ebc881f142161375a52f988910","after":"19e508905ea9916b11e6e53ec37d0882995333fb","ref":"refs/heads/master","pushedAt":"2024-05-15T16:06:20.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"DrPeterVanNostrand","name":null,"path":"/DrPeterVanNostrand","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9577225?s=80&v=4"},"commit":{"message":"fix: remove challenge seed from NI-PoRep SnarkPack transcript (#1755)","shortMessageHtmlLink":"fix: remove challenge seed from NI-PoRep SnarkPack transcript (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2286179782\" data-permission-text=\"Title is private\" data-url=\"https://github.com/filecoin-project/rust-fil-proofs/issues/1755\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/filecoin-project/rust-fil-proofs/pull/1755/hovercard\" href=\"https://github.com/filecoin-project/rust-fil-proofs/pull/1755\">#1755</a>)"}},{"before":"a8517eda5b0874ca09e74ea8cf06402a453aea53","after":"a9af8b70da2d4743ccf1ff6a6a783777057b1118","ref":"refs/heads/ni-porep-agg-ignore-seed","pushedAt":"2024-05-14T15:55:36.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"DrPeterVanNostrand","name":null,"path":"/DrPeterVanNostrand","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9577225?s=80&v=4"},"commit":{"message":"Add comment in seal_commit_phase1 re. NI-PoRep seed value","shortMessageHtmlLink":"Add comment in seal_commit_phase1 re. NI-PoRep seed value"}},{"before":null,"after":"a8517eda5b0874ca09e74ea8cf06402a453aea53","ref":"refs/heads/ni-porep-agg-ignore-seed","pushedAt":"2024-05-08T18:11:47.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"DrPeterVanNostrand","name":null,"path":"/DrPeterVanNostrand","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9577225?s=80&v=4"},"commit":{"message":"fix: remove challenge seed from NI-PoRep SnarkPack transcript","shortMessageHtmlLink":"fix: remove challenge seed from NI-PoRep SnarkPack transcript"}},{"before":null,"after":"69c23e7e3f3bd4493ed9bf14aba2d504ce4669e3","ref":"refs/heads/ni-porep-fixed-seed","pushedAt":"2024-05-08T12:46:32.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"vmx","name":"Volker Mische","path":"/vmx","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/42839?s=80&v=4"},"commit":{"message":"fix: use a fixed seed for Non-interactive PoRep\n\nThe Interactive PoRep requires a seed for the proof. The Non-interactive PoRep\ndoes not. In order to keep the public APIs simple, just make sure that the\npassed in seed in case of a Non-interactive PoRep is a fixed one. It was choosen\nto be all zeros as such a byte array is easy to generate.\n\nThat fixed seed is used for the proving (and the internal aggregation) as well as\nthe verification.\n\nThis commit also removes the restriction that a seed cannot be all zeros, which\nit also totally could be by coincidence (it's randomly generated in case of the\nInteractive PoRep).","shortMessageHtmlLink":"fix: use a fixed seed for Non-interactive PoRep"}},{"before":"d23f5068644efbbd34ca61cd0d1c2cd8f16caab6","after":"a3d59ecf97cf53a438d282e323cd22c7c9fa175d","ref":"refs/heads/ni-porep-seed","pushedAt":"2024-05-08T11:54:53.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"vmx","name":"Volker Mische","path":"/vmx","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/42839?s=80&v=4"},"commit":{"message":"refactor: move the check that public inputs match stored data\n\nMove the check that that the public inputs match the stored data out of\nthe `prove_layers_generate()` where it is called on every partition\n(with exactly the same data) one level up into `prove_layers()`. This\nway the same check is performed only once.\n\nThis also makes it possible to pass in the `replica_id` directly into\nthe `prove_layers_generate()` function. This way we don't need to pass\nin the full public inputs, which makes it more obvious which parameters\nare passed into the actual proving. It e.g. makes clear that the seed\nis only used for the challenge generation, but not for the proving\nafterwards (the challenges are of course also part of the proving,\nthough).","shortMessageHtmlLink":"refactor: move the check that public inputs match stored data"}},{"before":null,"after":"d23f5068644efbbd34ca61cd0d1c2cd8f16caab6","ref":"refs/heads/ni-porep-seed","pushedAt":"2024-05-08T11:44:26.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"vmx","name":"Volker Mische","path":"/vmx","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/42839?s=80&v=4"},"commit":{"message":"refactor: move the check that public inputs match stored data\n\nMove the check that that the public inputs match the stored data out of\nthe `prove_layers_generate()` where it is called on every partition\n(with exactly the same data) one level up into `prove_layers()`. This\nway the same check is performed only once.\n\nThis also makes it possible to pass in the `replica_id` directly into\nthe `prove_layers_generate()` function. This way we don't need to pass\nin the full public inputs, which makes it more obvious which parameters\nare passed into the actual proving. This should make audits easier.","shortMessageHtmlLink":"refactor: move the check that public inputs match stored data"}},{"before":"8424b8f0f886773d16fd61bee201c1c4e195191b","after":"b7f4e3a6142c85e12ddda66907306af69829ab7f","ref":"refs/heads/tree-d-poseidon","pushedAt":"2024-05-06T23:14:07.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"vmx","name":"Volker Mische","path":"/vmx","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/42839?s=80&v=4"},"commit":{"message":"feat: add binary to hash tree-d with Poseidon\n\nFor running it for a 32GiB sector:\n\n    mkdir /tmp/filbin32gib\n    fallocate --length 32gib /tmp/unsealed32gib\n    echo '{\"input_path\": \"/tmp/unsealed32gib\", \"output_dir\": \"/tmp/filbin32gib\", \"sector_size\": 34359738368}'|RUST_LOG=trace /usr/bin/time -v cargo run --no-default-features --release --bin tree-d-poseidon\n\nIf you rerun (or run the `tree-d` binary) delete the generated\n`/tmp/filbin32gib/sc-02-data-tree-d.dat` first.\n\nIn order to reduce the number of threads being used, you can set the\n`RAYON_NUM_THREADS` env var. I'm unsure if a single thread would work\nas expected, so I'd use at least two threads to be on the safe side\n(though one might also just try a single thread).","shortMessageHtmlLink":"feat: add binary to hash tree-d with Poseidon"}},{"before":"3d67354fa850a8a82f6e07ce75107881c37fa19e","after":"8424b8f0f886773d16fd61bee201c1c4e195191b","ref":"refs/heads/tree-d-poseidon","pushedAt":"2024-05-06T11:40:48.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"vmx","name":"Volker Mische","path":"/vmx","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/42839?s=80&v=4"},"commit":{"message":"feat: add binary to hash tree-d with Poseidon\n\nFor running it for a 32GiB sector:\n\n    mkdir /tmp/filbin32gib\n    fallocate --length 32gib /tmp/unsealed32gib\n    echo '{\"input_path\": \"/tmp/unsealed32gib\", \"output_dir\": \"/tmp/filbin32gib\", \"sector_size\": 34359738368'|RUST_LOG=trace /usr/bin/time -v cargo run --no-default-features --release --bin tree-d-poseidon\n\nIf you rerun (or run the `tree-d` binary) delete the generated\n`/tmp/filbin32gib/sc-02-data-tree-d.dat` first.\n\nIn order to reduce the number of threads being used, you can set the\n`RAYON_NUM_THREADS` env var. I'm unsure if a single thread would work\nas expected, so I'd use at least two threads to be on the safe side\n(though one might also just try a single thread).","shortMessageHtmlLink":"feat: add binary to hash tree-d with Poseidon"}},{"before":"f00acf04e97ef0b00bc3eb5a8f828117167d07f6","after":"3d67354fa850a8a82f6e07ce75107881c37fa19e","ref":"refs/heads/tree-d-poseidon","pushedAt":"2024-05-06T11:36:56.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"vmx","name":"Volker Mische","path":"/vmx","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/42839?s=80&v=4"},"commit":{"message":"feat: add binary to hash tree-d with Poseidon\n\nFor running it for a 32GiB sector:\n\n    mkdir /tmp/filbin32gib\n    fallocate --length 32gib /tmp/unsealed32gib\n    echo '{\"input_path\": \"/tmp/unsealed32gib\", \"output_dir\": \"/tmp/filbin32gib\", \"sector_size\": 34359738368'|RUST_LOG=trace /usr/bin/time -v cargo run --release --bin tree-d-poseidon\n\nIf you rerun (or run the `tree-d` binary) delete the generated\n`/tmp/filbin32gib/sc-02-data-tree-d.dat` first.\n\nIn order to reduce the number of threads being used, you can set the\n`RAYON_NUM_THREADS` env var. I'm unsure if a single thread would work\nas expected, so I'd use at least two threads to be on the safe side\n(though one might also just try a single thread).","shortMessageHtmlLink":"feat: add binary to hash tree-d with Poseidon"}},{"before":"ec628383a42e5f10cbd6fc37b61bf711341dac82","after":"f00acf04e97ef0b00bc3eb5a8f828117167d07f6","ref":"refs/heads/tree-d-poseidon","pushedAt":"2024-05-06T11:28:04.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"vmx","name":"Volker Mische","path":"/vmx","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/42839?s=80&v=4"},"commit":{"message":"feat: add binary to hash tree-d with Poseidon\n\nFor running it for a 32GiB sector:\n\n    mkdir /tmp/filbin32gib\n    fallocate --length 32gib /tmp/filbin32gib/unsealed\n    echo '{\"input_path\": \"/tmp/filbin32gib/unsealed\", \"output_dir\": \"/tmp/filbin32gib\", \"sector_size\": 34359738368'|RUST_LOG=trace /usr/bin/time -v cargo run --release --bin tree-d-poseidon","shortMessageHtmlLink":"feat: add binary to hash tree-d with Poseidon"}},{"before":"1dcfdebfa178499d35549ee49870c1a97139e9cb","after":"ec628383a42e5f10cbd6fc37b61bf711341dac82","ref":"refs/heads/tree-d-poseidon","pushedAt":"2024-05-06T11:24:40.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"vmx","name":"Volker Mische","path":"/vmx","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/42839?s=80&v=4"},"commit":{"message":"feat: add binary to hash tree-d with Poseidon\n\nFor running it for a 32GiB sector:\n\n    mkdir /tmp/filbin32gib\n    fallocate --length 32gib /tmp/filbin32gib/unsealed\n    echo '{\"input_path\": \"/tmp/filbin32gib/unsealed\", \"output_dir\": \"/tmp/filbin32gib\", \"sector_size\": 34359738368'|RUST_LOG=trace cargo run --release --bin tree-d-poseidon","shortMessageHtmlLink":"feat: add binary to hash tree-d with Poseidon"}},{"before":null,"after":"1dcfdebfa178499d35549ee49870c1a97139e9cb","ref":"refs/heads/tree-d-poseidon","pushedAt":"2024-05-06T11:15:55.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"vmx","name":"Volker Mische","path":"/vmx","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/42839?s=80&v=4"},"commit":{"message":"feat: add binary to hash tree-d with Poseidon","shortMessageHtmlLink":"feat: add binary to hash tree-d with Poseidon"}},{"before":"cd452c4b176dfa26ffe99595e3a7ed2fa391b679","after":"0c920258740a79ebc881f142161375a52f988910","ref":"refs/heads/master","pushedAt":"2024-05-01T18:52:41.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"cryptonemo","name":"nemo","path":"/cryptonemo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/55720287?s=80&v=4"},"commit":{"message":"chore: remove repetitive words (#1753)\n\nSigned-off-by: clonemycode <clonecode@aliyun.com>","shortMessageHtmlLink":"chore: remove repetitive words (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2273868905\" data-permission-text=\"Title is private\" data-url=\"https://github.com/filecoin-project/rust-fil-proofs/issues/1753\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/filecoin-project/rust-fil-proofs/pull/1753/hovercard\" href=\"https://github.com/filecoin-project/rust-fil-proofs/pull/1753\">#1753</a>)"}},{"before":"3f018b51b6327b135830899d237a7ba181942d7e","after":"cd452c4b176dfa26ffe99595e3a7ed2fa391b679","ref":"refs/heads/master","pushedAt":"2024-04-25T18:48:36.000Z","pushType":"push","commitsCount":11,"pusher":{"login":"cryptonemo","name":"nemo","path":"/cryptonemo","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/55720287?s=80&v=4"},"commit":{"message":"chore(fil-proofs-param): release 12.0.0","shortMessageHtmlLink":"chore(fil-proofs-param): release 12.0.0"}},{"before":"b988e18e996a813c8c5393757eeb1e0647fa2e40","after":"0b0709ffedb6ca05407ac21e5b83e77a369da984","ref":"refs/heads/simpler-error-handling","pushedAt":"2024-04-15T12:35:40.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"vmx","name":"Volker Mische","path":"/vmx","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/42839?s=80&v=4"},"commit":{"message":"refactor: simpler error handling\n\nhttps://github.com/filecoin-project/rust-fil-proofs/pull/1748 fixed a problem\nwhere errors (asserts) where not properly propagated upwards. There were\nthreads just hanging without making any progress.\n\nI usually follow the development paradigm of \"Make it work, make it right,\nmake it fast\" [1], where I interpret the \"make it right\" as \"make it simple\".\nWith \"simple\" I mean things like clear code, easy to follow, no new paradigms,\nfollowing the style of the existing code base.\n\nI'd like to be more concrete about that for this change, on why I find this\nversion \"simpler\".\n\nThe core issue was an assert within a spawned thread in a yastl pool. Someone\nnew to the code base coould re-introduce such a problem easily, hence it would\nbe best if we can actually prevent that. With removing the yastl thread pool,\nwhere it's not really needed for better performance, we can easily prevent\nthat.\n\nWe see three difference uses of the yastl thread pool within the `proof.rs`.\nOne is using it to pipeline operations, another one is for a highly parallizing\nan operation. Usually for data parallism we use Rayon in this code base. In\nthe pipelining use case, there is Rayon used within pipeline. The yastl thread\npool will spawn only a limited set of threads, so this looks alright. For the\nhighly parallelised case, we only use yastl and not Rayon (although we should\nlook into just Rayon instead).\n\nThe third use, where we mix yastl and Rayon for highly parallel operations is\nremoved with this PR. This is intentional. Having two thread pools, which by\ndefault use as many threads as cores could easily overprovision a system. This\ncould potentially lead to unintended slow downs. Another benefit of just using\nRayon is, that the number of threads can be controlled with an environment\nvariable. This gives more control when several instances of this code is run\nin parallel, which is the case for some storage providers.\n\nSwitching back from errors to asserts. I don't know the reason, why the\nasserts were changed to errors. Hence I'm switching it back to asserts, as\nnow it's easily possible. If one looks at the diff between the version prior\nto #1728 and this, then the diff is pretty minimal an straight forward. Also\none verification again runs only in debug mode and not also in release mode.\nThough if errors are desired, they can be easily be introduce by switching the\nasserts to anyhow's `ensure!()` macro.\n\nFollowing the error handling needs less context. With this change, the asserts\nare happening in the code right away, it works the way people coding Rust are\nused to. Prior to this change, more context is needed. Taking the \"invalid\ncomm_d\"-error as an example. It happens on line 456 [2]. Now reading through\nthe code what it means: first look for the `invalid_comm_d` variable. It\ndefines an instance of the `InvalidChallengeCoordinate` struct, which we take\na quick look at and see that's a local one, for specifcally error handling.\nThat instance is wrapped in an Arc which is interesting at a first glance,\nas in Rust shared state is usually tried to be avoided where possible. When\nlooking into the usage of `invalid_comm_d` it becomes clear that we need the\n`Arc`, as the might assign a different value to it in the error case. We need\na mutable reference here, but again usually in Rust immutable types are\npreferred. So if we can avoid the `Arc` as well some mutability, it's a win\nfor being more Rust idiomatic, hence easier for people familiar with Rust.\n\nFor me that falls under the \"code is written once, but read many times\"\ncategory. So making it easy to read is a win.\n\nIn the lower part of the change, the yastl usage for high data parallelism is\nremoved in favour of Rayon, see above for some of the reasons. Also using\nRayon here seems to use the thread pool more efficiently, at least on the\nmachine I've tested it on (with 64 threads).\n\nWhen looking at the diff between this change and the commit prior to #1748\n\n    git diff 8f5bd86be36a55e33b9b293ba22ea13ca1f28163.. -w -- storage-proofs-porep/src/stacked/vanilla/proof.rs\n\nThen the changes are very minimal, which I also count as a sign for being\n\"simpler\".\n\n[1]: https://wiki.c2.com/?MakeItWorkMakeItRightMakeItFast\n[2]: https://github.com/filecoin-project/rust-fil-proofs/blob/3f018b51b6327b135830899d237a7ba181942d7e/storage-proofs-porep/src/stacked/vanilla/proof.rs#L456-L457","shortMessageHtmlLink":"refactor: simpler error handling"}},{"before":"ba9c3d0269654ce1e5aa9b7f3734add9aa49596a","after":"b988e18e996a813c8c5393757eeb1e0647fa2e40","ref":"refs/heads/simpler-error-handling","pushedAt":"2024-04-15T11:09:00.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"vmx","name":"Volker Mische","path":"/vmx","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/42839?s=80&v=4"},"commit":{"message":"fix: better error handling\n\nAsserts in a yastl thread pool can lead to things hanging, instead of\nproper panicking. Spawning the validation was a performance optimization,\nbut it turns out it's not needed. Hence we can just remove the thread\npool from there. This has also the advantage that we don't have two\ndifferent threads pools (from Rayon and yastl) fighting for the same\nresources. Also the number of threads for the Rayon thread pool can be\nbound with an environment variable (the yastl one not), so there's more\ncontrol for running the operations.\n\nThere was another instance of using the yastl thread pool for verifying\nthe data before writing the Synthetic PoReps to disk. It again has the\nproblem with the assert. With using Rayon instead, we again have the\nadvantage of a more controllable thread pool and it's also slightly\nfaster.\n\nReplaces #1748.","shortMessageHtmlLink":"fix: better error handling"}},{"before":null,"after":"ba9c3d0269654ce1e5aa9b7f3734add9aa49596a","ref":"refs/heads/simpler-error-handling","pushedAt":"2024-04-11T19:04:08.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"vmx","name":"Volker Mische","path":"/vmx","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/42839?s=80&v=4"},"commit":{"message":"fix: better error handling\n\nInstead of using asserts, return proper errors. As this is non-trivial\nwith yastl, rely solely on rayon for parallelism. The performance\ncharacteristics are the same (it's maybe slightly, negligible faster).\n\nReplaces #1748.","shortMessageHtmlLink":"fix: better error handling"}},{"before":"c4186983e75e45a9637990762215ef627ba6a0d1","after":"089dad8c32dd20f8dedae1c9e6e882da0b0efd95","ref":"refs/heads/tmp-debug-synth-time","pushedAt":"2024-04-11T18:46:54.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"vmx","name":"Volker Mische","path":"/vmx","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/42839?s=80&v=4"},"commit":{"message":"WIP: add error handling","shortMessageHtmlLink":"WIP: add error handling"}},{"before":"0de3301baaffbd23e09fb543fe6de49bbc3814f6","after":"c4186983e75e45a9637990762215ef627ba6a0d1","ref":"refs/heads/tmp-debug-synth-time","pushedAt":"2024-04-11T18:22:09.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"vmx","name":"Volker Mische","path":"/vmx","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/42839?s=80&v=4"},"commit":{"message":"WIP: do direct verification again\n\nSee if it makes any difference.","shortMessageHtmlLink":"WIP: do direct verification again"}},{"before":"15922326cc1929ed3e5807c3855cdcd3308e8f4e","after":"0de3301baaffbd23e09fb543fe6de49bbc3814f6","ref":"refs/heads/tmp-debug-synth-time","pushedAt":"2024-04-11T18:16:30.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"vmx","name":"Volker Mische","path":"/vmx","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/42839?s=80&v=4"},"commit":{"message":"WIP: actually run that verification","shortMessageHtmlLink":"WIP: actually run that verification"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAETUCNxgA","startCursor":null,"endCursor":null}},"title":"Activity · filecoin-project/rust-fil-proofs"}