Not now but... it would be really nice if we could have coverage guided fuzzing.

1. Do a bunch of fuzzing.
2. For each input, record the code coverage from that input.
3. If the input exercises new code, add it to the corpus.

We'll still want random fuzzing, but this will help us make sure we at least cover as many error paths as possible.

Not now but... it would be really nice

Indeed it would 👍

Captured at #220

I do have one concern: the tests will remain really fast even if other parts slow down. Basically, we'll just start losing test coverage. But that's probably not a huge deal (maybe run a nightly fuzzing job in CI that's longer)?

Continuing from #219 (comment) and #219 (comment), I'd consider:

1. Skipping this in make test (leaving a single fuzz run from go test ./....
2. Running a bit of fuzzing in CI (30s, one job per fuzzer ideally).
3. A longer fuzzing run in the merge queue.

That gives us immediate "did we break something" feedback while leaving the slow tests to CI. Although I'm starting to think that should happen in this PR so we don't lose coverage and/or make the tests take longer.

Codecov Report

Attention: Patch coverage is 50.00000% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 71.66%. Comparing base (837ffde) to head (5d0cad7).

@@            Coverage Diff             @@
##             main     #219      +/-   ##
==========================================
+ Coverage   71.61%   71.66%   +0.04%     
==========================================
  Files          31       31              
  Lines        2336     2333       -3     
==========================================
- Hits         1673     1672       -1     
+ Misses        533      532       -1     
+ Partials      130      129       -1     

Let's merge #230, rebase and compare then.

Note, to compare coverage, don't just look at the numbers.

1. Generate a report with go test -cover -covermode=atomic -coverpkg github.com/filecoin-project/go-f3/gpbft -coverprofile coverage.out ./.... -coverpkg ensures we don't get a bunch of noise because we're changing the test framework.
2. Trim the counts with cat coverage.out | grep -v ' 0$' | cut -d' ' -f-2 > coverage.trimmed.
3. Do the same thing on the branch you're comparing against.
4. Diff the results.

That'll let you explore the actual coverage diff.

(Maybe codecov.io will do this? But I've never had a good experience with it)

Codecov will do it (apart from the fact that right now we are collecting coverage across all packages, we might want to change it).

I'm looking into parallelizing fuzzing "seeds" and it's not clear if it actually works.

I had not updated to the latest PR code when writing the below. I'm still nervous, but my initial observations were of an earlier version of the code. Invoking some of the fuzzing from go test seems like a good arrangement.

/opt/homebrew/opt/go/libexec/bin/go test -json github.com/filecoin-project/go-f3/test -fuzz . -run ^$
{"Time":"2024-05-16T10:59:41.884345+12:00","Action":"start","Package":"github.com/filecoin-project/go-f3/test"}
testing: will not fuzz, -fuzz matches more than one fuzz test: [FuzzAbsentAdversary FuzzImmediateDecideAdversary FuzzHonest_RequireStrongQuorumToProgress FuzzMultiSyncAgreement FuzzMultiAsyncAgreement FuzzRepeatAdversary]

@masih in case you're reading this in email, I retracted much of the above comment after realising I didn't have the latest code locally.

I found the current test failure, so can confirm that the workflow for isolating a failing test, tracing, IDE support etc are adequate here.

Folks this is ready for a final look; it:

• offers slightly higher coverage compared to main.
• covers the previous test cases as static fuzz corpus.
• refactors tests to remove duplicate code while generally keeping the same parameter search space as the tests in main branch.
• introduces fuzzing to CI at 30s per fuzz test.
  • I intend to reflect on suggestions to run longer nightly fuzzing / fuzz in parallel in a separate PR as this one is large enough already.

Please let me know if there is anything that I might have missed.

Happy fuzzing 🪲

Fuzzing failed with

--- FAIL: FuzzHonestMultiInstance_SyncAgreement (19.62s)
    fuzzing process hung or terminated unexpectedly: exit status 2
    Failing input written to testdata/fuzz/FuzzHonestMultiInstance_SyncAgreement/3612110feb3427c0
    To re-run:
    go test -run=FuzzHonestMultiInstance_SyncAgreement/3612110feb3427c0

but IMO let's get this merged as it is a pure improvement given the baseline coverage without continuous fuzzing.

fuzzing process hung or terminated unexpectedly: exit status 2

see: golang/go#56238