Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reconsider default fuzz and invariant settings #7848

Open
mds1 opened this issue May 3, 2024 · 0 comments · May be fixed by #7957
Open

Reconsider default fuzz and invariant settings #7848

mds1 opened this issue May 3, 2024 · 0 comments · May be fixed by #7957
Labels
T-feature Type: feature

Comments

@mds1
Copy link
Collaborator

mds1 commented May 3, 2024

Component

Forge

Describe the feature you would like

As the fuzzer has improved and we've learned more about how to catch various kinds of bugs, we've realized the current default parameters may not be optimal. For example, see comments from @mds1 and @grandizzy starting at #5868 (comment):

From my testing, trying to flesh out depth vs. runs relationship a bit:

  • Default fuzz settings do not catch it, regardless of number of runs (I only tried up to 10k runs)
  • Setting depth=500 catches it consistently with the default number of runs (256)
  • Setting depth=500 with runs=100 occasionally catches it
  • Setting depth=10000 with runs=1 occasionally catches it, but seems to be a bit less often than prior bullet

So:

  • I think we can close this as it's now catchable with the proper config
  • But we probably need a higher default depth than 15, as the depth seems very import, and should reconsider our default invariant config

Agree, we should revisit all defaults and update them to more relevant values, like

  • depth to 500 (For example echidna uses a default seqLen: 100 and testLimit: 50000 which is like 500 runs with depth of 100 in foundry terms)
  • shrink_run_limit defaults now to 2^18 which is not realistic anymore with our new shrinking mechanism (echidna uses shrinkLimit: 5000). From tests done in perf(invariant): sequentially shrink failed sequence #7756 we can shrink a sequence of 5000 calls in 215 seconds, so I think that default is acceptable
  • maybe default senders to 3 known addresses as echidna does with 0x1, 0x2, 0x3 addresses
  • others

Worth mentioning that this kind of failure will be identified must faster when we implement per type fuzzing from state, so new owner address will be exercised right away.

Let's revisit these and update the defaults.

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T-feature Type: feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant