Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fuzz RNG seed in simulation tests #218

Closed
Tracked by #226
masih opened this issue May 14, 2024 · 2 comments · Fixed by #219
Closed
Tracked by #226

Fuzz RNG seed in simulation tests #218

masih opened this issue May 14, 2024 · 2 comments · Fixed by #219
Assignees
Labels
testing Related to testing and validation

Comments

@masih
Copy link
Member

masih commented May 14, 2024

The number of rounds required for consensus in simulations is directly influenced by the latency modeling, which simulates
message propagation delay across the network.

Currently, many tests rely on repetitive executions with varying RNG seeds to simulate random effects. While this approach
allows for parallel testing, it still relies on deterministic seed choices.

To improve testing efficiency and separate concerns between development unit testing and fuzz testing, we should integrate
fuzz testing capabilities into our tests. This change will enable:

  1. Faster testing cycles during development, as developers won't need to wait for all repetitions to complete and fuzzing can be delegated to CI workflow marked as required to pass per PR.
  2. Configurable fuzz testing with adjustable lengths of time.
  3. Adherence to Golang best practices.

This improvement should also help catch edge cases like #196 more effectively, reducing the likelihood of similar issues in
the future.

@masih masih added the testing Related to testing and validation label May 14, 2024
@masih masih self-assigned this May 14, 2024
@anorth
Copy link
Member

anorth commented May 14, 2024

developers won't need to wait for all repetitions to complete

I don't understand this. Why not? Just using a different random seed doesn't make tests any better at covering the search space. Failures often happen deep in the iterations at present. If we do fewer iterations, we just won't find the bugs.

@masih
Copy link
Member Author

masih commented May 15, 2024

Description updated to clarify that the fuzzing can be delegated to a CI job run elsewhere.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
testing Related to testing and validation
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants