Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: Benchmarks unreliable as they run in the same process. #5

Open
samal-rasmussen opened this issue May 15, 2023 · 2 comments
Open

Comments

@samal-rasmussen
Copy link

I initially filed a bug for better-sqlite3 after seeing these benchmarks, and JoshuaWise was kind enough to spend time looking into it. His response here:
WiseLibs/better-sqlite3#1008 (comment)

So this is a bug report for these benchmarks reporting that they are unreliable, as they are not run in separate processes. Detailed description in JoshuaWise's comment.

@AndriiSherman
Copy link
Member

@samal-rasmussen, thanks a lot!
We have this repo open for specific purposes, so anyone can help us find any mistakes made in this benchmark repo.
We will analyze your feedback and the entire thread you've sent and improve this repo accordingly.

Furthermore, if you have any suggestions or changes to the code, please feel free to submit a PR with the modifications. We would love to learn from it and present accurate data to everyone.

@samal-rasmussen
Copy link
Author

samal-rasmussen commented May 20, 2023

It looks like you already have benchmark scripts that only test one library at a time. E.g:
https://github.com/drizzle-team/drizzle-northwind-benchmarks/blob/master/src/drizzle/benchmark.ts
https://github.com/drizzle-team/drizzle-northwind-benchmarks/blob/master/src/better-sqlite3/benchmark.ts

So you just need to remove the benchmark script that runs multiple libraries in the same process:
https://github.com/drizzle-team/drizzle-northwind-benchmarks/blob/master/src/common/benchmark.ts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants