Skip to content

Issues: aws-samples/foundation-model-benchmarking-tool

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

20 Open 11 Closed
20 Open 11 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

FMBench needs to be configurable so that no S3 access is needed
#85 opened May 13, 2024 by RajeshRamchander
To integrate the Open-Orca HF dataset to FMBench
#68 opened Apr 30, 2024 by madhurprash
@jabburke
To add support for bring your own 'SageMaker' pre existing endpoint
#45 opened Apr 7, 2024 by madhurprash
1
@madhurprash
FMBench to support benchmarking for embedding models enhancement New feature or request
#33 opened Mar 25, 2024 by madhurprash
@madhurprash
Need for Integration of FMBench Tool with External 3rd party Synthetic Data Generation Tools for Benchmarking usecase.
#32 opened Mar 16, 2024 by akramIOT
Compare different models for the same dataset enhancement New feature or request
#30 opened Feb 17, 2024 by aarora79
@aarora79
Try FMBench with instance count set to > 1 to see how scaling impacts latency and transactions per minute enhancement New feature or request
#29 opened Feb 17, 2024 by aarora79
Provide config file for FLAN-T5 out of the box enhancement New feature or request
#28 opened Feb 15, 2024 by aarora79
@madhurprash
Allow different experiments in the same config file to use different regions good first issue Good for newcomers
#27 opened Feb 15, 2024 by aarora79
@4mp
The business summary plot in the report needs to have a caption for disclaimer bug Something isn't working
#26 opened Feb 15, 2024 by aarora79
The README gives an impression that this tool is comparing instance types when it should be serving stacks documentation Improvements or additions to documentation
#25 opened Feb 15, 2024 by aarora79
@aarora79
Add Bedrock benchmarking to this tool enhancement New feature or request
#24 opened Feb 15, 2024 by aarora79
Merge model accuracy metrics also into this tool enhancement New feature or request
#23 opened Feb 15, 2024 by aarora79
Add support for models hosted on platforms other than SageMaker enhancement New feature or request
#22 opened Feb 15, 2024 by aarora79
Add documentation for the config file on readthedocs.io good first issue Good for newcomers
#21 opened Feb 15, 2024 by aarora79
@4mp
Code cleanup needed to replace the notebooks with regular python files help wanted Extra attention is needed
#20 opened Feb 15, 2024 by aarora79
@aarora79
Add Autoscaling functionality to FMBench enhancement New feature or request
#10 opened Feb 6, 2024 by madhurprash
Emit live metrics help wanted Extra attention is needed
#6 opened Jan 26, 2024 by madhurprash
2
Add support for different payload formats for bring your own datasets for that might be needed for different inference containers enhancement New feature or request
#5 opened Jan 26, 2024 by madhurprash
@madhurprash
1
Containerize FMBench and provide instructions for running the container on EC2 help wanted Extra attention is needed
#3 opened Jan 26, 2024 by madhurprash
@madhurprash
ProTip! Type g i on any issue or pull request to go back to the issue listing page.