Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for different payload formats for bring your own datasets for that might be needed for different inference containers #5

Open
madhurprash opened this issue Jan 26, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@madhurprash
Copy link
Collaborator

This tool currently supports the HF TGI container, and DJL Deep Speed container on SageMaker and both use the same format but in future other containers might need a different payload format.

Goal: To give user full flexibility to bring their payloads or contain code that generalizes payload generation irrespective of the container type that the user uses. Two options for solution to this issue here:

1/ Have the user bring in their own payload
2/ Have a generic function defined to convert the payload in support for the container type the user is using to deploy their model and generate inference from.

@madhurprash madhurprash self-assigned this Jan 27, 2024
@aarora79
Copy link
Contributor

Do this in the same way we have bring your own deployment script in that there is a inference function which is called from the run inference notebook.

@aarora79 aarora79 added the enhancement New feature or request label Feb 15, 2024
@madhurprash madhurprash changed the title Add support for different payload formats that might be needed for different inference containers Add support for different payload formats for bring your own datasets for that might be needed for different inference containers Mar 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
2 participants