Emit live metrics

[madhurprash]

Emit live metrics so that they can be monitored through Grafana via live dashboard. More information to come on this issue but the goal here is to provide full flexibility to the user to be able to view metrics in ways that best suits the needs of their business and technological goals.

[TBD] --> Some sort of an analytics pipeline sending and emitting live results for different model configurations, their results and different metrics based on the needs of the user.

[aarora79]

@madhurprash please get details from Shamik and add here in this ticket.

[shamikatamazon]

The intent of this functionality is to use Prometheus to store metrics and use Graphana as a tool to visualize graphs performance of an LLM workload. To achieve this, the client and possibly the serving engine should emit metrics which are scraped by a Prometheus agent and persisted in Prometheus. Graphana uses the data from Prometheus to display metrics in a dashboard that is provided by this project. The steps to deploy a model and the pipeline on an EC2 instance is outlined at https://github.com/shamikatamazon/genai/tree/main/benchmarking