Groq models don't return the usage

[ValenCassa]

Description

Seems like Groq models do not follow the OpenAI spec for usage. They return the usage object under a x-groq key in the last chunk:

{
  id: "chatcmpl-8566e76f-7ded-47d7-a2ca-d212098af00c",
  object: "chat.completion.chunk",
  created: 1716131679,
  model: "llama3-8b-8192",
  system_fingerprint: "fp_179b0f92c9",
  choices: [{ index: 0, delta: {}, logprobs: null, finish_reason: "stop" }],
  x_groq: {
    id: "req_01hy8ppkajfqk987jv2vx88m9r",
    usage: {
      queue_time: 0.07535312,
      prompt_tokens: 23,
      prompt_time: 0.006,
      completion_tokens: 19,
      completion_time: 0.022,
      total_tokens: 42,
      total_time: 0.027999999999999997,
    },
  },
};

Code example

import { createOpenAI } from '@ai-sdk/openai';
import { streamText } from 'ai';
import dotenv from 'dotenv';

dotenv.config();

const groq = createOpenAI({
  apiKey: process.env.GROQ_API_KEY ?? '',
  baseURL: 'https://api.groq.com/openai/v1',
});

async function main() {
  const result = await streamText({
    model: groq.chat('llama3-70b-8192'),
    prompt: 'Invent a new holiday and describe its traditions.',
  });

  for await (const textPart of result.textStream) {
    process.stdout.write(textPart);
  }

  console.log();
  console.log('Token usage:', await result.usage); // This results in NaN
  console.log('Finish reason:', await result.finishReason);
}

main().catch(console.error);

Additional context

No response

[sheldonj]

I don't know if this is related but even the normal open models for streamObject return NaN for usage values.

[lgrammel]

@sheldonj for OpenAI it should work. In case you use createOpenAI, you need to set compatibility to strict: https://sdk.vercel.ai/providers/ai-sdk-providers/openai#provider-instance (this was necessary to prevent breaking changes with OpenAI-"compatible" providers).

[sheldonj]

@sheldonj for OpenAI it should work. In case you use createOpenAI, you need to set compatibility to strict: https://sdk.vercel.ai/providers/ai-sdk-providers/openai#provider-instance (this was necessary to prevent breaking changes with OpenAI-"compatible" providers).

Confirmed that using strict mode returns usage! Thank you