Skip to content

fix(providers): bill cached prompt tokens at the cached rate for the remaining Chat Completions providers - #10284

Open
waleedlatif1 wants to merge 3 commits into
stagingfrom
fix/compat-cache-billing
Open

waleedlatif1 wants to merge 3 commits into
stagingfrom
fix/compat-cache-billing

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Follow-up to improvement(models): retire deep research, refresh model lifecycle, pricing, and reseller catalogs #10255: Meta, Mistral, Sakana, Cerebras, NVIDIA, Azure OpenAI chat-completions deployments and the shared Chat Completions path (Fireworks, OpenRouter, Together, Baseten, Nebius, Thinking Machines), and the self-hosted vLLM/LiteLLM core still priced the whole prompt_tokens at the input rate, overbilling every cached prompt token on models with a cachedInput price
  • All of them now meter each model call with the shared createChatCompletionUsageMeter, the same path Z.ai, Kimi, DeepSeek, Groq and xAI use, so cache hits bill at the cached rate and long-context tiers apply per call
  • Non-streaming responses now return their own cost, because the central fallback prices tokens.input only and would leave the cache-read split unbilled
  • Removes About 280 lines of duplicated token/cost bookkeeping

Type of Change

  • Bug fix

Testing

  • cache-billing.test.ts: added Meta, Mistral, Sakana and Fireworks (shared path) rows; all four fail against the pre-fix providers and pass now
  • settled-tool-streams.test.ts: every provider now prices each call, so the per-provider cost helper became one constant
  • Token expectations in the Azure, Mistral, Nebius, OpenRouter, vLLM and LiteLLM tests include cacheRead: 0
  • vitest run providers/ (2,214 tests), biome, check:test-patterns, check:unused-exports, check:explicit-any, pre-push type-check

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Nov 21, 2026 9:46am UTC

Request Review

@greptile-apps

greptile-apps Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[High impact] The PR appears safe to merge; no blocking issues were found.

Summary

The PR moves the remaining Chat Completions providers onto createChatCompletionUsageMeter.

  • Provider adapters now price cached prompt tokens at their cached rate.
  • Shared and self-hosted Chat Completions routes price cached tokens separately.

No new actionable issues were found.

Reviews (2) · Last reviewed commit: "test(providers): cover streamed cache hi..." · Reviewed by Greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

This branch was previously deployed

1 inactive deployment
Preview — 65d61688 Deployed Nov 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant