Cost, Latency & Caching

A model in your game costs nothing while you are testing alone. The number that matters is the one at ten thousand players, and it is worth calculating before you design around the feature rather than after.

Pick the cheapest model that holds the character

Do the arithmetic first

The shape of the calculation matters more than any particular number, because your own numbers will differ:

1. 1
   Tokens per exchange

   System prompt + history + player line (input), plus the reply (output). A short NPC exchange is often ~600 in and ~80 out.

2. 2
   Exchanges per session

   Be honest. Players who like a system use it far more than your average suggests.

3. 3
   Sessions per player per month

   From your existing analytics, not a guess.

4. 4
   Multiply by your player count

   Then multiply again by three, because engaged players are not average players.

Prompt caching: the biggest single lever

Your system prompt — the character, the rules, the world facts — is identical on every request and is usually most of your input tokens. Caching makes the API charge roughly a tenth for that portion on subsequent calls.

{
  "model": "claude-haiku-4-5",
  "max_tokens": 300,
  "system": [
    {
      "type": "text",
      "text": "<the long, unchanging character description>",
      "cache_control": { "type": "ephemeral" }
    }
  ],
  "messages": [{ "role": "user", "content": "Hello?" }]
}

Verify rather than assume: the API response reports how many tokens were written to and read from cache. If the read count stays at zero across repeated requests, something in your prefix is changing.

Cutting latency

• Stream. The largest perceived win by far — see Streaming NPC Dialogue.
• Use a faster model for lines that do not need brilliance.
• Ask for less output. Generation time scales with reply length; a two-sentence cap is a latency fix.
• Trim history. Keep the last few turns plus a short summary, not the whole conversation.
• Start early. Fire the request when the player enters the trigger volume, not when they press the talk button.

Budget controls that belong on the server

All of these live on your proxy, because that is the only place a player cannot bypass them.

• Per-player rate limits. One player should not be able to spend your monthly budget.
• A global daily ceiling. When it trips, serve pre-written lines instead of erroring.
• Spend alerts on the API account. Set these on day one, not after the first surprise.
• Per-player usage logging. Without it you cannot tell abuse from enthusiasm.

Fallbacks are a feature, not error handling

The network fails, budgets trip, and requests occasionally get declined. Every one of those paths needs a line the NPC can say, and the line should be good enough that a player who never sees a generated response still enjoys the character.

// Written by you, shipped with the game, works offline.
[SerializeField] private string[] fallbackLines =
{
    "Forge's cold. Come back later.",
    "Not much to say today.",
    "Mm. Busy.",
};

private string FallbackLine() =>
    fallbackLines[UnityEngine.Random.Range(0, fallbackLines.Length)];

What to take away

• Output tokens cost several times input — cap reply length.
• Cost per message grows with conversation length, because history resends.
• Cache the system prompt; keep everything volatile after it.
• A per-request value near the front of the prompt silently kills caching.
• Streaming and starting early buy more perceived speed than a faster model.
• Rate limits, ceilings, and spend alerts belong on the server, from day one.
• Ship fallback lines good enough to stand on their own.