I Couldn’t Fix My LLM Costs Until I Measured Tokens Per Feature

My LLM bill kept growing, so I did what seemed obvious: I looked for a cheaper model.

That helped a little, but it didn’t explain why the bill was growing.

The dashboard could tell me how many tokens the application used. It couldn’t tell me what those tokens were doing.

Were they coming from chat?

Document summaries?

Background classification?

An agent retrying the same tool call?

I was trying to optimize a total without knowing which product feature created it.

The useful unit wasn’t tokens per model.

It was tokens per feature.

Model-level totals hid the real problem

A provider dashboard usually groups usage by model, API key, project, or time period.

That is useful for billing, but not always for product decisions.

Imagine an application with four LLM-powered features:

• interactive chat
• document summarization
• support-ticket classification
• an agent that prepares weekly reports

If the bill increases by 30%, the model name doesn’t explain which feature changed.

Maybe chat traffic grew.

Maybe summarization started sending entire documents instead of selected sections.

Maybe the classifier received a much larger system prompt.

Maybe the report agent retried after tool failures and generated the same plan several times.

Those problems require completely different fixes.

Switching every request to a cheaper model would reduce the bill, but it could also hide the engineering mistake.

Tag every request with a feature

I started giving every LLM call a small amount of application context:

const context = {
  feature: "document_summary",
  operation: "initial_summary",
  customer_tier: "pro"
};

Enter fullscreen mode Exit fullscreen mode

The model provider doesn’t need these fields.

They belong in the application’s usage record.

I avoid using individual user IDs as the primary grouping dimension. For cost analysis, a product feature, workflow, or operation is normally more useful and creates fewer privacy problems.

A practical record looks like this:

{
  "timestamp": "2026-07-22T03:12:48.201Z",
  "feature": "document_summary",
  "operation": "initial_summary",
  "model": "example-model",
  "input_tokens": 4821,
  "output_tokens": 614,
  "total_tokens": 5435,
  "latency_ms": 2834,
  "status": "success"
}

Enter fullscreen mode Exit fullscreen mode

Once I had that record for every request, I could answer better questions:

• Which feature uses the most tokens?
• Which feature has the fastest usage growth?
• How many tokens does one successful operation require?
• Are retries increasing tokens without increasing completed work?
• Is the input growing faster than the output?
• Which feature is using an expensive model without needing it?

A small Node.js usage recorder

Here is a minimal implementation using an OpenAI-compatible chat-completions endpoint.

It uses only built-in Node.js modules and expects Node 18 or newer.

Create llm-client.mjs:

import { appendFile } from "node:fs/promises";
import { randomUUID } from "node:crypto";

const API_URL =
  process.env.LLM_API_URL ??
  "https://api.example.com/v1/chat/completions";

const API_KEY = process.env.LLM_API_KEY;
const USAGE_FILE =
  process.env.LLM_USAGE_FILE ?? "./llm-usage.jsonl";

if (!API_KEY) {
  throw new Error("LLM_API_KEY is required");
}

async function writeUsage(record) {
  await appendFile(
    USAGE_FILE,
    `${JSON.stringify(record)}\n`,
    "utf8"
  );
}

export async function createChatCompletion({
  feature,
  operation,
  model,
  messages,
  temperature = 0
}) {
  if (!feature || !operation) {
    throw new Error(
      "Every LLM request needs a feature and operation"
    );
  }

  const requestId = randomUUID();
  const startedAt = Date.now();

  try {
    const response = await fetch(API_URL, {
      method: "POST",
      headers: {
        "content-type": "application/json",
        authorization: `Bearer ${API_KEY}`,
        "x-client-request-id": requestId
      },
      body: JSON.stringify({
        model,
        messages,
        temperature
      })
    });

    const body = await response.json();

    if (!response.ok) {
      throw new Error(
        body?.error?.message ??
        `LLM request failed with status ${response.status}`
      );
    }

    const usage = body.usage ?? {};

    await writeUsage({
      timestamp: new Date().toISOString(),
      request_id: requestId,
      feature,
      operation,
      model,
      input_tokens:
        usage.prompt_tokens ??
        usage.input_tokens ??
        null,
      output_tokens:
        usage.completion_tokens ??
        usage.output_tokens ??
        null,
      total_tokens: usage.total_tokens ?? null,
      latency_ms: Date.now() - startedAt,
      status: "success"
    });

    return body;
  } catch (error) {
    await writeUsage({
      timestamp: new Date().toISOString(),
      request_id: requestId,
      feature,
      operation,
      model,
      input_tokens: null,
      output_tokens: null,
      total_tokens: null,
      latency_ms: Date.now() - startedAt,
      status: "error",
      error: error?.message ?? String(error)
    });

    throw error;
  }
}

Enter fullscreen mode Exit fullscreen mode

A feature calls the wrapper like this:

import {
  createChatCompletion
} from "./llm-client.mjs";

const result = await createChatCompletion({
  feature: "document_summary",
  operation: "initial_summary",
  model: "example-model",
  messages: [
    {
      role: "system",
      content:
        "Summarize the document into five concise bullet points."
    },
    {
      role: "user",
      content: "Document content goes here."
    }
  ]
});

console.log(result.choices[0].message.content);

Enter fullscreen mode Exit fullscreen mode

The wrapper writes one line to llm-usage.jsonl for every request.

It does not store the prompt or model response. For feature-level cost analysis, I usually need usage metadata, not user content.

Summarize tokens by feature

The raw JSONL file is useful for debugging, but the first report I want is much simpler:

Feature                  Requests   Input      Output     Total
document_summary         42         182,140    21,382     203,522
interactive_chat         391        96,241     44,829     141,070
weekly_report_agent      18         81,440     19,205     100,645
ticket_classification    804        51,462     8,214      59,676

Enter fullscreen mode Exit fullscreen mode

Create summarize-usage.mjs:

import { readFile } from "node:fs/promises";

const file =
  process.env.LLM_USAGE_FILE ?? "./llm-usage.jsonl";

const content = await readFile(file, "utf8");

const records = content
  .split("\n")
  .filter(Boolean)
  .map(line => JSON.parse(line))
  .filter(record => record.status === "success");

const features = new Map();

for (const record of records) {
  const current = features.get(record.feature) ?? {
    feature: record.feature,
    requests: 0,
    input_tokens: 0,
    output_tokens: 0,
    total_tokens: 0,
    missing_usage: 0
  };

  current.requests += 1;

  if (record.total_tokens == null) {
    current.missing_usage += 1;
  } else {
    current.input_tokens += record.input_tokens ?? 0;
    current.output_tokens += record.output_tokens ?? 0;
    current.total_tokens += record.total_tokens;
  }

  features.set(record.feature, current);
}

const result = [...features.values()]
  .sort((a, b) => b.total_tokens - a.total_tokens);

console.table(result);

Enter fullscreen mode Exit fullscreen mode

Run it with:

node summarize-usage.mjs

Enter fullscreen mode Exit fullscreen mode

The absolute totals are only the first layer.

I also calculate tokens per successful operation:

const tokensPerRequest =
  feature.total_tokens / feature.requests;

Enter fullscreen mode Exit fullscreen mode

For agent workflows, I prefer tokens per completed workflow rather than tokens per API request.

One user action might trigger five model calls. If I optimize each request separately without tracking the completed action, I can make the request-level metrics look better while the workflow still wastes tokens.

Add operation-level detail

A feature tag tells me where the usage came from.

An operation tag tells me what happened inside that feature.

For example:

weekly_report_agent
├── create_plan
├── call_data_tool
├── repair_tool_arguments
├── draft_report
└── revise_report

Enter fullscreen mode Exit fullscreen mode

Suppose weekly_report_agent consumes 100,000 tokens.

That total alone doesn’t reveal much.

If 45,000 tokens come from repair_tool_arguments, I probably don’t need a cheaper writing model. I need to understand why the tool call keeps failing.

If draft_report input tokens keep growing, I might be sending too much raw source material.

If create_plan runs three times for a single report, the retry or state-management logic needs attention.

The feature tells me where to look.

The operation tells me what to fix.

Measure retries separately

Retries are easy to miss because the successful response looks normal.

I add an attempt number to each record:

{
  feature: "weekly_report_agent",
  operation: "draft_report",
  attempt: 2
}

Enter fullscreen mode Exit fullscreen mode

Then I compare:

• total requests
• unique operation IDs
• successful operations
• retry tokens
• tokens per successful operation

This prevents a misleading result where traffic appears stable but token usage doubles because requests are being repeated internally.

An operation ID can be created once at the beginning of the workflow:

const operationId = randomUUID();

Enter fullscreen mode Exit fullscreen mode

Every retry keeps the same operation ID but increments the attempt:

{
  operation_id: operationId,
  attempt: 2
}

Enter fullscreen mode Exit fullscreen mode

Now retry waste can be measured directly instead of inferred from a monthly bill.

Convert tokens to cost outside the request path

I don’t hardcode model prices inside the API wrapper.

Prices change, and different providers may expose different input, cached-input, and output rates.

Instead, I keep a separate rate table:

const rates = {
  "example-model": {
    input_per_million: 1.00,
    output_per_million: 4.00
  }
};

Enter fullscreen mode Exit fullscreen mode

Then estimate cost during reporting:

function estimateCost(record, rate) {
  const inputCost =
    ((record.input_tokens ?? 0) / 1_000_000) *
    rate.input_per_million;

  const outputCost =
    ((record.output_tokens ?? 0) / 1_000_000) *
    rate.output_per_million;

  return inputCost + outputCost;
}

Enter fullscreen mode Exit fullscreen mode

The numbers above are placeholders, not current pricing.

Before using the report for billing decisions, I replace them with the current rates from the provider and record the effective date of that rate table.

Keeping pricing outside the request wrapper also lets me recalculate historical usage after a pricing change without modifying the original token records.

Missing usage is a metric too

Not every API response includes token usage in the same format.

Streaming responses may require an additional option to return usage. Some providers expose different field names. Failed requests may not return usage at all.

I don’t silently convert missing usage to zero.

Zero means the request used no tokens.

null means I don’t know.

Those are very different statements.

The report includes a missing_usage count for each feature. If that number grows, the cost report is becoming less trustworthy even if the visible totals look stable.

What I optimize first

Once usage is grouped by feature and operation, I work down this list:

1. Unnecessary calls

Is the feature calling the model when a cached result, deterministic function, or database query would work?

1. Repeated context

Is every request sending the same large document, tool schema, conversation history, or instructions?

1. Retry waste

Are timeouts, invalid tool arguments, or parsing failures causing the same operation to run again?

1. Oversized outputs

Does a classification task need 800 generated tokens, or would a small structured response be enough?

1. Model selection

After fixing the request shape and workflow behavior, is the current model still necessary for this operation?

Model selection matters. It just isn’t always the first problem.

The metric I was missing

A monthly LLM bill tells me the result.

Tokens per feature tell me where the result came from.

Tokens per successful operation go one step further: they connect infrastructure usage to something the product actually accomplished.

That changed the questions I ask.

Instead of:

Which model should I replace?

I can ask:

Why did document summarization input grow by 40%?

Why does one completed report require nine model calls?

Why are retry tokens increasing while completed workflows stay flat?

Those questions lead to engineering fixes, not just cheaper invoices.

I work on TokenBay, so I regularly deal with multiple models behind an OpenAI-compatible interface. Model-level usage is still useful, but feature and operation tags are what make that usage actionable inside an application.

The next thing I’m adding is a small budget guardrail: not a global monthly limit, but a token budget for each completed feature operation.