Metadata-Only Tracing: Privacy-First Observability for AI Agents

Agent tracing is useful because it reveals execution structure: which step ran, which tool failed, where a retry occurred, how long a model call took, and how the token budget changed.

The easiest implementation is to capture every prompt, argument, result, and response. It is also the easiest way to turn an observability system into a second copy of sensitive application data.

Metadata-only tracing takes a different approach. It records the behavior of an agent without storing its raw payloads by default. The result is not zero-risk telemetry, but it is a much smaller and more governable data surface.

The Design Goal

A useful metadata trace should answer operational questions such as:

• Which steps executed, and in what order?
• Which model and tool operations succeeded or failed?
• Where did latency accumulate?
• How many retries and fallbacks occurred?
• How many input, cached-input, and output tokens were used?
• Did retrieval return results, and how much context was assembled?
• Which validation or policy gate blocked the run?

It should not answer these questions unless a separate capture policy explicitly allows it:

• What did the user say word for word?
• What was the complete prompt or model response?
• Which email address, account number, or authorization token was used?
• What records or documents did a tool return?

That boundary keeps everyday traces useful without making full-fidelity capture the default.

Metadata-Only Does Not Mean Anonymous

Metadata can still be sensitive. A workflow name, fine-grained location, unique identifier, decision label, or rare error category may identify a person or reveal confidential business activity.

The relevant distinction is not “payload versus harmless metadata.” It is necessary, classified metadata versus unbounded content. Every field still needs a purpose, an owner, and a retention policy.

Avoid free-form metadata bags. A type such as Record<string, string | number> constrains value shapes, but it does not prevent a developer from adding email, prompt, or accessToken.

Define Metadata by Operation

Operation-specific types make the intended schema visible during code review and prevent arbitrary fields from spreading through the trace system.

type StepMetadata = {
  retrieval: {
    source: 'knowledge_base' | 'ticket_index';
    requestedTopK: number;
    resultCount: number;
    contextTokens: number;
  };
  model: {
    provider: 'openai' | 'anthropic' | 'google' | 'other';
    model: string;
    inputTokens: number;
    cachedInputTokens: number;
    outputTokens: number;
    finishReason: 'stop' | 'length' | 'tool' | 'other';
  };
  tool: {
    tool: 'lookup_order' | 'search_docs' | 'create_ticket';
    result: 'found' | 'not_found' | 'created' | 'rejected';
    retryCount: number;
  };
  policy: {
    policy: 'input_validation' | 'tool_authorization' | 'output_check';
    outcome: 'allow' | 'block';
    reason: 'valid' | 'invalid_shape' | 'not_authorized' | 'unsafe_output';
  };
};

type StepKind = keyof StepMetadata;

Enter fullscreen mode Exit fullscreen mode

Controlled vocabularies are intentional. They make dashboards stable, reduce high-cardinality fields, and force new data collection to be reviewed as a schema change.

Model names may remain dynamic, but they should still be length-limited and normalized. User-controlled strings should not be copied into these fields.

Use a Small, Versioned Event Envelope

The trace envelope should support parent-child relationships without carrying application payloads.

type TraceStatus = 'ok' | 'error';

type StepCompleted<K extends StepKind = StepKind> = {
  version: 1;
  event: 'step_completed';
  traceId: string;
  spanId: string;
  parentSpanId: string | null;
  timestamp: string;
  kind: K;
  name: string;
  status: TraceStatus;
  durationMs: number;
  errorCategory?:
    | 'timeout'
    | 'rate_limit'
    | 'validation'
    | 'authorization'
    | 'dependency'
    | 'unknown';
  metadata?: StepMetadata[K];
};

Enter fullscreen mode Exit fullscreen mode

Versioning matters because trace artifacts often outlive the code that produced them. A version field lets readers migrate or reject incompatible events rather than guessing their shape.

Do not include raw error messages or stack traces in the default event. Both frequently contain payload fragments, file paths, headers, or query values. Map exceptions to a controlled category and keep richer diagnostics behind a restricted capture mode.

Preserve Parent-Child Context in TypeScript

AsyncLocalStorage can carry trace and parent-span identifiers across promise chains without passing them through every function signature. The tracer below emits completion events and requires each operation to return metadata that matches its declared kind.

import { AsyncLocalStorage } from 'node:async_hooks';
import { randomUUID } from 'node:crypto';

type TraceContext = {
  traceId: string;
  spanId: string | null;
};

type StepOutput<T, K extends StepKind> = {
  value: T;
  metadata: StepMetadata[K];
};

export interface TraceSink {
  write(event: StepCompleted): Promise<void>;
}

const traceContext = new AsyncLocalStorage<TraceContext>();

function categorizeError(error: unknown): StepCompleted['errorCategory'] {
  if (!(error instanceof Error)) return 'unknown';
  if (error.name === 'AbortError') return 'timeout';
  if (error.name === 'ValidationError') return 'validation';
  if (error.name === 'AuthorizationError') return 'authorization';
  return 'dependency';
}

export async function runTrace<T>(
  work: () => Promise<T>,
): Promise<T> {
  return traceContext.run(
    { traceId: randomUUID(), spanId: null },
    work,
  );
}

export async function traceStep<K extends StepKind, T>(
  sink: TraceSink,
  kind: K,
  name: string,
  work: () => Promise<StepOutput<T, K>>,
): Promise<T> {
  const parent = traceContext.getStore();
  if (!parent) throw new Error('traceStep must run inside runTrace');

  const spanId = randomUUID();
  const startedAt = Date.now();

  try {
    const output = await traceContext.run(
      { traceId: parent.traceId, spanId },
      work,
    );

    await sink.write({
      version: 1,
      event: 'step_completed',
      traceId: parent.traceId,
      spanId,
      parentSpanId: parent.spanId,
      timestamp: new Date().toISOString(),
      kind,
      name,
      status: 'ok',
      durationMs: Date.now() - startedAt,
      metadata: output.metadata,
    });

    return output.value;
  } catch (error) {
    await sink.write({
      version: 1,
      event: 'step_completed',
      traceId: parent.traceId,
      spanId,
      parentSpanId: parent.spanId,
      timestamp: new Date().toISOString(),
      kind,
      name,
      status: 'error',
      durationMs: Date.now() - startedAt,
      errorCategory: categorizeError(error),
    });

    throw error;
  }
}

Enter fullscreen mode Exit fullscreen mode

The sink can write to a local NDJSON file during development or export approved events to an observability backend. Capture policy belongs before the sink so changing destinations cannot silently increase what is collected.

Instrument an Agent Without Capturing Content

The model and retrieval operations can use sensitive values in memory while returning only bounded operational metadata to the tracer.

const answer = await runTrace(async () => {
  const documents = await traceStep(
    sink,
    'retrieval',
    'retrieve_support_docs',
    async () => {
      const value = await searchDocuments(userQuestion);

      return {
        value,
        metadata: {
          source: 'knowledge_base',
          requestedTopK: 5,
          resultCount: value.length,
          contextTokens: countDocumentTokens(value),
        },
      };
    },
  );

  return traceStep(
    sink,
    'model',
    'generate_support_answer',
    async () => {
      const response = await callModel(userQuestion, documents);

      return {
        value: response.text,
        metadata: {
          provider: 'other',
          model: response.model.slice(0, 80),
          inputTokens: response.usage.inputTokens,
          cachedInputTokens: response.usage.cachedInputTokens ?? 0,
          outputTokens: response.usage.outputTokens,
          finishReason: response.finishReason,
        },
      };
    },
  );
});

Enter fullscreen mode Exit fullscreen mode

This trace can reveal an empty retrieval result, an oversized context, a length-limited response, or an unexpectedly expensive model call. It never needs the user question or document text.

What a Useful Trace Looks Like

support_agent  1,184 ms  ok
├─ retrieve_support_docs   96 ms  ok
│  source=knowledge_base resultCount=5 contextTokens=0
└─ generate_support_answer 1,072 ms ok
   inputTokens=640 cachedInputTokens=0 outputTokens=84 finishReason=stop

Enter fullscreen mode Exit fullscreen mode

The zero-token retrieval context is immediately suspicious even though the trace does not expose the documents. Metadata narrows the investigation; a developer can then enable selective local capture for that step if the issue cannot be reproduced otherwise.

Enforce Runtime Limits Too

TypeScript types disappear at runtime, and trace data may come from JavaScript adapters or external libraries. Validate events before writing them:

• Reject unknown keys and unsupported schema versions.
• Cap names and other strings to a small maximum length.
• Require finite, non-negative numeric values.
• Limit metadata field counts and serialized event size.
• Reject keys associated with payloads, credentials, headers, or free-form content.
• Fail closed when validation cannot complete.

Schema validation libraries can enforce the structural rules. The operation-specific builders should still remain the primary policy boundary.

Test the Negative Requirements

Privacy requirements are often about what must never appear. Encode those expectations in tests.

const forbiddenKeys = [
  'prompt',
  'response',
  'args',
  'resultBody',
  'authorization',
  'cookie',
  'email',
] as const;

function assertNoForbiddenKeys(event: unknown): void {
  const serialized = JSON.stringify(event).toLowerCase();

  for (const key of forbiddenKeys) {
    if (serialized.includes(`"${key.toLowerCase()}"`)) {
      throw new Error(`Forbidden trace key: ${key}`);
    }
  }
}

Enter fullscreen mode Exit fullscreen mode

Add representative secrets and personal data to test fixtures, run the agent, and assert that none of those values appear in the emitted trace. This is not a substitute for a broader security review, but it catches regressions when instrumentation changes.

Know When Metadata Is Not Enough

Metadata-only tracing is excellent for timing, topology, retries, token usage, policy outcomes, and broad error localization. It cannot explain every semantic failure.

When exact content is necessary, use a separate diagnostic mode with these constraints:

• Enable it for a specific trace, step, or short time window.
• Keep capture local or in a restricted incident environment.
• Allowlist fields instead of recording complete objects.
• Apply deterministic redaction and secret scanning.
• Display that enhanced capture is active.
• Delete the artifact automatically after a short retention period.

The escalation path should be obvious, but it should require intent.

A Practical Default

Start with a versioned event envelope, operation-specific metadata types, async parent-child context, controlled error categories, and runtime validation. Store timing, status, token usage, counts, and bounded labels. Keep prompts, outputs, tool payloads, retrieved text, headers, and environment values out of the default schema.

Metadata-only tracing will not answer every debugging question. It will answer a large portion of them while substantially reducing the amount of sensitive data your observability system must protect.

The next article will focus on the TypeScript runtime itself: async context propagation, module boundaries, serverless execution, and the hooks a tracing tool needs to handle cleanly.