The context tax, and how to stop paying it

Before you blame the model, audit the harness.

A canary-analysis agent on my team started costing four times what it did at launch. Same model, same number of deployments judged, same logic. The change was that someone had wired it to pull the full Dynatrace problem payload into context on every judgment, and a noisy service was returning 6k tokens of problem detail per call. The agent re-read all of it on every turn of a multi-step judgment. Nobody touched the model. The bill quadrupled anyway.

That is the shape of the whole problem. The complaint everyone has right now is "we're burning tokens," and the reflex is to drop to a smaller model or cap the output. That treats a symptom. The token bill is what shows up in billing. What doesn't show up is that you're using the context window like scratch space you'll never have to clean, when it's a lossy working memory with a signal-to-noise ratio. The more junk you pack in, the worse the model reasons over it.

None of this is the weights' fault. Hand a frontier LLM 4k tokens of exactly the right context and it out-reasons the same model drowning in 90k tokens of scrollback, stale tool output, and dead branches. The intelligence is constant. What you control is the assembly layer: retrieval, compaction, state, ordering. That layer is the harness, and the harness is where wasted tokens come from. Context engineering is just the discipline of deciding what the model sees at each step, on purpose, with the smallest set of tokens that does the job.

So the real question is not which cheaper model to buy. It's why you're paying to re-read your own garbage on every turn. That recurring charge is the context tax. Here's where it comes from, and how to stop feeding it.

Why the bill goes quadratic when the work doesn't

Context growth is not inherently quadratic. It goes approximately quadratic when the entire growing conversation is retransmitted on every turn, which is exactly what a naive chat loop does. Here is the loop, mechanically. Turn one you send 2k tokens and get 500 back. Turn two the request carries the original 2k, the 500, and your new 2k. Turn three carries all of it again. The conversation doesn't cost the sum of the turns. It costs the sum of the running totals, because the model re-processes the whole prefix on every call.

That's why a chat that feels linear bills like something closer to quadratic. Ten turns that each add 1.5k tokens is not ten units of work, it's the triangular number of those steps, roughly fifty-five. The fix is not magic, it's refusing to retransmit the whole history: prune it, summarize it, or cache it. Prompt caching takes the edge off, but only on the stable prefix, and only if you keep that prefix byte-identical across calls. Most harnesses don't.

The signal-to-noise problem most teams never measure

Cost is the boring half. The expensive half is that a long, polluted context makes the model dumber, and you should be able to name the two ways it happens in a design review without waving your hands.

Four common ways teams waste context

These aren't exotic. They're the defaults you get when nobody owns the assembly layer and "make it work" means "paste more in." Read the right column as the invoice: every habit on the left bills you for it.

Six interventions that usually matter more than model selection

Ordered by leverage, not by what's easiest to grab. "Use a smaller model" sits dead last on purpose. It's the first thing everyone reaches for and the weakest fix, because a small model fed a landfill still fails. Fix the context first. Then talk about the model.

# bad: the model pays to parse the whole blob, every turn it stays in context
result = api.get_incident(incident_id)        # 4,000 tokens of JSON
context.append(result)

# good: the harness extracts, the model sees the answer in ~25 tokens
result = api.get_incident(incident_id)
context.append(
    f"incident {incident_id}: status={result['status']} "
    f"sev={result['severity']} owner={result['owner']} "
    f"affected={','.join(result['affected_services'])}"
)

The tool still does full work. The harness decides what crosses into the window - one field, not one blob, multiplied across every turn the result stays in scope.

Treat context as an asset, not a recurring expense

Most prompt advice stops at the single message. The bigger win is structural. When you work with a coding agent or anything that supports persistent instruction files, you stop re-pasting your conventions into every prompt, which is just the re-explaining habit wearing a different shirt. You write the context down once, in files scoped by how often they're needed, and each one loads only when it earns its place. They nest: one always-on, one on-demand, one per task. Same discipline as the levers above, pointed at your own workflow.

# Sentinel SRE platform

## Stack
Python 3.12, FastAPI, OpenTelemetry, Dynatrace, Kubernetes, ArgoCD.

## Commands
test:   `make test`         # run before every commit
lint:   `make lint`
deploy: `argocd app sync sentinel`   # never deploy by hand

## Conventions
- Tool output is compacted before it hits the model. No raw blobs in context.
- Every metric maps to a DURESS dimension or it doesn't ship.
- No timelines or week estimates in any doc. Build it now.
- INFO in prod, DEBUG by explicit toggle with a TTL.

## Don't
- Don't paste full API responses. Extract the fields. See SKILL: tool-compaction.
- Don't re-explain the stack every turn. It's here. Read it once.

The always-on layer. Every line is paid for on every turn, so it stays short and declarative. This is the stable prefix - hold the ordering fixed so the cache survives.

---
name: tool-compaction
description: Use when adding or editing any tool that returns data to the
  model. How to compact tool output at the boundary so raw blobs never enter
  the window. Trigger on: new tool, API wrapper, agent tool result.
---

# Compacting tool output at the boundary

Rule: the model sees a summary, never the raw payload.

1. Call the API and get the full result in your code.
2. Extract only the fields the model needs to reason or act.
3. Return a short, typed, human-readable string.
4. If the full blob might be needed later, write it to a file and
   return the path, not the contents.   # see SKILL: externalized-state

The on-demand layer. The agent reads the description to decide whether to pull the body into context. Write the description for matching, the body for doing - the body stays out of the window until the task calls for it.

# Task: canary DSL - add latency-budget gate

## Goal
Fail a canary when p95 latency exceeds a per-service budget pulled from the
SLO config, not a hardcoded number.

## Decided
- Budget source: existing slo.yaml, keyed by service. No new file.
- Runs in the existing Kayenta judge loop. No new service.
- Failure blocks promotion, same path as the error-rate gate.

## Open
- p95 at judge time: trace-derived SLI or metric? (Leaning trace. Confirm.)

## Done
- Gate config schema (canary/dsl/gates.py)
- Unit test for budget parsing

The per-task layer. Externalized working memory for one piece of work. Decided / Open / Done is enough structure. Update it as you go; the running history never has to carry the state.

Match the model to the task, not the other way round

L6 in the levers section told you to right-size per step. This is what that looks like in a development workflow. The mistake is not using expensive models — it's using them at the wrong frequency. A frontier model reviewing your PR once a day is a bargain. That same model firing on every keystroke is a budget fire you lit yourself.

Three tiers, three jobs. They're defined by task frequency and the nature of the reasoning required, not by how good you want the answer to be. All three tiers can produce a wrong answer; the question is what kind of wrong and how often the task demands judgment versus pattern completion.

Measure outcomes or you're optimizing the wrong thing

This is where AI feature work usually collapses. Someone wraps a model call in a UI, it demos well, nobody instruments it, and three months later the bill is five figures with no one able to say whether the thing is any good. If you can't answer the next question with numbers, it isn't a feature, it's a liability with a chat interface.

Here is the mistake almost every team makes, and it's worth being blunt about because it's the strongest argument in this whole piece. They optimize the metrics that are easy to read off a dashboard: token cost, model cost, p95 latency. Those feel like rigor. They aren't. They're inputs. The thing the business actually buys from an AI feature is resolved tasks: tickets closed, deployments correctly judged, documents correctly extracted, questions correctly answered without a human stepping in. None of the input metrics tell you whether that happened. You can cut token cost 40% by switching to a smaller model and quietly drop your resolution rate from 80% to 55%, and your cost dashboard will show a win while your feature gets materially worse. Every retry, every human escalation, every wrong answer someone has to catch and redo is a cost the per-call metric never sees.

So measure the thing you sell. Resolution rate, or successful automation rate, per task type. Then cost per completed task, which is total spend divided by the things that actually worked, retries and escalations included. That single number is the honest one, and it's usually the one nobody is tracking. A change that triples tokens per call but doubles resolution and halves retries is a large win that every input metric will tell you is a loss. If your reporting can't show that, your reporting is lying to you in the direction of looking frugal.