Token Compression

Every LLM app starts innocent.

There is a prompt. Maybe a system message. Maybe one retrieved document. Then the model misses something, so you add a few more documents. Then you add chat history. Then tool output. Then a safety note. Then a schema. Then one more paragraph that explains the thing it missed in last Tuesday's demo.

Six weeks later, a simple user question is dragging a small filing cabinet into the context window.

That is the problem token compression is trying to solve.

Not "make the prompt tiny." That is easy, and usually wrong. The useful version is more boring: remove the parts the model does not need, keep the parts it will regret losing, and check whether the final answer still works.

What is being compressed?

A token is just the unit of text the model reads. Sometimes it maps to a word. Sometimes it is part of a word. Sometimes it is punctuation with a leading space. The exact split depends on the tokenizer, but the product problem is the same: long inputs cost more, take longer, and make attention fight through more noise.

A small version looks like this.

Search result: password reset help page.

Header, nav, footer, and related articles.

Old FAQ says reset links expire in 24 hours.

Current docs say reset links expire in 15 minutes.

If the user asks how long reset links last, most of that text is decorative. A compressor might keep:

Reset links expire in 15 minutes.

Old 24-hour FAQ is stale.

The important fact survived. So did the warning about the stale FAQ.

That example is intentionally small. The real version is usually uglier: transcripts with pleasantries, terminal logs with 500 passing tests, retrieved web pages with navigation chrome, or support tickets where the same detail is repeated eight times with slightly different wording.

Why this matters

The default answer to context problems has been "retrieve more" or "use a bigger context window." Both help. Neither makes noise disappear.

In a RAG system, the retriever often grabs the right document and four almost right documents. That can be a good retrieval result and still make a messy prompt. The LLM now has to find the useful ten lines inside a few thousand lines of prose, tables, boilerplate, and maybe old information that contradicts the current answer.

Cost is the obvious part. Tokens cost money. Latency is the second obvious part. More input means more reading. The part that is easiest to ignore is quality: irrelevant context can make the model worse.

I have seen this in agent systems too. A coding agent runs a test command, the test runner prints a wall of successful cases, and the one failure is somewhere near the bottom. Humans skim past the boring part. Models are forced to ingest it unless we clean it up first.

Compression is not summarization

Summarization rewrites. That is useful for many jobs, but it is a bad instinct when exact details matter.

These changes look tiny until they break the answer:

For workloads where exact details matter, the safer compressors tend to be extractive. They delete. They keep spans from the original input instead of inventing a cleaner version. That is less elegant than a summary, but it is safer for numbers, dates, IDs, code, file paths, JSON keys, citations, names, and negations.

The word not is cheap until you drop it.

The shape of a compressor

The simplest mental model is:

The compressor can be a model, a rules engine, or a command-specific filter. It can operate on tokens, lines, chunks, JSON fields, log groups, or document sections. The best unit depends on the data.

That constraint matters. Test output has failure lines and summaries. A meeting transcript has turns and decisions. In my experience, a compressor that ignores that structure will eventually delete the thing you needed.

How I would build one

I would not start with a general prompt compressor. I would pick one annoying input type and one measurable task.

For example:

• terminal output into coding-agent decisions
• support tickets into customer replies
• call transcripts into action items
• SEC filings into financial QA
• retrieved web pages into RAG answers

Then I would collect examples with four fields:

question

original context

expected answer

compressed context

The compressed context does not need to be beautiful. It needs to preserve the answer. That difference keeps the work honest.

For a RAG task, one example might look like this:

Question: How long does a password reset link last?

Original context: The retrieved page includes nav links, footer text, and related articles. An old FAQ says reset links last 24 hours. The current docs say reset links expire after 15 minutes.

Expected answer: Password reset links last 15 minutes.

Compressed context: Reset links expire after 15 minutes. The old 24-hour FAQ is stale.

From there, the training loop is not mysterious:

The last box is the one I would care about most. Token-level labels are a proxy. The actual question is whether the final system still answers correctly.

How to measure it

I would track four numbers:

Average quality can hide the failure you actually care about. A compressor can look fine on ordinary tickets and still be dangerous on legal clauses, medical records, source code, schemas, citations, or anything where exact wording matters.

For those inputs, I would start with light compression and protected spans. If the text contains an ID, date, URL, file path, or JSON key, make the compressor work hard before it is allowed to delete it. I would be even more careful around code blocks, citations, and negations.

Three useful examples

Three projects show the range:

• Bear / The Token Company puts a hosted compressor in front of the LLM you already use.
• RTK compresses command output before the LLM sees it, which makes it easy to try on a developer workflow.
• LLMLingua-2 is the build-it-yourself pattern: use a stronger model to make deletion labels, then train a smaller model to make those keep/drop calls cheaply.

Bear and The Token Company

The Token Company is the hosted version of this idea. Bear sits between your app and the LLM API: the full prompt goes in, a shorter prompt comes out, and you send that to the final model.

That is convenient, but it is also the tradeoff. I did not find a public Bear model you can host yourself. Their docs describe an extra API call before your LLM call, and their pricing is based on tokens saved. If you use it, another vendor sees the prompt and another meter stays in the request path.

RTK

RTK is the option I would try first because it compresses something developers already know is noisy: shell output.

Coding agents spend a surprising amount of context on commands:

• test suites where almost everything passed
• install logs full of package chatter
• giant git diff output
• repeated warnings
• progress bars
• directory listings that are technically correct and practically useless

RTK wraps commands and shows the agent a smaller version. The project describes 60 to 90 percent token savings on common development commands where command interception is supported.

RTK is useful because it does not treat terminal output as generic prose. It knows a test log has a shape.

This is why I find RTK more compelling than a generic "summarize my terminal" step. A test log has conventions. A good compressor should exploit them.

The quickest way to get a feel for RTK is to install it and run the same command twice:

brew install rtk
rtk --version

Start with something small:

$ git status
On branch main
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   src/password.ts
 
Untracked files:
  (use "git add <file>..." to include in what will be committed)
        .gitignore
        debug.log
 
no changes added to commit (use "git add" and/or "git commit -a")
 
$ rtk git status
* main
 M src/password.ts
?? .gitignore
?? debug.log

I tried that in a tiny demo repo with one modified file and two new files. The normal output is helpful for humans, but it spends a lot of words teaching Git. RTK keeps the state the agent actually needs while still looking like Git output, not a paraphrased summary. After hook setup, RTK can intercept supported commands before the agent sees the output, so the compact version becomes the default thing in context.

rtk init -g

Then try a command with real noise. I used a tiny Vitest file with one passing test and one failing test:

$ vitest run
RUN  v4.1.10 ./rtk-vitest-demo
 
src/password.test.js (2 tests | 1 failed)
  x guest reset links use the current policy
 
AssertionError: expected 60 to be 15
 
- Expected
+ Received
 
- 15
+ 60
 
Test Files  1 failed (1)
Tests       1 failed | 1 passed (2)
Duration    341ms
 
$ rtk vitest run
PASS (1) FAIL (1)
 
1. guest reset links use the current policy
   AssertionError: expected 60 to be 15
   at src/password.test.js:12:39

RTK is not a general compressor, and that is why it is a good place to start. It has a narrow input type, a clear before/after, and local feedback. LLMLingua-2 asks the same question in a more general way: can you teach a smaller model to make those keep/drop decisions cheaply?

LLMLingua-2

LLMLingua-2 is the version I would read if I wanted to build a compressor instead of buy one. The useful idea is simple: stop asking the final model to be clever about a giant prompt every time. Teach a smaller model which tokens are worth keeping.

One detail in the paper feels especially practical. A tempting shortcut is to ask: which tokens would the model probably guess anyway? If a token is obvious from the surrounding text, older compressors often treated it as less important and easier to drop. But "guessable" is not the same as "safe to remove." In a sentence like "refunds are not available after 30 days," the word not is tiny. The number 30 is easy to miss. Drop either one and the answer changes.

So LLMLingua-2 turns compression into a labeling problem. GPT-4 does a deletion-only pass: no new words, no reordering, no prettier paraphrase. The kept words are matched back to their positions in the original text. Now each token has a label: keep it or drop it. Then a smaller encoder learns that pattern and makes the cheap decision at runtime.

The paper reports 2x to 5x compression, 3x to 6x faster compression than prior methods, and 1.6x to 2.9x lower end-to-end latency. I would not treat those numbers as a promise for every workload. The part worth stealing is the recipe: spend the expensive model once to make the training signal, then let the smaller model do the repetitive work.

The mental model

Retrieval decides what enters the prompt. Ranking decides what appears first. Compression decides how much junk each piece carries.

That is the useful place to put it in your head. Compression is not a replacement for better retrieval, better chunking, or a larger context window. It is the step that asks one last boring question before the model reads: is this text still doing work?

Sometimes the answer is yes. Keep the weird error line, the old policy that contradicts the new one, the file path, the number, the not. Those are cheap to send and expensive to lose.

But if a prompt is full of copied navigation, repeated logs, passing test output, or the same support-ticket detail written five different ways, the model is not being given more context. It is being given more chores.

That is why I like compression as a practical tool. Not because smaller prompts are automatically better, but because every token should have a reason to be there.