Do token-saving tools like rtk and caveman really work?
A little, and far less than advertised. JetBrains ran paired tests on Claude Code and compared the bill: caveman advertised 65% and cut output tokens by 8.5%; rtk advertised 60–90% and cost 7.6% more at low reasoning effort, the same at high. Neither made task quality worse. Before you install one, use /usage to see where your tokens go; afterwards, read your usage, not the number the tool reports.
Codex
No public paired test on Codex. Judge it the same way: compare the token usage of your own account, not the number the tool reports.
Both rtk and caveman support Codex, but the two tests above ran on Claude Code only, so the numbers do not carry over. The reasoning does: OpenAI says your prompt, files, chat history, tool results and the response all use tokens. Command output is one part of that, and compressing one part saves little overall. To check for yourself, /usage daily and /usage weekly in the Codex CLI show the token usage of your account, and /status shows what is left. OpenAI lists other ways to save: write precise prompts, give only the relevant files, keep AGENTS.md short, turn off MCP servers you do not need, and use a smaller model for routine work.
What the official docs say
Tokens are small units of information that ChatGPT reads and writes. Your prompt, files, chat history, tool results, and ChatGPT's response all use tokens.Limit the number of MCP servers you use. Every MCP server adds more context to your messages and uses more of your limit. Disable MCP servers when you don't need them./usage — View account token usage or use a rate-limit reset. Inspect daily, weekly, or cumulative ChatGPT token activity from inside the TUI.Claude
Measured savings are small: caveman cut output tokens by 8.5%; rtk cost 7.6% more or the same. Task quality did not drop.
JetBrains ran both tests: Claude Code, claude-sonnet-5, the 86 coding tasks of SkillsBench, each task run once with the tool and once without, then compared in pairs. caveman was forced on for every reply, the best case: output tokens fell 8.5% (592k to 542k over 82 task pairs), cost about 10%. rtk used its default install: at low reasoning effort the median cost per task was 7.6% higher (80 pairs, p=0.004); at high effort it was +0.1% (p=0.99). The advertised numbers are what the two READMEs said when the tests ran, in July 2026.
What the official docs say
rtk advertised saving: 60–90%. Measured on real agent work: +7.6% more expensive at low reasoning effort (p=0.004), ±0% at high effort. Setup: Claude Code 2.1.201 · claude-sonnet-5 low and high efforts · SkillsBench. Task quality: unchanged in both arms, at both effort levels.Advertised saving: 65%. Measured saving: 8.5%. Output-token saving on real agentic tasks, with the skill forcibly activated. This is the ceiling, not the usual-case result.a tool's self-reported savings are a claim about its counterfactual, not about your bill. rtk's scoreboard said 96 million tokens saved while the invoice went up. If you evaluate any context-compression tool, measure the paired bill, not the tool's diff.On a Pro, Max, Team, or Enterprise plan, /usage also shows a breakdown of what counts against your plan limits: Attribution: recent usage attributed to skills, subagents, plugins, and individual MCP servers, each shown as a percentage of the total.Why the savings are small
The part these tools can compress is small to begin with. In coding work, most agent output is code, diffs and tool calls, and caveman leaves all of that as it is. Only the short explanations between tool calls get shorter.
Claude Code reads and searches files with its own Read and Grep tools, which do not pass through the Bash hook that rtk uses. About a fifth of what the model reads is tool output that rtk can reach. JetBrains puts the best possible saving at about 3% of the bill.
If you only want to filter noisy command output, the Claude Code docs already include a hook example that returns failures only.
What the official docs say
Advertised savings come from chat-style prose answers. Agentic output is different: code, diffs, tool invocations, and exact error strings dominate the token stream, and Caveman correctly leaves all of it verbatim.Custom hooks can preprocess data before Claude sees it. Instead of Claude reading a 10,000-line log file to find errors, a hook can grep for ERROR and return only matching lines, reducing context from tens of thousands of tokens to hundreds.What the two projects say now
Both READMEs have changed since the tests. rtk now says it cuts up to 90% of the bash output the agent reads, and that this is not the same as cutting the bill by 90%. caveman cites the 8.5% result itself, and has since added a proxy that compresses tool output; JetBrains did not test that part.
The compression in both tools is real, and task quality held up in both tests. What differs is the kind of work and how the saving is counted.
What the official docs say
RTK cuts up to 90% of the bash output your agent reads. That is what RTK measures, and it is not the same as cutting your bill by 90%.The JetBrains number is why the proxy exists. They measured the skill alone, in July 2026, before the proxy shipped.Common misconceptions
- The tool says it saved tens of millions of tokens, so the bill dropped by as much
No. In the full low-effort run, the rtk counter reported 96.2 million tokens saved while the bill for the same tasks went up. JetBrains gives three reasons. It compares against the full raw output, which Claude Code would have truncated anyway. It estimates tokens as characters divided by 4 when the command runs, while most input cost in a session is re-reads at the cached rate. And the hook never sees most of the context.
- Install one token-saving skill and usage drops by more than half
No. The advertised 65% comes from chat-style answers. When an agent writes code, most of its output is code, diffs and tool calls, and these skills leave that alone. The measured ceiling is 8.5% fewer output tokens with the skill forced on for every reply; normal use saves less.
- The tests found small savings, so these tools are a scam
No. Task quality held up in both tests and the compression is real. The JetBrains verdict on rtk was “Honest engineering, wrong counterfactual.” Both READMEs now say as much: rtk separates cutting bash output from cutting the bill, and caveman cites the 8.5% figure. The gap comes from the kind of work and how the saving is counted, not from fake tools.
Tips
- Before you install anything, see where tokens go: in Claude Code run /usage (press d or w for 24 hours or 7 days) and /context. If long sessions or subagents take most of it, a tool that compresses command output will not help.
- To test a tool, compare your usage: a few days without it and a few days with it, on similar work, in /usage (/usage weekly in Codex). Do not take the tool counter as the result.
- Do not judge from one run. In the rtk test, repeating the same task unchanged moved the median cost by 22%. The first caveman run, on 10 tasks, showed 29.5% saved; on 86 tasks it was 8.5%.
- After installing, watch for extra turns. In the rtk test, sessions took 13.8% more turns and 14.3% more cache reads; that is where the extra cost went.
- To filter test and log output only, the Claude Code costs page has a ready PreToolUse hook example that returns failures only. No third-party tool is needed.
- Every skill has its own cost on each turn: its description is always carried.