rtk: savings 160x lower than claimed

I spent a month running rtk — a proxy that filters shell command output to save LLM tokens. Its analytics reported 689 million tokens saved (93.3%).

rtk hooks into the ide and rewrites bash commands: git status becomes rtk git status, and the output gets filtered and compressed. Two numbers go into its database: how many tokens the raw output contained and how many remained after filtering. The difference is declared “saved”. Looks perfectly logical: the bigger the output, the bigger the savings.

The catch is the baseline. claude code caps the output of any bash command at 30 thousand characters (BASH_MAX_OUTPUT_LENGTH, documentation); everything above the limit is saved to a file, and only a ~2 kilobyte preview enters the context. codex simply truncates output at 10K, cutting out the middle.

As a result, the savings from rtk are nearly invisible — under $10 a week. And that is the upper bound when paying per API. On a subscription, apart from the pretty numbers in rtk gain, there is no visible effect at all. For comparison — the claimed 689 million tokens would cost about $6800 at fable pricing.

My point is — benchmarks are hard. Both rtk and caveman are tools with 164 thousand stars combined whose headline number is measured strictly against a straw man that does not exist in reality.