REAL TASK TEST
One app.
Two ways to run Codex.
We asked normal Codex and the LeanTask launcher to build the same expense tracker from identical starting files. Both used the same model, reasoning setting and acceptance tests.
THE OBSERVED RESULT
314,855 fewer tokens
in the LeanTask run.
68.05% lower total token usage in this pair
Input + output tokens. Cached input is already included.
01 / THE USE CASE
Build Pocket Ledger.
A deliberately created sample task with real model execution, rather than a customer production deployment. The brief was to build a complete local expense tracker using vanilla JavaScript, HTML and CSS, without external dependencies or services.
The same brief for both runs
- Add and delete expenses with descriptions, categories and valid dates.
- Keep amounts in integer cents and display USD totals.
- Search, filter by category and month, and show totals for the filtered list.
- Persist data locally, recover from malformed saved data, and render descriptions safely as text.
- Export filtered results as CSV and support mobile and desktop layouts.
The starter included 23 fixed acceptance tests. Neither run was allowed to change the test suite or package configuration.
02 / THE MEASUREMENTS
Reported usage, side by side.
These figures come from the completed Codex runs. They cover the whole measured task execution, including each run’s tool use, reasoning and recovery attempts.
| Metric | Normal Codex | LeanTask launcher |
|---|---|---|
| Input tokens, including cached | 435,704 | 136,659 |
| Cached input Subset of input | 396,800 | 111,488 |
| Uncached input | 38,904 | 25,171 |
| Output tokens | 26,973 | 11,163 |
| Total tokens Input + output | 462,677 | 147,822 |
| Elapsed time | 14m 17s | 5m 57s |
| Native shell commands | 13 | 6 |
| LeanTask MCP calls | 0 | 3 |
| Fixed acceptance tests | 23 / 23 passed | 23 / 23 passed |
Separate prompt-rewriter inference: 0 tokens. The model-based rewriter was not enabled. LeanTask’s added instructions and tool overhead are included in its run’s reported usage.
How the difference was calculated
462,677 − 147,822 = 314,855 tokens314,855 ÷ 462,677 × 100 = 68.05%
Cached input is not added a second time. Reasoning tokens are part of output and are not added again. Token counts are not equivalent to dollar costs or Codex subscription allowance.
ADDITIONAL ANALYSIS / ACCOUNTING ESTIMATE
What if the sandbox checks hadn’t failed?
We cannot recover the exact usage of a run that did not happen. We can isolate two recorded model requests that retried browser verification after the initial sandbox failure. Subtracting their usage gives a retry-excluded subtotal of 379,221 tokens for normal Codex.
| Calculation | Tokens |
|---|---|
| Normal Codex, measured total | 462,677 |
| Exclude local-file browser retry Request 9: 41,350 input + 236 output | −41,586 |
| Exclude browser cleanup-and-retry request Request 10: 41,678 input + 192 output | −41,870 |
| Normal Codex, retry-excluded subtotal 462,677 − 83,456 | 379,221 |
| LeanTask, original measured total | 147,822 |
The numerical gap between this adjusted subtotal and the measured LeanTask run is 231,399 tokens (61.02%). That is an accounting comparison, not a measured sandbox-free savings rate.
What this estimate keeps—and what it cannot tell us
It retains the original browser-verification attempt, the later successful simulated-DOM tests, the patch retry, final checks and answer. It also leaves later requests’ recorded context and cache usage unchanged, including any failure-related text they read.
A successful browser check would have produced different output and could have changed subsequent actions. This subtraction does not model that alternate path, remove every downstream effect of the failure, or establish that LeanTask caused the remaining difference. No sandbox-free runtime or dollar-cost estimate is inferred.
Source: per-request usage records in the normal run’s detailed session log. The sum of all 13 recorded requests reconciles exactly with its completed-run input, cached-input, output and total token counts. The two excluded requests contain 79,104 cached-input tokens, already included in their 83,456-token total. Selected records and the calculation are included in the downloadable measurement record.
03 / THE METHOD
Same task. Same starting point.
- Model and reasoning
gpt-6-astra, high reasoning, for both runs.- Starting files
- Identical Git commit and matching workspace fingerprints; the same 2,227-character prompt and fixed tests.
- Execution environment
- macOS, Codex CLI 0.153.4, Python 3.13.13 and Node.js 25.2.1. Both used fresh conversations and a workspace-write sandbox.
- Configuration
- Both skipped user configuration and set model and reasoning explicitly. Existing policy rules remained. This is a controlled CLI comparison, not a replay of all desktop settings.
- Run order
- LeanTask, then normal Codex; selected using a recorded random seed of 21. One run per approach, without repetitions.
- LeanTask setup
- Version 0.3.0b6, run directly from cloned source. Its launcher attached five task-scoped MCP tools. No global companion-plugin installation was involved.
Both runs had a 15-minute limit and were allowed to implement and verify the app independently. The same acceptance tests were also run separately, and both finished apps received independent browser checks.
04 / WHAT ACTUALLY HAPPENED
A smaller total is only part of the story.
Tools were used, but small outputs grew.
The launcher added 764 characters of task guidance. Codex made three LeanTask file reads and used one command capture for the test run.
The files and log were small. Their returned character counts increased after metadata was included; this run demonstrated neither repeated-read reuse nor a benefit from output compression.
Retries and extra verification added work.
The normal run encountered one patch-verification failure and retried. It also tried additional browser verification: the sandbox rejected a temporary HTTP server, and local-file Chrome attempts failed.
It subsequently completed simulated-DOM checks. Those different verification choices and recovery attempts are included in the measured total.
| LeanTask operation | Source characters | Returned characters |
|---|---|---|
| Three file reads | 3,336 | 4,059 |
| One test-log capture | 1,170 | 1,730 |
Character diagnostics are not an attribution of model tokens. Other context, tool schemas and execution choices also affect the run total.
05 / QUALITY CHECKS
Both apps met the checked requirements.
Each passed all 23 fixed tests and JavaScript syntax checks. The original package configuration and test files were unchanged. Both runs also reported passing simulated-DOM checks.
Passing these checks is evidence of the tested behavior; it is not a complete audit or proof that the implementations have identical quality.
Independent browser checks
- Empty state, expense entry, totals and newest-date-first ordering.
- Live search, combined filters and individual deletion.
- Persistence across reload and visible invalid-input errors.
- HTML-like descriptions displayed as literal text.
- 375px and desktop layouts; no horizontal overflow at 375px.
Verification limits: CSV controls were exercised and CSV function tests passed, but downloaded files were not independently verified. Malformed-storage handling was reviewed in source and exercised in the runs’ simulated DOMs, not independently in the real browser.
06 / WHAT THIS RESULT SUPPORTS
A useful first observation.
More evidence is still needed.
In this pair, the LeanTask run completed the checked app-building task with fewer reported tokens and less elapsed time. That is the measured result.
One pair cannot separate the effects of the launcher from model variability, prompt caching, retries or verification choices. It does not establish a typical savings rate, a smaller bill, or a benefit from the companion plugin, prompt rewriting, remote datasets or multi-turn memory.
A stronger evaluation would repeat several tasks, counterbalance the run order, use consistent verification requirements, and report every attempt—including failures and optimizer overhead. Read our benchmark protocol.
FOLLOW-UP WORK / EXPECTED USE CASE
Recurring work can offer more opportunities to save.
Follow-up fixes, repeated checks and ongoing development on the same task give LeanTask more opportunities to reuse useful context. Focused reads, saved task notes and retrievable logs can help avoid sending the same large files or outputs again. The more redundant context these tools avoid, the greater the opportunity for token savings.
This is an expected use case, not a measured finding of this single app-building test. Ordinary Codex also supports continued conversations and efficient reads. Actual savings depend on what is reused and how much context is avoided; growing history and tool overhead can offset the benefit. Repeated work does not automatically guarantee greater savings.
07 / INSPECT THE EVIDENCE
The numbers and the task brief.
Download the curated measurement record, the exact task prompt, or the original starter with its fixed tests. The measurement record includes the reported completion usage, run conditions and verification limits; it does not include full execution transcripts.
The starter contains no generated solution. Run the same prompt against two identical checkouts to conduct a new comparison; its results may differ. Usage is reported through Codex’s JSON event stream.
Accounting scope and rejected setup attempts
The comparison includes the two completed task runs. It excludes the coordinating conversation, fixture creation, independent evaluation and report preparation.
Two earlier setup attempts used the stale configured gpt-5.4 model and were rejected before code generation. They returned no usage data. Their usage is unknown, not treated as zero; they are excluded from the successful-model pair and prevent treating this table as a total account-spend calculation.