Methodology
How the agent token test was run
The homepage chart is this test. A model is asked for the same 42 charts in five libraries, with a capped copy of that library's docs in the prompt. The number is the median output tokens until the checker accepts a drawing.
The run did not record a ChartKit version. 0.5.0 is the version in the ChartKit package on 27 October 2026, the day these medians were published. The package version is still 0.5.0, and the library has changed since that day, so today's package is not this run. One line of llms.txt, the maps install note, has changed since these figures were published. The file is still the same length. None of these tasks is a map. The cap is 6,376 bytes.
Caveats
Read these before treating the chart as a ranking of libraries.
We wrote the library and the test
ChartKit's authors wrote the tasks, the prompts, the checker, and the length cap. Nobody outside the project scored this run.
The task set is ours, and it is small
42 tasks, 3 each of 14 chart types, each with a few rows we wrote for the harness. This is not a sample of production charts, and it is not a suite someone else published.
Five libraries, chosen as the harness set
The scoreboard is ChartKit, Observable Plot, Chart.js, Recharts, and ECharts. uPlot, Nivo, and the commercial libraries in the paint benchmarks are not in this test. The bench pinned Observable Plot 0.6.17, Chart.js 4.5.1, Recharts 3.10.1, ECharts 6.1.0 on the day the figures were published. The result file does not record which versions were installed when the models ran.
The docs are not the same kind of document
ChartKit's prompt includes the whole of llms.txt, written as instructions for an agent. The other libraries get the opening of a getting-started guide, cut so it is no longer than that file. Cutting them to our length was our choice. We did not measure what any model had already seen in training. A model that already knows Chart.js is not in the same position as a model reading ChartKit's spec for the first time, and this test does not separate those cases.
A token here is the provider's count
There is no shared tokenizer. Anthropic reports output tokens. OpenAI and xAI chat responses report completion tokens. Those units are not the same, and the chart still puts them on one axis. A .5 in the table is the average of the two middle values when the number of working charts is even. It is not a fraction of a token from the provider.
Tokens are not quality, speed, or cost
The table is output tokens on charts the checker accepted. It says nothing about how the picture looks to a person, how fast the library draws, or how large the bundle is. The harness has placeholder prices, and this page does not publish a dollar figure. Input tokens are stored by the harness and are not in the table.
Correct means the checker accepted it
Nobody graded the charts by eye, and there is no comparison against a reference picture. A hand-written drawing fails. A short wrong answer fails and drops out of the median, so a low number is only the cost of the answers that passed.
Unfinished tasks leave the median
A task that never passes is listed and left out. The median is not recorded as infinity. A median over a handful of successes can look tidy. The published notes say which rows that applies to. The other models do not publish a working-chart count, so their cells do not show how many tasks they finished.
Declining a chart is not drawing one
When a library has no such chart, an honest decline counts in the pass rate and is left out of the token median. Plot, Chart.js, and Recharts are expected to decline several of the 14 types. That is the rule of the checker, not a result.
One pass per model, and no measured variance
Temperature was 0. Each model went through the task list once. There is at most one retry for a fixable error. Temperature 0 does not promise the provider will return the same text twice, and this test did not repeat a model, so there is no measured spread from randomness. The spread in the chart is the spread across models.
These model ids, on this day
The figures are the registry ids below, published 27 October 2026. A provider can change a model without changing the id. ChartKit in the test is the 0.5.0 tree from that day.
What this page does not contain
The nine-model run without docs was not published. The harness can run that condition. A separate one-model scoreboard, with docs and without, is on the benchmarks page. Those figures are not this table. This test also does not measure a multi-step agent, tool use, or charts outside these 42 tasks.
Results
With docs. Each cell is the median output tokens for that model and library, counting only charts the checker accepted. Across the 9 models, the median of ChartKit's medians is 94 tokens. The next library is Plot at 332. That ratio is 3.5×. By model, ChartKit uses 2.1–5.3× fewer output tokens than whichever library was next. ChartKit's published median model calls on a working chart are 1.0. repair() does not add a call.
| Model | ChartKit | Plot | Chart.js | Recharts | ECharts | CK vs next |
|---|---|---|---|---|---|---|
| Haiku 5.5 | 133 | 711 | 1,193 | 1,285 | 882 | 5.3× |
| Sonnet 5.5 | 124 | 333 | 887 | 1,499 | 544 | 2.7× |
| Opus 5.5 | 127 | 356 | 901 | 1,713 | 469 | 2.8× |
| Fable 5.1 | 115 | 300 | 656 | 1,468 | 379 | 2.6× |
| GPT-6 Luna | 94 | 464 | 820 | 874 | 675 | 4.9× |
| GPT-6.1 Terra | 82.5 | 332 | 808 | 1,594 | 488 | 4.0× |
| GPT-6.1 Sol | 86.5 | 244 | 491 | 1,273 | 492 | 2.8× |
| GPT-6 Astra | 88 | 266 | 492 | 1,358 | 536 | 3.0× |
| Grok 4.7 | 78.5 | 166 | 444.5 | 3651 | 284 | 2.1× |
Grok 4.7. ChartKit 42/42 working, first try 100%. Plot 26 working (15 declined, 1 DNF spike-alerts). ECharts 36 working (5 declined, 1 DNF ring-tasks). Chart.js 24 working (18 declined). 1Recharts is 5 working and 19 DNF. That median is unreliable. Most failures were browser render.
The homepage bars are the median of these 9 rows. Did-not-finish runs are excluded from every cell. Pending models, if a later run adds one, stay out of the median and show as pending in this table.
What we tested, and why
The question was how many output tokens a model spends to get a chart the checker will accept, when the answer is a ChartKit spec, and when the answer is code for another library. The homepage uses that comparison. The agents page is the product page for the loop. This page is the test.
Each task is a short request plus a small data table: a line of monthly revenue, a bar of signups, a gauge at 72, and so on. The model sees the request and the data. The checker already knows the chart type, the fields, and the numbers.
Models and tasks
9 models finished. Temperature 0. Maximum output tokens per response 4,096. One pass through the task list per model. The registry id is the id the client sent.
| Name | Registry id | Provider |
|---|---|---|
| Claude Haiku 5.5 | claude-haiku-5-5 | Anthropic |
| Claude Sonnet 5.5 | claude-sonnet-5-5 | Anthropic |
| Claude Opus 5.5 | claude-opus-5-5 | Anthropic |
| Claude Fable 5.1 | claude-fable-5-1 | Anthropic |
| GPT-6 Luna | gpt-6-luna | OpenAI |
| GPT-6.1 Terra | gpt-6.1-terra | OpenAI |
| GPT-6.1 Sol | gpt-6.1-sol | OpenAI |
| GPT-6 Astra | gpt-6-astra | OpenAI |
| Grok 4.7 | grok-4.7 | xAI |
42 tasks, 3 of each type: line, bar, donut, stacked-area, scatter, combo, heatmap, gauge, kpi, sparkline, mini-area, progress-ring, spike, candlestick. The libraries on the scoreboard are ChartKit, Plot, Chart.js, Recharts, ECharts.
With docs and without
Without docs, every library gets one short paragraph: how to produce a drawing, and where the fields go. The paragraphs do the same job at a similar length. ChartKit's paragraph does not add channel shapes the others lack.
With docs, that paragraph stays, and the library's own text is placed in front of it. ChartKit's text is llms.txt, 6,376 bytes, and that length is the cap. The others are cut to it, on a character boundary, and trimmed back to a line break when that still leaves most of the excerpt. The published table is this with-docs condition. The nine-model without-docs condition was not published.
- ChartKit. 6,376 bytes. The full llms.txt. That length is the cap. The other libraries are cut to it.
- Plot. 6,374 bytes. Observable Plot getting started, from the Plot 0.6.17 docs, retrieved 27 October 2026. Plot does not publish an llms.txt.
- Chart.js. 6,327 bytes. The Chart.js installation page plus the start of the usage guide. Chart.js does not publish an llms.txt.
- Recharts. 6,329 bytes. Recharts getting started plus the start of quickstart. Recharts publishes an llms.txt that is an index of links, not a spec. The index blurb at the top of each page was removed.
- ECharts. 6,077 bytes. The Apache ECharts Get Started page plus Import ECharts. ECharts publishes an llms.txt index, and the handbook link on it is Get Started.
The system line on every call is "You produce only the requested chart artifact. No prose." ChartKit is asked for one JSON object. The other libraries are asked for a module that imports that library and returns the drawing it produced.
How the tokens were counted
The harness does not tokenize the text itself. It stores the count on the provider's response. When the response has output_tokens, that field is used. Otherwise completion_tokens is used. Anthropic's Messages API reports output_tokens. OpenAI and xAI chat responses report completion_tokens. A gpt-6 id that Chat Completions rejects is sent to OpenAI's Responses API, which reports output_tokens. The published rows do not say which of those two OpenAI paths answered.
For a chart the checker accepted, the cell adds those output counts across the model calls that led there: the first answer, and the retry when there was one. repair() adds no tokens. An honest decline is not a working chart, so it stays out of the median. A task that never passes stays out too. The median of an odd count is the middle value. The median of an even count is the average of the two middle values.
How a chart counted as correct
A ChartKit answer has to validate, render as the chart type the task asked for, use the expected fields, and carry the task's numbers. The drawing is checked too: bars sit inside the plot, a line chart has a line, a donut has its slices, a scatter has its points, a gauge or progress ring has an arc, a KPI has the value. Gauge, KPI, and progress ring also have to put the number on the metric.
A code answer has to import the library and call it. The type is read from the drawing, not from a note the model attaches. The numbers are read from the drawing the same way. Recharts has to produce an SVG with data marks, mounted in a browser. Chart.js has to paint pixels on a canvas. A canvas tag with a blank canvas fails. Observable Plot has to return the SVG from Plot.plot. ECharts can render to SVG on the server, and that SVG has to contain the marks. Code that touches the document runs in a browser instead.
Where the library has no chart of that type, the prompt tells the model to decline. A substitute chart fails. The declines the checker expects:
- Plot. donut, gauge, kpi, progress-ring, candlestick (15 of 42 tasks).
- Chart.js. heatmap, gauge, kpi, progress-ring, spike, candlestick (18 of 42 tasks).
- Recharts. heatmap, gauge, kpi, progress-ring, spike, candlestick (18 of 42 tasks).
- ECharts. kpi, progress-ring (6 of 42 tasks).
ChartKit is asked for all 42.
Retries
ChartKit repair() runs locally on a failed answer. If the repaired spec passes, the model is not called again. That repair adds no call and no tokens.
Otherwise the harness sends the failure back once, and only when the failure was validation, rendering, compiling, or not actually using the library. The retry includes the error text and the previous answer. A wrong chart type, or the wrong numbers, is not sent back.
A request the network rejects, a rate limit, or a server error is retried up to five more times, with a pause that doubles from two seconds. Those are not extra attempts at the task. A failed request adds no tokens. The response that succeeds is counted once.
Each of the 9 models was run once. The table is that pass. There is no second seed.
ChartKit version and the day of the run
The run did not record a ChartKit version. 0.5.0 is the version in the ChartKit package on 27 October 2026, the day these medians were published. Eight of the models were published that day, and Grok 4.7 was added later the same day. The raw per-model result files are not published with the site, so there is no separate clock time from the model calls. The harness scripts that score a chart are unchanged since that day. The ChartKit package they call has moved since, under the same version number, and these cells were not re-scored.
Agents is the product page. Benchmarks keeps the earlier one-model scoreboard.