Search articles

Type a keyword to start searching
Back to selected work
Case 062026

Testing AI coding tools on visualization projects

Vis Agent Bench

Evaluate AI coding agents on shared visualization development tasks, measuring completion quality, time, and human intervention costs.

No public output image is available. A system architecture diagram follows below.

Assess both the delivered result and the human intervention required to produce it.
6 signals
measures recorded
3 cases
initial tasks
Vis Agent Bench workflow illustration, not measured results
Architecture diagram, not an output image

01 / Context

The problem

A fair comparison requires the same code, context, and permissions, with no answers left in the test project. Chart usability and layout also need inspection in the browser.

Implementation and progress

The platform prepares isolated projects, runs agents, and records their work. Reports include automated checks, browser captures, and anonymous review materials. Model samples are still limited, so findings need further testing.

02 / System

How the system works

  1. 01

    Prepare the project

    Builds minimal fixtures from real projects while excluding answers, prior patches, and unequal context.

  2. 02

    Record the run

    Record time, tool calls, token use, recovery, and human help, alongside the files produced by each run.

  3. 03

    Review the result

    Automated checks cover code and page behavior. Browser captures support anonymous human review of visuals and usability.

03 / Decisions

Design choices

01

Allow fixes and retries

Agents can diagnose, edit, and test as they would in regular development. The evaluation records that whole process, including time spent fixing mistakes.

02

Make reports traceable

Reports include versions, run records, screenshots, and reviews so readers can retrace the findings.

04 / My work

My role

  • 01Defined evaluation measures, selected tasks, and checked for answer leaks
  • 02Designed runner and evaluator interfaces and run record formats
  • 03Built the main features, organized model pilots, and analyzed results
AI EvaluationAgent HarnessPlaywrightTelemetryHuman Review