Financial content and visualization
Turning financial scripts into narrated animations.
The visuals need to follow the narration, and a failed image should be repairable. Much of our work went into layout, timing, and retrying individual steps.
We led the project and technical design, and built core parts of the Narrative DSL, agent tools, and media timing.
The system has been used in domestic and international businesses, within products and on external channels. Some tools and design rules are also used for merger animations, infographics, and charts.
01 / BACKGROUND
A script needs page breaks, a visual for each part of the narration, and a schedule for revealing elements. If a generated image shifts the layout, cropping and animation are affected too. Combining pictures and audio leaves these problems unresolved.
We split the work into steps. The model interprets the script and plans visuals; code calculates coordinates, aligns timing, and checks results. When a step fails, the agent gets the reason and chooses whether to adjust parameters or regenerate that visual.
02 / SCRIPT AND MEDIA
A page change can move region coordinates. A narration change can require new animation timing. The work has to account for both.
Read the script, divide it into pages, and assign each part of the narration to a visual region.
A generated image contains several regions. Each crop needs complete content without neighboring artwork or positioning borders.
TTS sentence timestamps determine when text and graphics appear, so viewers see the relevant content as they hear it.
Keep completed steps and resume from the failed one. Retry and budget limits prevent repeated model calls from continuing indefinitely.
03 / AGENT RUNTIME
The agent can call image, speech, and image-processing tools. Each call returns a result or an error that informs the next step.
Decompose the script, pages, regions, and narrative order
Call image, TTS, segmentation, and parameter tools
Read region, crop, and border validation results
Use the error to choose a parameter change, regeneration, or a stop
Adjust crop parameters or regenerate the current visual
Compose after region and image checks pass
Time animations to the narration and render the video
The model chooses the next tool. Code limits tool access, retries, and budget, and checks that images are usable before composition.
04 / MODELS AND CODE
Coordinates and timing can be calculated directly. Choosing what to illustrate and how to respond to a failed generation needs more context.
The model identifies emphasis, page boundaries, and narrative structure.
The model creates elements that require style, metaphor, and variation.
The agent chooses parameter changes, local regeneration, or termination.
SVG and layout rules generate page frames, safe margins, and region references.
Sentence timestamps determine element reveals and rendering.
Code checks region matches, non-empty crops, border residue, and retry limits.
05 / RECOVERY
These are a few of the failures the system handles. Crop problems can sometimes be fixed by changing parameters; when regeneration is needed, it is limited to the current visual.
Small layout drift makes coordinate-based crops contain mixed regions.
Color, weight, or shape drift leaves positioning marks inside the final asset.
A region is found, but useful content is insufficient for animation.
06 / CONTEXT AND GUIDANCE
Along with the script, the agent needs the current job state, details of any errors, and design guidance it can consult.
Plans, tool calls, and completed steps remain in the job state. The agent uses them to choose the next action; code checks permissions and stopping conditions.
Region matching, crop checks, and border detection report which parts of an image still need attention.
Narrative, layout, and chart guidance lives in skills that the agent can consult and we can revise.
07 / FEEDBACK
More runs do not automatically produce better results. Someone needs to collect the layouts, visual choices, and failures worth keeping, review them, and add them to the skills and checks that later runs use. Within a run, the checks point out a deviation, the agent repairs it locally, and the same check runs again; retries cost time and model budget, so at the limit the task stops, falls back, or goes to a person.
08 / OTHER USES
Script interpretation, media timing, and image checks also found uses beyond whiteboard videos.
Script pages, region generation, sentence-level voice, and hand-drawn motion.
Break a merger explanation into animation actions and time them to narration.
Use narrative and layout skills to guide infographic generation.
Let the agent use Standard Chart alongside chart design guidance.
09 / MY WORK
We led the project and technical design, and built core parts of the Narrative DSL, agent tools, and media timing.
An earlier team research work shortlisted at PacificVis 2024. This video is not a demonstration of the current production system.
Designed the approach to step-by-step generation and local retries.
Designed script planning, the Narrative DSL, media timing, and intermediate file structures.
Added image generation, CV segmentation, image checks, and parameter adjustment to the agent tools.
Built core modules and adapted the tools for whiteboard videos, animation, infographics, and charts.
