Search articles

Type a keyword to start searching
Back to selected workProject notes / 042024—2026

Financial content and visualization

Narrative Visualization

Turning financial scripts into narrated animations.

The system organizes financial video generation around layout generation, synchronized narration and animation, and local retries.

Whiteboard narrative video · a financial explainer generated by the system. Press play to watch.
My role

I am responsible for the overall project and technical design, and contributed to some early core development.

Where it is used

The system has been used in domestic and international businesses, within products and on external channels. Some tools and design rules are also used for merger animations, infographics, and charts.

01 / BACKGROUND

From generated visuals to video composition

A script needs page breaks, a visual for each part of the narration, and a schedule for revealing elements. If a generated image shifts the layout, cropping and animation are affected too. Combining pictures and audio leaves these problems unresolved.

We split the work into steps. The model interprets the script and plans visuals; code calculates coordinates, aligns timing, and checks results. When a step fails, the agent gets the reason and chooses whether to adjust parameters or regenerate that visual.

02 / SCRIPT AND MEDIA

Script, layout, and media synchronization

A page change can move region coordinates. A narration change can require new animation timing. The work has to account for both.

  1. 01

    Script

    Read the script, divide it into pages, and assign each part of the narration to a visual region.

  2. 02

    Layout

    A generated image contains several regions. Each crop needs complete content without neighboring artwork or positioning borders.

  3. 03

    Timing

    TTS sentence timestamps determine when text and graphics appear, so viewers see the relevant content as they hear it.

  4. 04

    Job state

    Keep completed steps and resume from the failed one. Retry and budget limits prevent repeated model calls from continuing indefinitely.

03 / AGENT RUNTIME

How the system works

The agent can call image, speech, and image-processing tools. Each call returns a result or an error that informs the next step.

  1. 01MODEL

    Plan

    Decompose the script, pages, regions, and narrative order

  2. 02MODEL

    Use tools

    Call image, TTS, segmentation, and parameter tools

  3. 03CODE

    Check results

    Read region, crop, and border validation results

  4. 04MODEL

    Choose a fix

    Use the error to choose a parameter change, regeneration, or a stop

  5. 05MODEL

    Retry a step

    Adjust crop parameters or regenerate the current visual

  6. 06CODE

    Validate

    Compose after region and image checks pass

  7. 07CODE

    Render

    Time animations to the narration and render the video

The model chooses the next tool. Code limits tool access, retries, and budget, and checks that images are usable before composition.

04 / MODELS AND CODE

Where we use models and where we use code

Coordinates and timing can be calculated directly. Choosing what to illustrate and how to respond to a failed generation needs more context.

VARIABLEGive to the model
  • Content understanding

    The model identifies emphasis, page boundaries, and narrative structure.

  • Visual expression

    The model creates elements that require style, metaphor, and variation.

  • Recovery decisions

    The agent chooses parameter changes, local regeneration, or termination.

INVARIANTGive to code
  • Structure & coordinates

    SVG and layout rules generate page frames, safe margins, and region references.

  • Media time

    Sentence timestamps determine element reveals and rendering.

  • Result checks

    Code checks region matches, non-empty crops, border residue, and retry limits.

05 / RECOVERY

Recovering from generation errors

These are a few of the failures the system handles. Crop problems can sometimes be fixed by changing parameters; when regeneration is needed, it is limited to the current visual.

F-01

Regions overlap

Small layout drift makes coordinate-based crops contain mixed regions.

How the system detects it
CV boundaries no longer match the planning reference
How the system continues
Rematch boundaries; regenerate only the current visual when recovery fails
F-02

The model rewrites borders

Color, weight, or shape drift leaves positioning marks inside the final asset.

How the system detects it
Crops still contain border-color residue
How the system continues
Adjust crop and border-removal parameters, then rerun the same checks
F-03

A crop is empty

A region is found, but useful content is insufficient for animation.

How the system detects it
Valid-pixel and content checks fail
How the system continues
Return a structured ToolResult so the agent can tune or redraw

06 / CONTEXT AND GUIDANCE

What the agent needs to know

Along with the script, the agent needs the current job state, details of any errors, and design guidance it can consult.

  1. 01

    Current task

    Plans, tool calls, and completed steps remain in the job state. The agent uses them to choose the next action; code checks permissions and stopping conditions.

  2. 02

    Check results

    Region matching, crop checks, and border detection report which parts of an image still need attention.

  3. 03

    Design guidance

    Narrative, layout, and chart guidance lives in skills that the agent can consult and we can revise.

07 / FEEDBACK

Reusing design experience

Useful layouts, visual choices, and failure cases are reviewed and added to skills and validation rules for later tasks. Within each run, code detects deviations and the agent repairs them locally before revalidation. Time and model budgets bound retries; at the limit, the task stops, falls back, or is referred to a person.

08 / OTHER USES

Other uses for the same tools

Script interpretation, media timing, and image checks also found uses beyond whiteboard videos.

NARRATIVEAGENT
RUNTIME
DSL · TOOLS · GATES · SKILLS
01

Whiteboard narratives

Script pages, region generation, sentence-level voice, and hand-drawn motion.

02

Merger animation

Break a merger explanation into animation actions and time them to narration.

03

Infographic generation

Use narrative and layout skills to guide infographic generation.

04

Chart generation

Let the agent use Standard Chart alongside chart design guidance.

  1. 01

    Responsible for the overall project and technical design, covering content generation and use across channels

  2. 02

    Defined the Narrative DSL, media timing, and agent tool interfaces

  3. 03

    Contributed to some early core modules involving image validation and failure recovery

AI-nativeAgent RuntimeNarrative DSLQuality GatesVisual Skills
Earlier research work