Runway API Hackathon
Director's Canvas
Agent-directed video production as a generative interface. Type a one-line brief — watch a LangGraph agent decompose it into shots, generate Runway reference stills, animate each into a clip, and stitch a final MP4. The agent's output is the interface.
Pipeline
How it works
A single brief flows through four stages. Every Runway call returns a state mutation — that's why the canvas paints live.
1. Brief
You type a one-line brief. The agent decomposes it into 3–6 shots with cinematic prompts — subject, action, lighting, mood.
2. Reference stills
Shot 0 calls Runway gen4_image. Shots 1+ call gen4_image_turbo with shot 0's image as a character anchor — 2–4× cheaper, <10 s each.
3. Animation
Each reference still is animated via Runway gen4.5 (image→video). The shot's own reference is the first frame, so visual style carries directly into motion.
4. Export
FFmpeg concatenates all clips into a single MP4, served directly from the frontend. One button. No NLE required.
Runway API
What makes the Runway usage non-trivial
Standard integrations call image_to_video once and return a URL. Director's Canvas does something structurally different.
Context-aware model selection
The agent picks the right model automatically. gen4_image for shot 0 (no prior refs available). gen4_image_turbofor shots 1+ (refs available, 2–4× cheaper, <10 s). gen4.5 for all image→video animation. No user decision required.
Chained state across N calls
Shot 0 runs synchronously first. Its reference URL is promoted to storyboard.style_ref_url and passed as character1 to every subsequent gen4_image_turbo call. Shots 1+ run in parallel (bounded to 4 concurrent) with that anchor in place.
Agent-directed iteration
“Regenerate shot 3 — make it more dramatic.” The agent rewrites the prompt, re-calls Runway, and patches only that shot's state. The rest of the storyboard is untouched.
Full pipeline to a deliverable
The agent doesn't stop at a URL. It calls FFmpeg to concat all clips into a single MP4 and serves it directly from the frontend. The output is something you can actually share.
Cross-shot consistency
How character consistency works
The astronaut in shot 4 looks like the astronaut in shot 1 — not because the user re-uploaded anything.
gen4_image_turbo accepts up to 3 referenceImages. Director's Canvas exploits this to keep characters and visual style coherent across shots:
- character1— shot 0's reference still (the primary anchor, always first)
- style1— the immediately preceding shot's reference (local continuity)
- style2 — the shot two positions back (extra reinforcement for longer storyboards)
The prompt can address the anchor explicitly: @character1 walks through the airlock. The Consistent pill in the canvas header lights up once the anchor is set.
Access
BYOK + budget guard
Bring Your Own Key
Click Add Keyin the canvas header to enter your Runway API key. It's stored in localStorage, sent as X-Runway-Api-Key on every request, and injected into the agent via LangGraph configurable. It is never logged. When your key is active, charges go to your Runway account and the per-thread budget check is skipped.
Per-thread budget guard
When using the shared server key, the BFF injects runway_calls_remaining into every request. The agent checks this before each Runway call and raises BudgetExceededError when it hits 0 (default: 20 calls ≈ 10 shots). The error surfaces in chat with a clear message and a prompt to add your own key.
MOCK mode
No RUNWAY_API_KEY set? The full pipeline runs with deterministic placeholder media — same canvas, same status pills, same export flow. Anyone can try it without burning credits.
Technology
Stack
| Layer | Technology |
|---|---|
| Agent | LangGraph Deep Agents + Gemini 3.1 Flash-Lite |
| Video | Runway Gen-4 Image / Gen-4 Image Turbo / Gen-4.5 |
| Transport | AG-UI + CopilotKit Intelligence (durable threads) |
| UI | Next.js + React + useFrontendTool (A2UI pattern) |
| BFF | Hono — CopilotKit runtime + BYOK injection + budget guard |
| Export | FFmpeg concat (LIVE) / placeholder URL (MOCK) |
| MCP | mcp-use server — exposes the director to Claude / ChatGPT |
Get running
Quickstart
- 1
Init CopilotKit Intelligence
npx @copilotkit/cli@latest init
Select Intelligence when prompted. This sets up the Postgres-backed thread store.
- 2
Add your keys
cp .env.example .env && cp .env apps/agent/.env
Set GEMINI_API_KEY in both files. Optionally set RUNWAY_API_KEY — without it the director runs in MOCK mode (same UI, deterministic placeholder media, no credits burned).
- 3
Install + run
npm install && npm run dev
Boots the Docker infra (Postgres + Redis + Intelligence), then UI + BFF + agent. Open localhost:3000 → /director.
No Runway key? That's fine — MOCK mode runs the full pipeline with placeholder media. Add your key via the Add Key button in the canvas header at any time.
Reference