Reelly — Auto-Edit Pipeline Architecture

01

The defect, in three lines of shipping code

These three lines are the entire selection logic in the app today. Read together, they describe a product that picks the worst moment available and then throws away more than half the footage.

Location	Code	Consequence
ReelStudio:83	CaptureDate.ordered(clipURLs)	Order is capture time. Nothing else is considered.
ReelRenderer:74	clipAssets[clipCursor % count]	First 46 clips fill 46 slots positionally. Clips 47–100 are never looked at.
ReelRenderer:87	CMTimeRange(start: .zero, …)	Every cut is the head of a clip — the framing settling, the blurry pan.

Why this matters more than it looks

The benchmarked selection layer already exists in ios/AutoClipping/Indexing/ — seven files, scored 3/3 on short clips and 4/4 on long ones against hand-labelled ground truth. Nothing outside that folder calls it. Every prior judgement about whether Reelly works was made about a build that never ran it.

02

Four stages

Clips enter as picked camera-roll assets and leave as an MP4. Two stages already exist; two are new. The split between them is the whole design: the director decides semantics — which beat, which slot, what story — and a pure resolver decides arithmetic. Models are unreliable across dozens of simultaneous numeric constraints, and that is exactly where a wrong number fails an export.

01

Catalogue

built · unwired

Sixteen frames are sampled across the clip, labelled with their timestamps, and sent in one call to a vision model. It returns a screenplay: three to eight chronological beats, each with a time range, a present-tense action line, a camera note, and any dialogue. Plus clip-level facets — setting, shot type, subjects, mood.

It deliberately does not nominate a best moment. It documents what is there, which means the beats arrive as candidate windows already.

in clip file → out ClipIndex · ~$0.0034 and ~3.3s per clip

02

Direct

new · one call

One call that sees every beat in the footage at once — roughly 500 of them — alongside each slot's exact duration. It decides the story, names a beat for each slot, and may take several beats from one clip or reorder time to serve the edit.

It does no arithmetic. It names a beat and says why; timing is somebody else's job.

in ~100 loglines + ~500 beats + 46 slot durations → out story, arc, placements

03

Resolve

new · pure

Turns a named beat into an exact in-point, checks it against the clip's real duration, and validates every invariant the director could have broken — a missing slot, a duplicated one, an invented clip id, a beat index out of range.

Deterministic and unit-tested. This is where the numbers are safe.

in director response → out [ClipPlacement]

04

Render

built one arithmetic fix

Walks the template's segments and pops a placement for each clip-backed one. It chooses nothing — both existing modulo loops are deleted. Effects, the music bed, timed SFX and the watermark pass are all untouched.

The fix: how much media to take must be measured from the in-point forward, not from the clip's total length.

in [ClipPlacement] → out MP4

03

One clip's journey

Concretely: subway_train arriving station.MP4, one of the thirteen real Japan trip clips in the repo. Forty-seven seconds long. It opens on a static platform and stays there. At around twenty-six seconds a train pulls in — the only thing in the clip anyone would want to watch.

Today this clip contributes a 2.35-second cut of an empty platform, because 2.35 seconds is what slot 7 asks for and the read starts at zero.

Note what a director can do here that a per-clip chooser cannot. This clip holds two usable moments — the train arriving, and the doors opening onto a departing crowd. Both can appear in the finished reel. Allocating one clip per slot would have thrown one of them away by construction.

0:000:100:200:300:400:47

what ships today — head trim, static platform what the pipeline picks — train arriving

What is measured and what is illustrative

The 47-second duration and the ~0:26 golden moment are real: hand-labelled ground truth from the model bake-off in .claude/experiments/golden-moment/. The beat boundaries drawn above are representative of what a screenplay looks like, not a recorded index run for this clip. Beat timing also carries real error — frames are sampled with a ±0.3s tolerance, and the recorded timestamp is the one that was requested, not the one the decoder returned.

So the journey, end to end, for this one clip:

Step	What happens to this clip	Result
pick	Selected in the media grid with ~99 others; resolved from Photos to a readable file	URL + PHAsset id
catalogue	16 frames sampled across 47s (~3s apart), one vision call, screenplay returned	5 beats + facets
validate	Beats checked against the real 47s duration; degenerate or out-of-range beats dropped	5 beats kept
direct	All 5 beats join a pool of ~500. The director takes "train arrives" for slot 7 — and separately takes "doors, crowd off" for slot 9	2 slots, 1 clip
resolve	Beat 4's range becomes an exact in-point; runway checked against the real 47s; every invariant validated	inPoint ≈ 26.0s
render	`take = min(2.35, 47.0 − 26.0)` → 2.35s read from 26.0s, zoom effect applied, music over	cut 7 of 47

04

What crosses each boundary

Four shapes. The one that carries the architecture is the last: a single ClipPlacement per clip-backed segment, produced once and consumed by every render call site.

ClipIndex — what the model returns

slug · scene heading, INT./EXT. style
logline · the whole clip in one sentence
screenplay · 3–8 beats: start, end, action, camera, dialogue
setting · shotType · subjects · enumerated facets
peopleCount · hasSpeech · mood · tags

exists · persisted on ClipAsset · schema v3

ClipFacts — what the director is shown

id · captureDate · duration
beats · index, range, action, camera, dialogue
facets · slug, logline, setting, shotType, subjects

new · a value snapshot. No SwiftData models and no file I/O cross this line, which is what keeps the resolver pure and testable. Capture dates travel as data, so the director can spend chronology knowingly.

Director response — the edit

story · one sentence, the through-line
arc · opener / build / turn / payoff / closer, mapped to slots
placements · per slot: clipID, beatIndex, why, breaksChronology

new · the director names beats and gives reasons. No timings, no arithmetic, no slot maths.

ClipPlacement — the plan

clipID · durable identity, survives cache eviction
segmentIndex · binds to a slot, not an array position
inPoint · where the read starts

unchanged · and it already supports one clip in several slots — same clipID, different segmentIndex and inPoint. Carries no URL on purpose: a cache path goes stale, so files are materialized late.

The structural win

Three places currently decide clip order and slot mapping independently: the studio, the renderer, and the timeline editor. With placements as the unit, the edit is decided once and all three consume the answer. Preview-matches-export stops being a rule someone has to remember and becomes a property of the types.

A consequence worth catching early

Because one clip can now legitimately occupy several slots at different moments, keying the editor's thumbnails by segmentIndex rather than by file URL stops being merely correct and becomes load-bearing. Keyed by URL, one clip in three slots would show the same frame three times.

05

The director

An earlier draft of this section described a scoring heuristic and a greedy planner. Both are gone, for three reasons that are worth stating plainly because they were the plan's weakest point.

The problem	Why it was fatal
A ruleset cannot construct a story	The index has no focus, shake, exposure or composition signal, so the most a heuristic could honestly claim was defect avoidance. That is a floor, not an edit
No stage ever saw the whole trip	Catalogue is per-clip, scoring per-beat, greedy assignment local. "What story does this footage tell?" was a question the architecture had nowhere to ask
Clip-granular selection caps creativity	One clip per slot means a clip holding two good moments contributes one. The real search space is ~500 beats, not ~100 clips — picking at clip level discards most of it before anything decides

So instead: one call, everything visible at once. Every clip's logline and facets, every beat with its time range and action line, every slot with its exact duration. The director decides what story the footage tells and which moment fills which cut.

Slot duration is given to it as a pacing signal, not a constraint to solve: a payoff belongs on the 7.32-second hold, not on a 0.48-second flash.

What it must return

01

A story

One sentence naming the reel's through-line. Not decoration — it is the only artifact that makes a disappointing edit arguable rather than merely re-runnable.

02

An arc

Opener, build, turn, payoff, closer, each mapped to slot ranges. Declaring structure is what separates a deliberate choice from a scrambled one.

03

A beat per slot, with a reason

Clip id, beat index, and one line on why this beat, here. Several slots may name the same clip. No timings — naming a beat is the whole job.

04

Every chronology break, flagged

The director may reorder time to serve the edit, but must say where it did. Chronology is information it may spend, not a rule it must obey.

What this costs

Roughly 30k tokens in and 4k out — a cent or two per reel, against the ~$0.34 already spent indexing. The prompt is separate from the indexing prompt, so iterating on the director never invalidates an index. That means the edit can be re-tuned dozens of times against one indexed trip for the price of a coffee.

06

Where the numbers stay safe

Handing the edit to a model gives up determinism, and it would be dishonest to pretend otherwise: the same footage can now produce different reels. What it does not give up is correctness, because the director never touches a number.

Determinism moves rather than disappears. It used to live in the planner; it now lives in the resolver, which is pure, and in the validator, which checks every invariant the director could have broken. Those are the things under unit test. The director gets an eval instead.

What could go wrong	Caught by	What the user gets
Fewer or more than 46 placements	validator	repair pass re-asks for the missing slots
Two placements claim one slot	validator	first kept, second re-asked
An invented clip id, or a beat index out of range	validator	placement rejected, slot repaired
The reel collapses onto six clips	beats-per-clip cap, tunable	dump shows the distribution
A beat too short for its slot	resolver	slides earlier in the beat, else stretches
No valid in-point at all	resolver	that one slot falls back to head-trim
The call fails outright	client	whole reel falls back to today's behaviour — still exports
The edit is simply incoherent	an eval, not a test	story, arc and reasons make it diagnosable

The honest trade

Every failure above degrades to something that still exports. The worst case is a reel identical to the baseline — a legible result rather than a crash. What genuinely gets harder is diagnosis: a heuristic's bad pick has a traceable rule, a director's does not. The required story, arc and per-placement reasons exist precisely to buy that back.

07

The target, drawn to scale

Every segment of tokyo-vlog.json at its real duration — extracted from an actual CapCut project, 47 segments across 111.4 seconds. Width is proportional to time, so this is a scale drawing of the thing being filled.

0:000:200:401:001:201:401:51

under 1s — 9 slots, selection barely shows 1–4s — 29 slots 4s and over — 8 slots, where selection earns its keep black block — consumes no clip

Two things fall out of the drawing. The opening five cuts are all around half a second, where "the golden moment" collapses into "any frame that isn't blurry" — no choice is visible there. And the eight long slots on the right, including that closing 7.3-second hold, are simultaneously where selection is most visible and where runway is scarcest. Those eight are the ones to judge the result on.

Known caveat, accepted deliberately

This template is 16:9 and 111 seconds — a YouTube-shaped vlog, not a vertical reel. The decision was to run the evaluation against the real edit anyway, so if the verdict is "wouldn't post it", the reason has to be recorded: selection quality, or format.

08

What review changed

Twenty-four findings, all resolved. Eleven from a four-section pass, thirteen from an independent second model. Three of them would have cost real time.

Was	Is	Cost avoided
Wire selection into `ReelStudio.generate()`	Wire into `ImportSession.prepare()`	The real user flow never calls `generate()` — it is the old home-screen path. The design doc named the wrong function too, so the error was inherited
Change `.zero` to the in-point	Also fix how much media is taken	`take` was measured from the clip's total length. Any in-point could read past the end and fail the whole export — and a drag in the editor triggers it, so it is ordinary, not rare
Score beats on speech and dialogue	Penalise them	Source audio is discarded. The rule selected silent talking heads
Build persistence, cache, progress UI, thumbnails, then evaluate	Prove the ranking first with the existing Python harness	About a week of infrastructure spent before knowing whether ranking beats head-trim
Four new types across nine files	Two new types across five	An on-device motion analyser deferred — it buys ±0.4s on slots whose median is 2.2s
Rank beats with a Swift heuristic, then assign greedily, one clip per slot	One director call over all ~500 beats; a pure resolver does the timing	A ruleset that could only avoid defects, an architecture with nowhere to ask what story the footage tells, and a hard cap of one moment per clip

09

Build order

The sequence is arranged so the cheapest possible answer to "does this work" comes before any product infrastructure.

Phase 0

Baseline · no code

Export one reel from ~100 real trip clips through today's build. Keep the file. Roughly thirty minutes, and without it "it's better" cannot be falsified.

Phase 1

Harness proof

Index one trip with the existing Python harness. Add a test target. Write the director prompt and run it over that JSON. Build the resolver and validator as pure tested functions. Fix the renderer arithmetic. Render the directed reel and compare against the baseline. Re-tune the prompt as often as you like — a cent a run, and the index never has to be rebuilt.

Gate

Watch both · post one, or don't

Binary and subjective by design. Deciding not to post is equally informative — but record which reason it was: the edit, the beat quality, or the 16:9 format.

Phase 2

Wire it in · only if the gate passes

Clip identity and byte cache, awaitable indexing with progress and cancellation, placements through the timeline editor, per-segment thumbnails at the in-point.

Honest framing for phase 1

The comparison reel comes out of a partly offline pipeline — indexing runs in Python, not in the app. It proves the edit, not the shipped path. Phase 2 is what makes it a product.

What the gate now tests, and why that is a bigger bet

Before the director, the gate asked "does picking better moments beat head-trim". It now asks "can a director construct a story". That is the more valuable question and the harder one to diagnose, because a disappointing reel has four candidate causes instead of one: the prompt, the beat quality, the template, or the index. The assignment dump — story, arc, per-slot reasons, chronology flags — exists to separate them.