Reelly — Auto-Edit Pipeline Architecture

01

The defect, in three lines of shipping code

These three lines are the entire selection logic in the app today. Read together, they describe a product that picks the worst moment available and then throws away more than half the footage.

Location	Code	Consequence
ReelStudio:83	CaptureDate.ordered(clipURLs)	Order is capture time. Nothing else is considered.
ReelRenderer:74	clipAssets[clipCursor % count]	First 46 clips fill 46 slots positionally. Clips 47–100 are never looked at.
ReelRenderer:87	CMTimeRange(start: .zero, …)	Every cut is the head of a clip — the framing settling, the blurry pan.

Why this matters more than it looks

The benchmarked selection layer already exists in ios/AutoClipping/Indexing/ — seven files, scored 3/3 on short clips and 4/4 on long ones against hand-labelled ground truth. Nothing outside that folder calls it. Every prior judgement about whether Reelly works was made about a build that never ran it.

02

Four stages

Clips enter as picked camera-roll assets and leave as an MP4. Two stages already exist; two are new and both are pure functions, which is the point — the decisions become testable instead of only watchable.

01

Catalogue

built · unwired

Sixteen frames are sampled across the clip, labelled with their timestamps, and sent in one call to a vision model. It returns a screenplay: three to eight chronological beats, each with a time range, a present-tense action line, a camera note, and any dialogue. Plus clip-level facets — setting, shot type, subjects, mood.

It deliberately does not nominate a best moment. It documents what is there, which means the beats arrive as candidate windows already.

in clip file → out ClipIndex · ~$0.0034 and ~3.3s per clip

02

Score

new · pure

Each beat is ranked for a specific slot length. This is a defect-avoidance ruleset, not a quality model — the index contains no focus, shake, exposure or composition signal, and pretending otherwise would be dressing up a string match.

Free to tune: no model call, no re-indexing, no schema version bump.

in ClipFacts → out ranked candidates

03

Plan

new · pure

Chooses which 46 of ~100 clips to use, keeps them in the order they were filmed, then picks the beat inside each one for the slot it landed in. A repair pass fixes long slots that landed on clips without enough runway.

Deterministic and unit-tested. Same input, same reel, every time.

in template × [ClipFacts] → out [ClipPlacement]

04

Render

built one arithmetic fix

Walks the template's segments and pops a placement for each clip-backed one. It chooses nothing — both existing modulo loops are deleted. Effects, the music bed, timed SFX and the watermark pass are all untouched.

The fix: how much media to take must be measured from the in-point forward, not from the clip's total length.

in [ClipPlacement] → out MP4

03

One clip's journey

Concretely: subway_train arriving station.MP4, one of the thirteen real Japan trip clips in the repo. Forty-seven seconds long. It opens on a static platform and stays there. At around twenty-six seconds a train pulls in — the only thing in the clip anyone would want to watch.

Today this clip contributes a 2.35-second cut of an empty platform, because 2.35 seconds is what slot 7 asks for and the read starts at zero.

0:000:100:200:300:400:47

what ships today — head trim, static platform what the pipeline picks — train arriving

What is measured and what is illustrative

The 47-second duration and the ~0:26 golden moment are real: hand-labelled ground truth from the model bake-off in .claude/experiments/golden-moment/. The beat boundaries drawn above are representative of what a screenplay looks like, not a recorded index run for this clip. Beat timing also carries real error — frames are sampled with a ±0.3s tolerance, and the recorded timestamp is the one that was requested, not the one the decoder returned.

So the journey, end to end, for this one clip:

Step	What happens to this clip	Result
pick	Selected in the media grid with ~99 others; resolved from Photos to a readable file	URL + PHAsset id
catalogue	16 frames sampled across 47s (~3s apart), one vision call, screenplay returned	5 beats + facets
validate	Beats checked against the real 47s duration; degenerate or out-of-range beats dropped	5 beats kept
select	Clip-level score puts it in the chosen 46; capture time places it 8th in the trip	→ slot 7
score	Beats ranked for slot 7's 2.35s. Head beat rejected. Train beat has runway and a settled camera note	beat 4 wins
place	In-point set inside the winning beat	inPoint ≈ 26.0s
render	`take = min(2.35, 47.0 − 26.0)` → 2.35s read from 26.0s, zoom effect applied, music over	cut 7 of 47

04

What crosses each boundary

Four shapes. The one that carries the architecture is the last: a single ClipPlacement per clip-backed segment, produced once and consumed by every render call site.

ClipIndex — what the model returns

slug · scene heading, INT./EXT. style
logline · the whole clip in one sentence
screenplay · 3–8 beats: start, end, action, camera, dialogue
setting · shotType · subjects · enumerated facets
peopleCount · hasSpeech · mood · tags

exists · persisted on ClipAsset · schema v3

ClipFacts — the planner's input

id · captureDate · duration
beats · validated, clamped to real duration
facets · setting, shotType, subjects

new · a value snapshot. No SwiftData models and no file I/O cross this line, which is what keeps the planner pure and testable.

MomentCandidate — a ranked window

start · end · from a beat
score · from the ruleset, for a given slot length
why · the rule that decided it, for the debug dump

new · never persisted; recomputed freely because it costs nothing

ClipPlacement — the plan

clipID · durable identity, survives cache eviction
segmentIndex · binds to a slot, not an array position
inPoint · where the read starts

new · deliberately carries no URL. A cache path would go stale on eviction or relaunch; files are materialized late, just before render.

The structural win

Three places currently decide clip order and slot mapping independently: the studio, the renderer, and the timeline editor. With placements as the unit, the planner decides once and all three consume the answer. Preview-matches-export stops being a rule someone has to remember and becomes a property of the types.

05

The planner, in the order that matters

The first draft of this algorithm was incoherent. It promised chronological order and greedy-by-score assignment in the same breath, which are different algorithms, and it promised no clip is used twice and that one clip can fill all 46 slots. The sequence below removes the contradictions instead of arbitrating them.

01

Select

Score at clip level to choose which 46 of ~100 clips are worth using at all.

02

Order

Keep the chosen clips in capture order. Chronology holds by construction because it is decided before any score is consulted — the trip's sequence is the story, and it never competes with ranking.

03

Assign

The i-th chosen clip goes to the i-th clip-backed slot.

04

Choose

Only now, knowing the slot's exact duration, pick the best beat within that clip. No eligible beat is discarded early, which was the second flaw in the first draft.

05

Repair

A long slot that landed on a clip without enough runway swaps with a clip within three slots. Fixes long-slot starvation while moving chronology by a cut or two rather than abandoning it.

06

Fill

Fewer clips than slots? Reuse is permitted, explicitly and under test. No-repeat is a conditional invariant — it holds only when eligible clips outnumber slots, and the tests say so rather than pretending both always hold.

06

Scoring is defect avoidance, not taste

Worth being blunt about what the index can and cannot support. It has no measure of focus, shake, exposure, composition or expression. What it has is prose about what happens, a camera note, a shot type, and timestamps. So the ruleset targets known defects — the things that make a cut unwatchable — rather than claiming to recognise a good shot.

Rule	Why it exists
Reject the clip's opening second	This is the defect being fixed — the phone coming up, framing settling
Require in-point + slot duration ≤ clip duration	Runway. Without it the segment gets time-stretched to fill the slot
Prefer settled camera notes for long slots, motion for short ones	A 7.3-second hold on a whip-pan is unwatchable; a half-second cut wants energy
Penalise beats carrying dialogue or speech	Source audio is replaced entirely by the template's music — a talking head plays silent
Prefer concrete physical events in the action line	Keyword match against arrival, turning, jumping. Explicitly a heuristic, and tunable
Prefer beats away from the clip's tail	Avoids the camera-lowering frames at the end

The one that nearly shipped backwards

Weighting speech and dialogue positively was in the plan until review. The renderer discards all source audio, so that rule would have systematically selected silent talking heads under a music bed. A signal that cannot reach the output is noise at best; here it was actively inverted.

07

The target, drawn to scale

Every segment of tokyo-vlog.json at its real duration — extracted from an actual CapCut project, 47 segments across 111.4 seconds. Width is proportional to time, so this is a scale drawing of the thing being filled.

0:000:200:401:001:201:401:51

under 1s — 9 slots, selection barely shows 1–4s — 29 slots 4s and over — 8 slots, where selection earns its keep black block — consumes no clip

Two things fall out of the drawing. The opening five cuts are all around half a second, where "the golden moment" collapses into "any frame that isn't blurry" — no choice is visible there. And the eight long slots on the right, including that closing 7.3-second hold, are simultaneously where selection is most visible and where runway is scarcest. Those eight are the ones to judge the result on.

Known caveat, accepted deliberately

This template is 16:9 and 111 seconds — a YouTube-shaped vlog, not a vertical reel. The decision was to run the evaluation against the real edit anyway, so if the verdict is "wouldn't post it", the reason has to be recorded: selection quality, or format.

08

What review changed

Twenty-four findings, all resolved. Eleven from a four-section pass, thirteen from an independent second model. Three of them would have cost real time.

Was	Is	Cost avoided
Wire selection into `ReelStudio.generate()`	Wire into `ImportSession.prepare()`	The real user flow never calls `generate()` — it is the old home-screen path. The design doc named the wrong function too, so the error was inherited
Change `.zero` to the in-point	Also fix how much media is taken	`take` was measured from the clip's total length. Any in-point could read past the end and fail the whole export — and a drag in the editor triggers it, so it is ordinary, not rare
Score beats on speech and dialogue	Penalise them	Source audio is discarded. The rule selected silent talking heads
Build persistence, cache, progress UI, thumbnails, then evaluate	Prove the ranking first with the existing Python harness	About a week of infrastructure spent before knowing whether ranking beats head-trim
Four new types across nine files	Two new types across five	An on-device motion analyser deferred — it buys ±0.4s on slots whose median is 2.2s

09

Build order

The sequence is arranged so the cheapest possible answer to "does this work" comes before any product infrastructure.

Phase 0

Baseline · no code

Export one reel from ~100 real trip clips through today's build. Keep the file. Roughly thirty minutes, and without it "it's better" cannot be falsified.

Phase 1

Harness proof

Index one trip with the existing Python harness. Add a test target. Write the planner as a pure tested function fed from that JSON. Fix the renderer arithmetic. Render the planned reel and compare against the baseline.

Gate

Watch both · post one, or don't

Binary and subjective by design. Deciding not to post is equally informative — but record which reason it was, selection or format.

Phase 2

Wire it in · only if the gate passes

Clip identity and byte cache, awaitable indexing with progress and cancellation, placements through the timeline editor, per-segment thumbnails at the in-point.

Honest framing for phase 1

The comparison reel comes out of a partly offline pipeline — indexing runs in Python, not in the app. It proves the ranking, not the shipped path. Phase 2 is what makes it a product.