Reelly · iOS · auto-edit architecture
Reelly renders a reel from a template and your clips. The engine works. The selection doesn't exist — it sorts by capture time and takes the opening of every clip, which in handheld footage is the phone coming up. This is the pipeline that replaces it: catalogue every moment in the footage, hand all of them to a director that decides what story to tell, then let an editor fit that intent to the template's fixed rhythm.
These three lines are the entire selection logic in the app today. Read together, they describe a product that picks the worst moment in every clip and then ignores more than half the footage.
| Location | Code | Consequence |
|---|---|---|
| ReelStudio:83 | CaptureDate.ordered(clipURLs) | Order is capture time. Nothing else is considered. |
| ReelRenderer:74 | clipAssets[clipCursor % count] | First 46 clips fill 46 slots positionally. Clips 47–107 are never looked at. |
| ReelRenderer:87 | CMTimeRange(start: .zero, …) | Every cut is the head of a clip — the framing settling, the blurry pan. |
A complete, benchmarked selection layer already sits in ios/AutoClipping/Indexing/
— scored 3/3 on short-clip ground truth and 4/4 on long, across a six-model bake-off.
Nothing outside that folder ever called it. Every prior judgement about whether Reelly works
was made about a build that never ran it.
The build that exists was made to record demos and cut ad creatives, and it is good at that. It is not deleted — it is separated, and the real product is written fresh beside it rather than on top of it.
The split is cleaner than it looks, because the demo path never renders anything.
DemoMode.runSimulatedRender is a fifteen-step fake progress ramp returning a
pre-baked file, and demo mode holds zero references to the renderer. What
genuinely is a mess is that demo logic sits inside production functions —
sixteen branch sites across six files, seven of them buried in the timeline editor. Every
one is a fork in a function that is supposed to be doing real work.
Zero DemoMode references outside Demo/. Greppable,
so it is a lint rule rather than a discipline. Today there are sixteen; the target is none.
The two paths produce video differently — demo plays pre-baked files, the real path composes. A promo recorded from the demo path is not evidence the real path works, and must never be accepted as verification of a real-path change.
Modelled on how a cut actually gets made: someone documents the footage, someone decides the story, someone makes the cuts work, someone renders it. The filters between them are pure arithmetic — no media, no model.
One call per clip over frames sampled at 1 per 2 seconds. Writes down a prose synopsis and the moments that actually happen, each holding the 1–3 second highlights a short slot can cut to. Documents everything, chooses nothing.
out synopsis · moments · highlights · facets
Before cataloguing: skip clips over 12 minutes, skip finished edits. Before directing: a clip is eligible for a slot only if it is at least as long as the slot. Both are arithmetic, and the second is what removes stretching from the system entirely.
One call. Sees every moment across every clip at once, plus the template's full shape and its music profile. Composes a story to that rhythm and says which moment goes in which cut. Broad strokes only — no timings, no arithmetic, no repair.
out story · arc · a beat per slot
The craft stage. Takes the director's intent into the real footage and chooses exactly one number per slot: the in-point. Absorbs anything the director got wrong, locally, never by asking again.
out cut list
Cut list to MP4. Trim, normalize aspect, apply per-slot effects, replace source audio with the template's music bed and timed SFX, export.
out MP4
Why the work splits exactly here
| Catalogue must not choose | If indexing nominated a best moment per clip, the director would inherit judgement made without seeing the rest of the trip. |
| Story judgement only exists globally | Catalogue is per-clip; any scoring is local. "What story does this footage tell?" can only be asked by something seeing all of it at once. |
| A ruleset cannot construct a story | The catalogue carries no focus, shake, exposure or composition signal. A hand-written scorer could only avoid known defects. That is a floor, not an edit — which is why there is no scoring stage. |
| Director and editor are different jobs | A director that also handled timing would be doing arithmetic across dozens of simultaneous constraints — unreliable, and exactly where a wrong number breaks an export. |
| Repair belongs to the editor | Nothing calls back to a model to fix a timing problem. Repair stays deterministic, preview and export stay in agreement, and every director output is renderable. |
The old schema asked for 3–8 beats covering the whole clip. On a long clip that forces every entry to be enormous, and the model resolves the conflict by merging distinct events. The give-away is "followed by" appearing inside a single action line:
| 179s clip, measured | What the model wrote |
|---|---|
| beat 0 0.1–23.8s | "A young man walks onto a train platform, followed by shots of the snowy exterior…" |
| beat 2 95.3–119.1s | "A shot of the snowy scenery outside is followed by a young woman in a face mask…" |
Measured across 13 clips: median beat 4.0s but max 55.0s, 14 of 50 over ten seconds, 20 of 37 adjacent pairs separated by a gap of up to 11.9s, and one beat of exactly zero length. So the fix is not more frames — it is removing the cap and the cover-everything rule, and letting the count follow what actually happens.
The schema
A caption gives the director nothing to build a story from.
38 of 46 slots are under 4 seconds, so the leaf level is the common case, not an edge case.
These were one dialogue field. The subway clip reports hasSpeech: false
yet a populated string: a station display reading "Bound for Ueno & Ikebukuro".
Carried unchanged — this half was benchmarked and works.
Measured economics — gemini-3.1-flash-lite via OpenRouter, 2026-08-09
| Frame size | Tokens / frame | KB / frame | Upload, whole trip |
|---|---|---|---|
| 384x216 | 1,070 | 6.9 | 15.6 MB |
| 768x432 default | 1,070 | 17.8 | 40 MB |
| 1024x576 | 1,070 | 26.1 | 59 MB |
| 1920x1080 | 1,070 | 61.2 | 138 MB |
Pixel size has no effect on token cost — flat at every size tested. So resolution is free in tokens and expensive on the wire, which makes frame size a bandwidth and device-time decision rather than a cost one. Cost is a function of frame count alone: ~1 cent per minute of footage at 1 frame / 2s, or $0.61 for the whole trip.
Frame timestamps carry roughly ±0.3s of error, and the sampler currently records the requested time rather than the one the decoder returned. Moment and highlight bounds are guidance, never precise cut points — compensating for that is the editor's job. And the flat-rate-by-resolution finding is a quirk of this model and provider on this date; a change to either invalidates it.
One call, everything visible at once: every clip's synopsis and facets, every moment and highlight, the template's full shape — slot count, each slot's duration and effects, where the black slot falls — its music profile, and the eligibility map.
Slot duration is a pacing signal, not a constraint to solve: a payoff belongs on a seven-second hold, not a half-second flash. The template is fixed. The director chooses what goes into a known skeleton; it does not design the skeleton.
Rhythm is inferred, never annotated
Only about four structural facts in the template are unambiguous. Everything between them is interpretation — and which reading is right depends on the footage. A trip with one spectacular moment wants a different arc than a steady build, over the same template. Annotating phases in the JSON would freeze one arbitrary reading across every trip.
| Unambiguous fact | Evidence |
|---|---|
| Slots 0–4 are one gesture | all 0.48–0.63s, all carry zoomIn |
| Slot 5 is a break | the black slot |
| Slots 13–23 are a coherent block | every slot in a tight 3.10–4.97s band |
| Slot 46 is the resolve | longest at 7.32s, and final |
The director must return an arc assigning every clip-backed slot to exactly one
phase from a fixed vocabulary. It forces a committed structure rather than 46 ad-hoc picks,
makes the edit explainable, and is mechanically checkable: does
payoff or closer land on long slots, and does opener
land on the flash burst? A payoff mapped to a 0.48-second slot means the director did not
read the template. That is an assertion, not a judgement.
The music profile is authored once, into the template
The director cannot hear the track, and the music never changes for a given template — so deriving its properties at render time would recompute a constant on every export. BPM and energy come from DSP over the bundled audio, mood and genre from an audio model, song context from web search. Energy is expressed in slot ranges, not seconds, because "slots 13–23 sit under the loudest section" is actionable and "upbeat" is not. One field stays hand-authored: what each SFX means — the stinger at 4.62s is a sponsor beat, not a whoosh, and no analysis recovers that.
Rules it works under: multiple moments from one clip are encouraged — a well-shot clip with three good moments should carry three cuts. Story order is permitted, but it must declare its arc, give a reason per cut, and flag every chronology break, so a surprising edit is a stated choice rather than a mystery. And no timings — it names a moment, never computes an in-point.
The template is a fixed skeleton and none of it is the editor's to change. The durations came from a real edit cut to that track — slot boundaries are the music. An editor that adjusted a duration would drift the whole reel off the beat.
For each slot, choose one number: the in-point. The out point is not a
decision — it is inPoint + slotDuration. Everything else the editor does is
in service of that one choice, or of recovering when it is constrained.
| Job | What it does |
|---|---|
| Place the moment | Turn "highlight 2 of clip 7" into an in-point. Bounds are guidance, not walls — it may start before or after them if that is the better frame to come in on |
| Fit the slot | If the moment sits too near the tail, slide the window earlier until it fits. Full speed, moment still inside |
| Make it land | With duration fixed, choose an in-point whose entry frame reads well: not mid-motion, not a jarring jump from the previous shot's exit |
| Absorb mistakes | An unusable pick becomes a neighbouring moment or another unused one — locally, never by re-asking the director |
| Clamp to reality | Bounds checked against the actual asset, not the catalogued duration, which can differ after a re-export or transcode |
Stretching and freeze-holding are not fill strategies — they are disqualifiers, applied upstream as eligibility. Slowed handheld footage without optical-flow interpolation looks bad, and a freeze mid-reel reads as a bug rather than a choice. With hundreds of moments competing for 46 slots there is better than 10× supply, so a forced fit is never necessary. The cut list has no representation for a stretched cut.
The cut list is the artifact: what gets logged, what the UI draws shot cards from, and what a
human overrides. Each entry carries a source of directed or
repaired and a one-line note, so a reel that disappoints is arguable rather than
merely re-runnable.
One honest consequence: the editor reads media, so it is not a pure function. Its placement rules can be tested with injected media facts; its frame inspection has to be verified on real footage.
subway_train arriving station.MP4, from the real trip folder. Forty-seven seconds.
It opens on a static platform and stays there; around twenty-six seconds a train pulls in.
Today this clip contributes a 2.35-second cut of an empty platform, because 2.35 seconds is what slot 7 asks for and the read starts at zero.
The 47-second duration, the four moment boundaries and the ~0:26 train arrival are real — from the catalogue run and hand-labelled ground truth. The nested highlights are illustrative: the new schema has not been run yet, so no recorded highlight exists for this clip. That is precisely what the first run will produce.
| Stage | What happens to this clip | Result |
|---|---|---|
| intake | 47s, raw footage — passes both filters | catalogued |
| catalogue | 24 frames at 1 per 2s, one call → synopsis, moments, nested highlights | ~$0.006 |
| eligibility | 47s clip vs every slot — longest slot is 7.32s | eligible for all |
| direct | Its moments join hundreds from 107 clips. The director takes the train for slot 7 and the doors for slot 9 | 2 slots, 1 clip |
| edit | Highlight bounds become in-points; entry frames checked; runway confirmed against the real asset | inPoint ≈ 26.0s |
| render | 2.35s read from 26.0s, media measured from the in-point forward, zoom applied, music over | cut 7 of 47 |
The line that matters is direct. One clip, two slots — impossible under any
design that picks one moment per clip, and the reason the search space is moments rather than
clips.
Every segment of tokyo-vlog.json at its real duration — extracted from an
actual CapCut project, 47 segments across 111.4 seconds. Width is proportional to time.
38 of 46 slots under four seconds is why highlights nest inside moments.
Nothing approaches the 12-minute cap, so it excludes nothing today and exists to bound a pathological input.
The JSON preserves four CapCut features the renderer does not yet apply: per-clip pan and scale keyframes (only the opener's zoom is mapped), three text overlays, a timeline fade-out at 109.13s, and the music fade. So even a perfect edit will not match the reference vlog until those land — worth knowing before judging the gate reel against it.
This template is 16:9 and 111 seconds — a YouTube-shaped vlog, not a vertical short. The decision was to evaluate against the real edit anyway. If the verdict is "wouldn't post it", the reason has to be recorded: the edit, the moment quality, or the format.
Twenty-six decisions survive. The ones below are the load-bearing reversals — each overturned something that was written down and looked reasonable.
| Was | Is | What forced it |
|---|---|---|
Wire selection into ReelStudio.generate() | Build fresh; separate the demo path | The real user flow never calls generate(). The design doc named the wrong function, so the error was inherited |
| Rank moments with a Swift heuristic, assign greedily, one clip per slot | One director call over every moment | A ruleset can only avoid defects; nothing in the pipeline ever saw the whole trip; and one-clip-per-slot caps creativity by construction |
| Director resolves in-points; resolver re-asks it on failure | Editor owns timing and repairs locally | An LLM round-trip inside composition would fire on every preview rebuild, and preview and export could settle differently |
| Score beats on speech and dialogue | Penalise speech; split it from on-screen text | The render discards all source audio. The rule selected silent talking heads, and conflated a station sign with talking |
| 3–8 beats covering the whole clip | Moments scaled to content, highlights nested | Measured: max beat 55s, 14 of 50 over 10s, 11.9s gaps, one zero-length beat. The cap was merging distinct events |
| Cap frames at 16 per clip | Sample at 1 frame / 2s, uncapped | A flat cap gives 11.9s spacing on a 179s clip. Uncapped costs four cents more and buys a uniform guarantee |
| Shrink frames to save money | Frame size is a bandwidth decision | Measured flat at 1,070 tokens/frame from 384px to native 1080p. No cost lever exists |
| Stretch or freeze a clip that is too short | Ineligible, filtered before the director runs | Slowed handheld looks bad and a freeze reads as a bug. With 10× supply a forced fit is never necessary |
| Annotate the template with rhythm phases | The director infers rhythm and returns an arc | Only ~4 structural facts are unambiguous; the right reading depends on the footage, so precomputing freezes one across every trip |
| Show a progress bar for the indexing wait | Move indexing off the critical path | The answer to a long wait is not to have it in front of the user |
The engineering review cleared 24 findings — against an architecture that has since been replaced twice. The Edit stage, eligibility, the music profile and the entire catalogue redesign are un-reviewed. That is the honest state, and it is worth a second pass before implementation.
Arranged so the cheapest answer to "does this work" arrives before any product infrastructure. Nothing here is built yet.
One reel from the 107 clips the dumb way: chronological, head of each. Keep the file. Without it, "it's better" is unfalsifiable. This is the last useful thing the old path does.
Rewrite the catalogue prompt and schema, then run it over the 107 clips with the Python harness (~$0.61). Director prompt and call. Editor and eligibility as tested functions. Renderer taking a cut list. Cut list dumped beside the reel. No UI, no persistence, no cache. Re-tune the director as often as it takes — a cent a run against an index that never has to be rebuilt.
Binary and subjective by design. Deciding not to post is equally informative — but record which reason it was: the edit, the moment quality, or the 16:9 format.
Import and cataloguing on device, content-identity caching so re-imports are free, the review surface with shot cards and overrides, export to Photos. Two things already decided land here: separate extraction and network concurrency pools, and cataloguing off the Generate critical path.
It is iCloud. The trip is 27.3 GB. If clips are not already on device, each one is fetched over the network before a frame can be decoded, and no amount of concurrency helps. Everything else fits in a few minutes: API time at 8-way concurrency is about a minute, frame extraction 30–90 seconds, the director call another 30–60.
The comparison reel comes out of a partly offline pipeline — cataloguing runs in Python, not in the app. It proves the edit, not the shipped path. And the gate now tests a bigger bet than it used to: not "does better trimming beat head-trim" but "can a director construct a story". More candidate causes when it disappoints, which is what the cut list dump exists to separate.