I’m reviewing AI course notes and including two ways of making videos programmatically. My favourite method uses an open-source pipeline called claude-shorts. It utilizes Claude Code to completely automate the longform-to-shortform translation without relying on an external SaaS subscription. Here is how the pipeline works.
1. Transcribe: GPU Accelerated.
The long-form video (such as a podcast or YouTube link) is transcribed using tools like faster-whisper to get precise word-level timestamps.
2. Analyze & Score: Claude 3.5 Sonnet Brain.
The full transcript is fed to Claude. Instead of just looking for keywords, Claude analyses the narrative arc and scores potential clips against a 5-dimension rubric: hook strength, coherence, emotional weight, value density, and narrative payoff.
3.Review the Edit Decision List (EDL): Terminal UI.
Claude outputs a clean, machine-readable table directly in the terminal showing candidate segments, their exact timestamps, the calculated hook, and its structural rationale.
4.Snap & Reframe: Audio-Aware Boundary Check.
Once a clip is approved, the system uses audio-aware boundary snapping to ensure cuts don't happen mid-word or mid-sentence. For screen recordings, cursor-tracking logic identifies mouse movements to dynamically reframe the 16:9 video into a 9:16 vertical crop.
The data is handed off to Remotion (a React-based video framework), which programmatically burns in animated, word-level karaoke-style captions and exports a finished .mp4 optimized for YouTube Shorts, TikTok, or Reels.
