Writing Making of

Built with sound: teaching Claude to direct a short film

It started with a thread on X about Claude directing stop-motion films. A few more threads like it followed, and I wanted to know how much of that I could actually do myself. So I turned what I read into a skill for Claude Code (instructions it loads when a task needs them) and started with something deliberately simple: make the picture and the sound agree.

The idea: every movement is a sound

A tiny Rube Goldberg machine, flat vector, in the Opper colours. A spring launches a ball, it rolls down a ramp, different things get contacted and create sound, ends up lighting a neon sign.

A cyan marble bouncing across four coloured chime bars, with a funnel on the right The marble plays four notes on the bars, tuned to the key of the music.

Every contact sits on the beat of the music, and every contact has its own sound effect, generated to describe that exact material and motion: a spring twang, a wooden tock, a domino click, a bell.

The tools

  • Rendering. The film is a web page. HyperFrames (open source) renders an HTML page to video, frame by frame in a headless browser, so anything you can draw on the web can become a film. The animation is GSAP, with every value a function of time so each frame is exact. I also looked at Remotion (video in React), Motion Canvas and Manim (code-driven explainers); HyperFrames fit best because Claude can write plain HTML and CSS and see the result straight away.
  • Sound. The music and the thirteen effects are from ElevenLabs. FFmpeg mixes them, lowers the music a little at each contact so the effect cuts through, and masters to streaming loudness.
  • One timeline. A single cue file holds every time in the film. The picture and the sound both read it, so they cannot drift apart.

What I learned

  • Sound first, then picture. Generate the music, measure where its beats actually are, and put the actions on them.
  • One set of instruments for the whole piece. Mixing styles between sections sounded like several songs glued together.
  • Check what the critic says. I asked an AI to review the frames.

What it cost

About $0.11, for the music candidates and the sound effects. The rest is code.

Next: the same approach, but with generated pictures: paper cut-outs.

VideoClaude CodeHyperFrames