Rendering Films in a Browser

Our social posts start from real case studies, so we wanted a way of moving real pictures that didn't mean opening an editing timeline every morning. We ended up writing a small renderer. It runs on the Chrome and Node that are already on the computer, with nothing else to install.

A scene is a short spec

A post is described in a few lines: which pictures, what shape, which scene, how long. The scenes are the same moving blocks we use on our case study pages, such as strips, walls and card stacks, so a film and its page share the same motion. The film below is one of them.

Draw each frame at an exact time

The renderer serves the files locally and opens a headless browser. For each frame, it asks the scene to draw one exact moment, takes a picture and encodes it. Since time is set frame by frame, no frame can drop, and how fast the machine happens to be has no effect on the motion. That's what makes it different from recording a screen.

WebCodecs does the encoding

Browsers now have direct access to video encoders. The WebCodecs API encodes and decodes audio and video frame by frame, with hardware acceleration, and VideoEncoder turns frames into encoded chunks. The specification is a W3C Working Draft. It isn't yet Baseline across browsers, which doesn't matter here, since we choose the browser.

One thing to know is that WebCodecs doesn't do muxing, so it won't give you a file you can play. We wrote a small MP4 writer for that, which also puts the index first, so a film starts playing before it's finished loading.

Real pixels

The rule that matters most to us: move the real pictures. Nothing is generated or faked. A browser moves the same file that's on the site, and the film is encoded once, at the end. For some posts we go one step further and redraw a piece of UI in code, which keeps the type perfectly sharp and lets each part move on its own.

Check the seams

Social films loop, so the renderer compares the last frame with the first and reports whether the seam is clean. It also plays the film back and compares it with what was captured, because a film that looks fine in the encoder can still surprise you in the player.

What it doesn't do

It has no timeline and no sound design, and it isn't meant for editing footage. It's for one thing: making real work move, quickly and at high quality. If you're making formats for a launch, the planning is a different problem, covered in One Launch, Every Format.

You can see more of what comes out of it, including the things that didn't work, in LaBS. The motion itself comes from our website work, which is the next post's subject: fast pages with very large pictures.