A Video Codec for Two Neural Networks

I shrank a one-minute dashcam video to 216 KB. That's 174 times smaller than the original. The catch is that nobody watches it. Two neural networks do, and they don't care that my rebuilt frame on the right looks like the inside of a lava lamp.

Most of the savings came from one part, a five-color map of the road. I found a way to store it 27% smaller than AV1 does, with no loss at all. The trick is to guess each pixel from its neighbors. My entry finished 32nd out of 70 on the final leaderboard.

The judges

The comma video compression challenge gives you a one-minute dashcam video. You make the smallest file you can, as long as a script can turn it back into video.

Here's the odd part. Nobody watches the result. Two neural networks do.

The video comes in pairs of frames. The first network, SegNet, labels every pixel in the second frame of each pair: road, sky, car, and so on. The second network, PoseNet, looks at both frames and says how the camera moved, using six numbers.

Your score adds up three things. Lower is better:

score = 100 × SegNet error
      + √(10 × PoseNet error)
      + 25 × (archive bytes / 37,545,489)

The first two measure how far each network's answers drift from what it said about the real video. The last one is the size of your file, compared to the original video.

Here's what SegNet says about a real frame, next to what it says about my rebuilt one:

They differ on 162 pixels out of about 200,000. Every one of those is on or next to an edge. PoseNet's six numbers are close too.

That's all the score ever sees: one five-color picture and six numbers for each pair of frames. The color of the asphalt doesn't matter. Neither does the texture of the sky, or whether a person would call the picture a road.

So the rebuilt video can look like this:

Store what the judges said

If the score only reads the judges' answers, you don't need the video. You can keep the answers and build something that earns them again.

That's the idea behind Quantizr's entry. For each pair of frames, you keep three things:

• SegNet's answer for the second frame, the five-color map from above.
• PoseNet's six numbers.
• A small neural network that I call the painter. Give it a map and six numbers, and it paints both frames.

Quantizr trained the painter against the two judges. It learned to paint whatever earns the right answers. Nothing rewards a picture for looking like a road, so it doesn't paint one. That's the lava lamp.

I started from a smaller version of this design by EthanYangTW. Here's where its bytes went:

The map took up almost 80% of the file. If I wanted a smaller file, I had to shrink the map.

The map has to be perfect

Quantizr stored the map as grayscale AV1 video, which makes sense for 600 pictures in a row. AV1 has a quality setting called CRF. Turn it up and the file gets smaller. The picture gets a little blurry: a blotch moves, or an edge shifts by a pixel. In a normal video, nobody notices.

But would the painter notice? I took the exact map and squeezed it with AV1 at six settings. Then I gave each result to the painter instead of the original. The chart shows how much worse PoseNet's error got. It's a log scale, and I measured 60 pairs spread across the video. I used a different AV1 encoder than the one behind the original file. Its CRF numbers aren't comparable to anyone else's.

A frame has about 200,000 pixels. Change eight of them and PoseNet's error doubles. Change fifty-six and it gets almost seven times worse. SegNet notices much less.

Here are 223 changed pixels, and what they do to the painting:

Every changed pixel turns into a smudge in the painting. SegNet shrugs it off. PoseNet doesn't.

The kind of change matters too. I tried two kinds by hand. One moves a pixel on an edge by one step, which is what lossy codecs do all the time. The other drops a wrong-colored speck into the middle of a flat area.

Moving forty edge pixels per frame hurt about as much as AV1 at CRF 30 does. Forty specks were more than twenty times worse than that. My guess is that the painter trained on AV1 maps, where edges wobble by a pixel, and seldom saw a speck.

Then I tried something that sounded safe. The map should match SegNet's answer for the real frame. But AV1 had already changed about 80 pixels in each frame I tested. So I gave the painter SegNet's true answer instead.

SegNet liked it. Its error dropped by about a third. PoseNet's error got fourteen times worse. A more accurate map didn't help. It wasn't the map the painter learned from, and that was enough.

I also tried to repair damaged maps with a patch list, a denoiser, and retraining. None of it paid off.

So the map has to match what the painter learned from, bit for bit. Lossy compression only works when the viewer can't tell the difference. This viewer is a neural network that has only ever seen one file.

That makes it a lossless compression problem, and lossless looked hopeless. General-purpose compressors like zstd and Brotli made the map twice as big as AV1 did, or worse. Two codecs I wrote for maps like this did no better.

Guessing pixels

The map changes little. About 99% of pixels have the same color as the same pixel in the previous frame. All but one pixel in 1,500 matches that pixel, the one to its left, or the one above. If you can guess each pixel from what you've already seen, you only pay when you guess wrong.

The coder reads the map one pixel at a time, down each column, left to right. Before each pixel, it looks at nine pixels it already knows: six in this frame and three in the previous one. Those nine colors are the pixel's context. For each context, the coder keeps a tally of how earlier pixels with that context turned out. Then it asks up to four questions, in order:

1. Is it the same color as the pixel to its left?
2. If not, the same as the pixel above it?
3. If not, the same as this spot in the previous frame?
4. If not, which of the five colors is it?

Each answer costs bits. The price depends on how sure the tally was.

Below is a patch of the real map, with the real numbers from the coder. Pick a pixel and see what it knew.

The decoder repeats the same updates in the same order, so the file doesn't have to store any tallies. It's about 300 lines of C++, and it decodes the whole map in a couple of seconds. You can read it here.

None of this is new. JBIG compressed black-and-white fax pages the same way. I added the previous frame and three more colors.

With five colors and nine neighbors, there are about ten million possible contexts. That sounds like way too many for 118 million pixels. I'll come back to that.

Here's how the map shrank, one change at a time:

Two changes did most of the work. First, each tally forgets old data once it gets too big, and I'd set that limit too low. Raising it saved 17 KB with a one-line change. Second, I read the map down columns instead of across rows, which saved another 6 KB. Pixels in the same row agree a bit more often than pixels in the same column. My guess is that this made the first question easier.

I also measured the best this kind of model can do on this data. My coder is within two percent of that. To go lower, I'd need a better model.

Where the bytes go

Stored at one byte per pixel, the map would be 118 MB. My coder gets it down to 159 KB. That's about one bit for every 93 pixels.

sky: 260 byteshood: 76 bytes

Almost all the bytes go to the edges between colors. Those pixels make up about 2% of the map and use about 95% of the bytes.

The top third of every frame is sky, and nothing else. It never changes, so the coder learns it fast. All 600 skies together cost about 260 bytes. That is the exact length of this paragraph. The hood at the bottom costs only 76 bytes, about one short sentence.

Remember the ten million contexts? The coder only ever uses about seven thousand of them, and half of all the bytes go to fifteen.

The result

The final file is 216 KB:

The map is about three quarters of it. The rest is the painter and a few small extras.

I did most of the searching with an AI agent. It tried one idea at a time and kept a list of what failed, so we didn't repeat dead ends.

The file scored 0.288, and the contest maintainers tagged it as a "new approach."