Measuring Video Stability: What Your Video Encoder Already Knows

I shot 6,000 video clips across trips to Italy, Japan, and New Zealand. Which clips are actually usable?

Some are tripod-steady. Some have smooth pans. Many are too shaky - handheld walking shots, bumpy car footage. Manual review would take 50 hours. I needed automated stability analysis that could scan all clips and tell me which segments are stable, which are shaky, and where to cut.

The obvious solution had fundamental flaws. But a better answer was hiding in the video files all along.

The Naive Approach: Pixel Difference

My first attempt was straightforward: if consecutive frames look very different, the camera must have moved.

def analyze_stability(video):
    motion_scores = []
    prev_frame = None
    for frame in video:
        if prev_frame is not None:
            diff = abs(frame - prev_frame).mean()
            motion_scores.append(diff)
        prev_frame = frame
    return average(motion_scores)

I ran this across all 6,000 clips. It worked - mostly. Videos got categorized into buckets (ultra-stable, excellent, good, fair, shaky) and the rankings felt roughly right. Tripod shots scored well. Walking footage scored poorly. I moved on to building the rest of the system.

But when I started pulling “stable” clips into DaVinci Resolve, something was off. Clips marked as “good” still had visible shake. Smooth pans were being penalized the same as jerky handheld footage. And a few clips I knew were rock-solid kept showing up as “shaky.”

A Closer Look: The Hiroshima Train Station

Let me show you what I mean. Here’s a clip from Hiroshima (C5749.MP4). The Pixel Difference algorithm rated it “good” with a mean pixel difference of 7.67:

Watch it. The first 15 seconds are clearly shaky - I was walking through the station. Then I stopped, framed the shot, and got some usable footage.

This single example reveals two fundamental problems with Pixel Difference.

Problem 1: A Global Score Hides Local Issues

The algorithm averaged everything together and called this “good” (7.67). But look at the per-second breakdown:

I nearly skipped this clip entirely, missing a perfectly usable 6-second segment at 23-29s.

The fix: Chunked analysis. Break the video into 1-second segments, calculate stability per chunk, output a timeline of usable segments.

Problem 2: Pixel Difference Measures the Wrong Thing

Here’s where it gets interesting. Look at this clip from Rome Airport (C1250.MP4):

According to Pixel Difference:

See all that green? According to Pixel Diff, this clip is mostly stable.

Now watch it. This is clearly shaky handheld footage - I’m walking through an airport, the camera is bouncing around. Yet Pixel Difference rated it “good” with a mean of 7.75.

The fundamental problem: Pixel Difference measures how much pixels changed, not whether the camera moved.

This creates two failure modes:

1. Motion blur hides shake. When a camera shakes, pixels smear across the frame. That smearing makes consecutive frames look more similar, not less. The very thing that makes footage look bad (blur from shake) makes the score look good.

2. Uniform scenes hide shake. A plain ceiling, clear sky, or smooth wall looks nearly identical frame-to-frame even if the camera is shaking wildly - there’s simply nothing to measure. Conversely, a detailed scene (like LED screens) shows high pixel difference even with a locked-off tripod.

Pixel Difference conflates camera motion, subject motion, and scene texture into one ambiguous number. We need something that tracks where pixels moved, not just that they changed.

A Better Technique

What if we could measure where pixels moved rather than how much they changed? Here’s that same Rome Airport clip analyzed two ways - Pixel Difference on top, a better technique on bottom:

Pixel Difference sees low, fairly uniform “motion” throughout - it can’t detect the shake because motion blur makes frames look similar. The better technique correctly identifies the constant jerky movement that makes this footage unusable.

We need a method that measures where pixels moved, not how much they changed. And it turns out video encoders already compute exactly this.

The Solution: Motion Vectors

Video compression already solved this problem. When H.264 compresses video, it asks: “Where did each block of pixels move from the previous frame?” The answer - a Motion Vector - is literally “this 16x16 block shifted 12 pixels left and 3 pixels down.”

Motion Vectors encode spatial displacement, not appearance change. A tripod shot of an LED screen has tiny Motion Vectors (nothing moved) even though pixels are changing color. A shaky handheld shot has large, erratic Motion Vectors even if the content is static. We’re just reading the encoder’s notes.

GOP Structure (Group of Pictures):

  I ← B ← B ← P ← B ← B ← P ← B ← B ← P ← B ← B ← I
  ↑                                               ↑
  Full                                           Full
  Frame                                          Frame

  ════════════════════════════════════════════════
  │               ~0.5 to 2 seconds              │
  ════════════════════════════════════════════════

  I = Intra (complete image, ~50KB)
  P = Predicted (reference previous, ~15KB)  
  B = Bidirectional (reference both, ~8KB)

Frame N:     [A][B][C][D]
             [E][F][G][H]
             
Frame N+1:   [X][A][B][C]    <- Everything shifted right by one block
             [X][E][F][G]       **Motion Vector**: (16, 0) for each block

Extracting Motion Vectors

Modern video libraries can expose this data. With PyAV (Python’s FFmpeg bindings):

EXPORT_MVS_FLAG = 268435456  # Enable Motion Vector export

container = av.open(video_path)
stream = container.streams.video[0]
stream.codec_context.flags2 = EXPORT_MVS_FLAG

for frame in container.decode(video=0):
    if frame.side_data:
        mv = frame.side_data.get(MOTION_VECTORS)
        # mv contains thousands of Motion Vectors
        # Each one tells us: this block moved (dx, dy) pixels

Each 4K frame contains around 40,000 Motion Vectors with x/y displacement in quarter-pixel precision.

Motion Vectors are 2D only - the encoder has no depth information. A dolly-in and a zoom look identical (radial expansion). This is fine for stability detection since camera shake primarily manifests as X/Y displacement.

Visualizing Motion Vectors

Here’s the C5749 clip with Motion Vectors overlaid (green arrows show how each block moved). Watch how the arrows become smaller and calmer after the 14-second mark when I stopped walking:

And here’s what those patterns look like schematically for different camera movements:

The pan has high magnitude but high consistency (all arrows point the same way). Shake has high magnitude but low consistency (arrows point everywhere). This is why direction consistency helps distinguish intentional movement from unwanted shake.

From this we can compute several useful metrics.

The Math

Each Motion Vector tells us how far a block moved horizontally ($dx$) and vertically ($dy$) in pixels. A block that shifted 12 pixels right and 3 pixels down has $dx = 12$ and $dy = 3$.

For each frame $n$, we compute the mean Motion Vector magnitude across all $k$ blocks:

$M_n = \frac{1}{k} \sum_{i=1}^{k} \sqrt{dx_i^2 + dy_i^2}$

The $\sqrt{dx^2 + dy^2}$ is just the Pythagorean theorem - the actual distance each block moved, regardless of direction. We average this across all blocks to get a single number representing “how much is moving” in that frame. A static shot has $M_n \approx 0$; a fast pan might have $M_n > 100$.

Jerk (our key metric) is the frame-to-frame change in motion magnitude:

$J_n = |M_n - M_{n-1}|$

Smooth motion (steady pan, stable tripod) has low jerk - the magnitude stays consistent. Camera shake has high jerk - the magnitude spikes erratically as the camera jerks around.

For each 1-second chunk of video, we take the maximum jerk as the chunk’s score:

$J_{chunk} = \max_{n \in chunk}(J_n)$

Why max instead of mean? Because a single bad frame ruins a shot. If 24 frames are smooth and 1 frame has a violent jerk, that chunk is unusable. The max catches this; the mean would hide it.

Why Not Direction Consistency? (And Why Zoom Detection Fails)

You might think we need another metric: direction consistency - whether all blocks are moving the same way. This would distinguish a smooth pan (all vectors pointing left) from chaotic shake (vectors pointing everywhere).

$C = \left\| \frac{1}{k} \sum_{i=1}^{k} \frac{\vec{v}_i}{|\vec{v}_i|} \right\|$

But here’s the insight: jerk already captures this implicitly. A smooth pan has high magnitude but consistent magnitude frame-to-frame - so low jerk. Camera shake has magnitude that spikes erratically - so high jerk. We don’t need to measure direction because the consistency of motion over time is exactly what jerk measures.

Think of it this way: a pan moves 10 pixels left, then 10 pixels left, then 10 pixels left. Jerk ≈ 0. Shake moves 10 pixels left, then 15 pixels up, then 3 pixels right. The magnitude jumps from 10 to 15 to 3 - jerk spikes on every frame. The directional chaos shows up as magnitude variance, which jerk captures.

What about zoom? In theory, zoom creates a radial pattern - vectors pointing outward from center. But on repeating architecture, the encoder matches one cornice to another cornice 10 meters away. The Motion Vectors show visual matches, not actual motion. For my use case, I accept zoom as a known blind spot: a smooth zoom is usually intentional and usable anyway.

The Subject Motion Problem

Jerk alone has a blind spot: what if a person walks through frame? A woman walking through an art gallery (C2713.MP4) - camera on tripod, perfectly stable - got flagged as shaky because her movement created erratic Motion Vectors.

The key insight: camera shake moves all blocks uniformly, while subject motion moves some blocks while others stay still.

This is exactly what Coefficient of Variation measures:

$CV = \frac{\sigma}{\mu} = \frac{\text{std}(magnitudes)}{\text{mean}(magnitudes)}$

• Low CV (< 0.7): All vectors have similar magnitudes = uniform motion = camera shake
• High CV (> 1.0): Some vectors are large, others small = non-uniform motion = subject moving on stable camera

Here’s what high CV looks like in practice. This frame from C1255 shows the Jewel waterfall creating large Motion Vectors while the rest of the frame stays relatively still:

The white lines are Motion Vectors - notice how many more appear in the waterfall area (showing movement) compared to the static background. This non-uniform distribution produces CV = 1.33, correctly classifying this as “subject motion” rather than camera shake.

The combined algorithm becomes:

def classify_stability(max_jerk, avg_cv):
    if max_jerk <= 15:
        return 'stable'  # Low jerk = stable regardless
    elif avg_cv < 0.7:
        return 'shaky'   # Uniform motion = camera shake
    elif avg_cv > 1.0:
        return 'stable'  # Non-uniform = subject motion, camera fine
    else:
        return 'shaky'   # Mixed zone, be conservative

Validation: Human vs Algorithm

To test this, I watched two clips frame-by-frame and manually classified every second as “stable” or “shaky” based on whether I’d actually use that footage. Then I compared my judgment to what the algorithm produced. 96 seconds total, no cherry-picking.

C2713: Nicole at the Met, New York (38 seconds)

Result: 100% agreement with my manual review. The algorithm matched my judgment on every second:

The CV approach correctly distinguished between camera shake (sec 1, 3-8) and subject motion (sec 21-36). Without CV, the subject’s movement would have been misclassified as camera shake.

C1255: Reacting to the Jewel, Singapore (58 seconds)

Result: 91% strict agreement with my manual review, 98% practical accuracy.

The 4 errors at sec 2-5 mean I’d skip usable footage - the algorithm wrongly flagged it as shaky. That’s lost opportunity. The 1 error at sec 42 is worse: actual shake marked as stable, so I’d waste time with unusable footage. Both error types matter, but at 95% accuracy across 96 seconds, the algorithm is reliable enough for triage.

Combined Results

For a triage tool, this is the right failure mode: be conservative. The 4 false positives flag footage for review that turns out to be usable - a minor inconvenience. The single false negative is the only real error, and even that was borderline footage I’d probably stabilize in post anyway.

Where the Methods Disagree: Times Square

Here’s the most striking example of why Motion Vectors beat Pixel Difference. This is Times Square at night - LED screens everywhere:

Pixel Difference sees constant change - the billboards are animated, pixels are changing everywhere. It rates only 30% of this clip as stable. But watch the footage: the camera is mostly steady. The Motion Vector approach correctly identifies 64% as stable because it measures where pixels moved, not that they changed. The LED screens change color but don’t shift position.

Practical Implementation

Storage Design

I chose 1-second chunks, merged for storage. Each chunk stores the maximum jerk value - if any frame in that second has high jerk, the whole chunk gets flagged. Adjacent chunks with the same rating merge into segments.

Store raw values, classify at query time:

JERK_THRESHOLD = 15  # adjustable without reprocessing
shaky_segments = [s for s in segments if s['max_jerk'] > JERK_THRESHOLD]

Try adjusting the threshold yourself:

Final Storage Format

Rather than one database row per second (which would be 300,000+ rows for my collection), I store a JSON summary per clip. Here’s the actual output for C1250.MP4:

{
  "segments": [
    {"start": 0.0, "end": 2.0, "rating": "stable", "max_jerk": 2.7},
    {"start": 2.0, "end": 7.0, "rating": "shaky", "max_jerk": 76.2},
    {"start": 7.0, "end": 8.0, "rating": "stable", "max_jerk": 4.8},
    {"start": 8.0, "end": 18.0, "rating": "shaky", "max_jerk": 73.5},
    {"start": 18.0, "end": 19.0, "rating": "stable", "max_jerk": 3.7},
    {"start": 19.0, "end": 24.0, "rating": "shaky", "max_jerk": 63.0}
  ],
  "chunk_size": 1.0
}

This tells me exactly where the problems are: seconds 2-7 (early camera adjustments with the worst jerk at 76.2) and 8-18 (the whip pan and its aftermath). The brief stable moments at 7-8s and 18-19s are preserved - they might be useful for transitions even if they’re short. Adjacent chunks with the same rating get merged into segments, keeping storage compact (typically 3-10 segments per clip) while preserving full timeline coverage.

with ProcessPoolExecutor(max_workers=6, max_tasks_per_child=1) as executor:
    futures = {executor.submit(analyze_video, v): v for v in videos}

Limitations and Edge Cases

This approach works well for my use case - quickly triaging 6,000 clips to find usable segments - but it’s not perfect. A few things to keep in mind:

Resolution Affects Results

Lower resolution = fewer blocks = less granular motion detection. I tested the same clip (C5749) at multiple resolutions:

The 360px proxy detected 18 more seconds of stable footage than 4K - downscaling averages out micro-jitter that creates real MV jerk at higher resolutions. For reliable results, analyze the highest resolution available. If the proxy shows shaky, the original definitely is; if the proxy shows stable, verify on the original.

High-Contrast Edges Help

Encoders find MVs by searching for matching blocks. High contrast (architecture, faces) = precise MVs. Low contrast (sky, fog) = unreliable MVs. Footage of mostly blue sky may produce spurious results, but travel footage with scene content works reliably.

It’s a Triage Tool, Not a Judge

A clip flagged as “shaky” probably is. A clip flagged as “stable” is worth reviewing but might still have issues (focus hunting, rolling shutter). For 6,000 clips, reducing manual review from 50 hours to 5 hours is the win.

If Your Camera Records Gyroscope Data, Use It

Everything above describes a universal solution - Motion Vectors exist in any H.264/H.265 video file, regardless of what camera recorded it. iPhone footage from 2015, stock video from the internet, screen recordings - they all have MVs we can analyze.

But some cameras embed something even better: actual gyroscope and accelerometer data recorded during filming. This is ground truth for camera motion - no inference required.

Sony Cameras: IMU Data Built In

Sony mirrorless cameras (a7 IV and later, a6700, FX series, ZV series, some RX models) embed gyroscope data directly in the MP4 file. The data structure looks like this:

MP4 Container
├── Video Track (H.264/H.265)
├── Audio Track (AAC)
└── Metadata Track (RTMD - Real Time Metadata)
    ├── Gyroscope samples (X, Y, Z angular velocity in °/s)
    ├── Accelerometer samples (X, Y, Z acceleration in m/s²)
    ├── Lens information (focal length, focus distance)
    └── IBIS/OIS compensation data

The gyroscope samples at ~2000 Hz - far more granular than 24-60 video frames. Tools like Gyroflow use this data for precise video stabilization, and telemetry-parser can extract it. If you noticed the “Ground Truth” bar in the C1250 Rome Airport example earlier, that’s this gyroscope data - and it closely matches what Motion Vector analysis found.

Other Cameras with Embedded IMU

What About iPhones?

iPhones have gyroscopes but Apple doesn’t embed this data in the video file. Third-party apps can record it separately, but that requires syncing.

Most cameras don’t embed gyro data - notably absent: Nikon, Panasonic, Fujifilm, Pentax, Olympus, older Sony, and every smartphone. Motion Vectors work on everything: Sony footage, iPhone videos, stock footage, old vacation videos. If you have a camera with embedded gyro, use it. If not, Motion Vector analysis gives you 95%+ accuracy anyway.

Putting It All Together

This stability analysis is one piece of a larger system for turning 6,000 clips into video essays. The full pipeline:

1. Describe: AI vision models (Qwen, Gemma) watch each clip and generate rich descriptions - not just “people walking” but temporal narratives of what happens when.

2. Analyze: Stability analysis (this essay), sharpness detection, and motion characterization tag each clip with technical metadata.

3. Search: Semantic search lets me find clips by meaning - “peaceful temple scenes in Kyoto” or “Nicole reacting to something amazing.”

4. Write: I write the video essay narrative first. The system suggests clips that match each section, filtered by stability and other constraints.

5. Export: Selected clips export directly to DaVinci Resolve as a timeline, with chapters and markers intact.

The stability analysis solved a specific problem: I kept finding semantically perfect clips that were unusable because they were shaky. Now “find me stable shots of the Colosseum at sunset” actually returns stable shots.

This is one essay in a series on building this system. Future essays will cover AI-powered clip description, narrative construction with semantic search, and the DaVinci Resolve integration.

What I Learned

The encoder already knows. That’s the key insight. Video compression algorithms have been solving motion estimation for decades - not because anyone cared about “stability analysis,” but because predicting where pixels move is fundamental to compression. We’re just reading data that was always there.

This pattern appears everywhere in software: the solution often exists in adjacent systems, computed for entirely different reasons. Database query planners know which columns are selective. Web servers know which endpoints are slow. Compilers know which functions are hot. The challenge is recognizing when your problem maps to data someone else already computed.

Try It Yourself

If you want to experiment with Motion Vector extraction, the key libraries are:

• PyAV for accessing FFmpeg’s codec internals
• sqlite-vec for efficient segment storage and querying
• Any H.264/H.265 video file - Motion Vectors are universal

The core algorithm is simple enough to implement in an afternoon. The edge cases (subject motion, resolution sensitivity, memory leaks at scale) take longer. If you build something interesting or have questions, I’d enjoy hearing about it - my email is in the footer.