WebCodecs Video Processing: 5 Core Patterns for Native Browser Video Encoding and Decoding

WebCodecs Video Processing: Native Video Codec Power in the Browser

FFmpeg.wasm is large and slow to load, Canvas frame-by-frame processing has poor performance, WebRTC is a black box with no customization — browser video processing has long been constrained. WebCodecs API provides low-level video codec capabilities, supporting H.264/H.265/VP9/AV1, zero-copy raw frame access, with performance approaching native apps. In 2026, WebCodecs is fully supported in Chrome and Edge, making browser video editing and real-time encoding a reality.

This article covers 5 core patterns, guiding you through VideoDecoder → VideoEncoder → frame processing → real-time transcoding → video recording.

---

Core Concepts

Concept	Description
WebCodecs	Browser-native video/audio codec API
VideoDecoder	Video decoder, converting encoded data to raw frames
VideoEncoder	Video encoder, converting raw frames to encoded data
VideoFrame	Video frame object containing pixel data and metadata
EncodedVideoChunk	Encoded video data chunk
VideoColorSpace	Video color space information
codec string	Codec identifier string
hardwareAcceleration	Hardware acceleration configuration

---

Problem Analysis: 5 Major WebCodecs Challenges

1. Low-level API abstraction: Manual frame lifecycle and memory management required
2. Codec support differences: Different browsers/platforms support different codecs
3. Keyframe management: Decoding requires correct keyframe sequences
4. Performance tuning: Hardware acceleration configuration and framerate control
5. Difficult debugging: Codec error messages are not user-friendly

---

Step-by-Step: 5 WebCodecs Patterns

Pattern 1: VideoDecoder Video Decoding

async function decodeVideoStream(data: ArrayBuffer) {
  const decoder = new VideoDecoder({
    output: (frame: VideoFrame) => {
      processFrame(frame);
      frame.close();
    },
    error: (e: Error) => console.error("Decoder error:", e.message),
  });

  decoder.configure({
    codec: "avc1.64001f",
    codedWidth: 1920,
    codedHeight: 1080,
    hardwareAcceleration: "prefer-hardware",
    optimizeForLatency: true,
  });

  const chunk = new EncodedVideoChunk({ type: "key", timestamp: 0, data });
  decoder.decode(chunk);
  await decoder.flush();
  decoder.close();
}

Pattern 2: VideoEncoder Video Encoding

async function encodeFrames(frames: VideoFrame[]) {
  const encodedChunks: EncodedVideoChunk[] = [];

  const encoder = new VideoEncoder({
    output: (chunk: EncodedVideoChunk) => encodedChunks.push(chunk),
    error: (e: Error) => console.error("Encoder error:", e.message),
  });

  encoder.configure({
    codec: "avc1.64001f",
    width: 1920,
    height: 1080,
    bitrate: 5_000_000,
    framerate: 30,
    keyInterval: 30,
    latencyMode: "quality",
    hardwareAcceleration: "prefer-hardware",
  });

  for (let i = 0; i < frames.length; i++) {
    encoder.encode(frames[i], { keyFrame: i % 30 === 0 });
    frames[i].close();
  }

  await encoder.flush();
  encoder.close();
  return encodedChunks;
}

Pattern 3: Real-Time Frame Processing and Filters

class VideoProcessor {
  private canvas: OffscreenCanvas;
  private ctx: OffscreenCanvasRenderingContext2D;

  async init(width: number, height: number) {
    this.canvas = new OffscreenCanvas(width, height);
    this.ctx = this.canvas.getContext("2d")!;
  }

  private onDecodedFrame(frame: VideoFrame) {
    this.ctx.drawImage(frame, 0, 0);
    frame.close();
    const processedFrame = new VideoFrame(this.canvas, { timestamp: frame.timestamp });
    this.encoder.encode(processedFrame);
    processedFrame.close();
  }

  applyGrayscale() {
    const imageData = this.ctx.getImageData(0, 0, this.canvas.width, this.canvas.height);
    const data = imageData.data;
    for (let i = 0; i < data.length; i += 4) {
      const gray = data[i] * 0.299 + data[i + 1] * 0.587 + data[i + 2] * 0.114;
      data[i] = data[i + 1] = data[i + 2] = gray;
    }
    this.ctx.putImageData(imageData, 0, 0);
  }
}

Pattern 4: Video Format Transcoding

async function transcodeVideo(
  inputChunks: EncodedVideoChunk[],
  inputCodec: string,
  outputCodec: string,
  width: number, height: number
): Promise<EncodedVideoChunk[]> {
  const outputChunks: EncodedVideoChunk[] = [];

  const decoder = new VideoDecoder({
    output: (frame) => { encoder.encode(frame); frame.close(); },
    error: (e) => console.error(e),
  });

  const encoder = new VideoEncoder({
    output: (chunk) => outputChunks.push(chunk),
    error: (e) => console.error(e),
  });

  decoder.configure({ codec: inputCodec, codedWidth: width, codedHeight: height });
  encoder.configure({ codec: outputCodec, width, height, bitrate: 4_000_000, framerate: 30 });

  for (const chunk of inputChunks) decoder.decode(chunk);
  await decoder.flush();
  await encoder.flush();
  decoder.close();
  encoder.close();
  return outputChunks;
}

Pattern 5: MediaStream Real-Time Recording

async function recordScreenWithOverlay() {
  const stream = await navigator.mediaDevices.getDisplayMedia({
    video: { width: 1920, height: 1080, frameRate: 30 },
  });

  const track = stream.getVideoTracks()[0];
  const processor = new MediaStreamTrackProcessor({ track });
  const reader = processor.readable.getReader();

  const encoder = new VideoEncoder({
    output: (chunk) => saveChunk(chunk),
    error: (e) => console.error(e),
  });

  encoder.configure({
    codec: "avc1.64001f", width: 1920, height: 1080,
    bitrate: 8_000_000, framerate: 30, keyInterval: 30,
  });

  while (true) {
    const { done, value: frame } = await reader.read();
    if (done) break;
    encoder.encode(frame);
    frame.close();
  }

  await encoder.flush();
  encoder.close();
}

---

Pitfall Guide

Pitfall 1: Not closing VideoFrame causing memory leaks

// ❌ Wrong: frames not closed
// ✅ Correct: close after use
output: (frame) => { processFrame(frame); frame.close(); }

Pitfall 2: Unsupported codec

// ✅ Correct: check support
const support = await VideoDecoder.isConfigSupported({ codec: "avc1.64001f", codedWidth: 1920, codedHeight: 1080 });
if (!support.supported) throw new Error("Codec not supported");

Pitfall 3: First frame not a keyframe

// ✅ Correct: ensure first frame is a keyframe
const chunk = new EncodedVideoChunk({ type: "key", timestamp: 0, data });

Pitfall 4: Encoder bitrate too low

// ✅ Correct: set reasonable bitrate for resolution
encoder.configure({ bitrate: 5_000_000 }); // 5Mbps for 1080p

Pitfall 5: Processing heavy frames on main thread

// ✅ Correct: use Web Worker
const worker = new Worker("video-processor.js");
worker.postMessage({ frame }, [frame]);

---

Error Troubleshooting

#	Error	Cause	Solution
1	NotSupportedError	Codec not supported	Use isConfigSupported to check
2	InvalidStateError	Encoder/decoder state error	Ensure configure before encode/decode
3	DataError	Encoded data corrupted	Check data integrity and alignment
4	OutOfMemoryError	Frames not closed causing memory leak	Call frame.close() after each use
5	EncodingError	Invalid encoding parameters	Check bitrate, resolution, framerate
6	Hardware acceleration unavailable	Hardware acceleration not available	Fallback to software mode
7	Key frame required	Missing keyframe	Ensure first frame and periodic keyframes
8	Timestamp discontinuity	Non-monotonic timestamps	Ensure frame timestamps are monotonically increasing
9	Codec string invalid	Invalid codec string format	Use standard codec strings
10	AbortError	Operation aborted	Check if close() was called

---

Advanced Optimization

1. Web Worker parallel encoding: Move codec operations to Worker to avoid main thread blocking
2. Hardware acceleration first: prefer-hardware config leverages GPU codec
3. Keyframe interval tuning: Adjust keyInterval per scenario, short for live streaming
4. Adaptive bitrate ABR: Dynamically adjust bitrate based on network conditions
5. WebGPU accelerated filters: Use Compute Shader for video frame effects

---

Comparison

Dimension	WebCodecs	FFmpeg.wasm	Canvas frame-by-frame	WebRTC
Codec Performance	⭐⭐⭐⭐⭐	⭐⭐⭐	⭐⭐	⭐⭐⭐⭐
Format Support	⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐	⭐⭐⭐
Customization	⭐⭐⭐⭐⭐	⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐
Load Size	⭐⭐⭐⭐⭐	⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐
Browser Support	⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐⭐⭐
Learning Curve	⭐⭐⭐	⭐⭐⭐⭐	⭐⭐⭐⭐⭐	⭐⭐⭐

---

Summary: WebCodecs gives browsers native-level video codec capabilities, far outperforming FFmpeg.wasm and Canvas solutions. WebCodecs suits applications requiring browser-side video processing, especially video editors, real-time transcoding, and screen recording. With full Chrome/Edge support in 2026, it's the foundation for next-gen Web video applications.

---

Online Tools

• JSON Formatter: /en/json/format
• Hash Calculator: /en/encode/hash
• cURL to Code: /en/dev/curl-to-code