Breaking into the Pipeline: Real-Time Frame Manipulation with the Insertable Streams for MediaStreamTrack API

Understanding the Pipeline

Traditionally, processing live camera streams on the web required painting frames to an offscreen <canvas> at 30 or 60 fps, applying manipulations, and calling canvas.captureStream(). This method introduces memory thrashing, unnecessary color space conversions, and pipeline latency.

The Insertable Streams for MediaStreamTrack API (part of the WebCodecs and Media Capture Transform specifications) bypasses canvas workarounds by exposing low-level frame streams:

• MediaStreamTrackProcessor: Consumes a raw MediaStreamTrack and exposes a ReadableStream yielding discrete VideoFrame chunks.
• TransformStream: Operates on each VideoFrame, allowing direct pixel manipulation, metadata injection, or WebGL/WebGPU-backed transformations.
• VideoTrackGenerator (or MediaStreamTrackGenerator): Serves as a WritableStream that consumes processed VideoFrame objects and outputs a fresh, standard MediaStreamTrack.

This newly minted track feeds straight into video players, WebRTC peer connections, or a standard MediaRecorder.

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>
        Insertable Streams for MediaStreamTrack API Demo</title>
    <link href="style.css" rel="stylesheet" />
    <script src="insertableStreamsforMediaStreamTrack_API.js" defer></script>
</head>
    
<body>
    <main class="container">
        <header>
            <h1>Insertable Streams for MediaStreamTrack</h1>
            <p class="subtitle">Real-time Web Stream frame transformation & recording</p>
        </header>
    
        <section class="viewport-grid">
            <div class="card">
                <h2>Live Transformed Stream</h2>
                <video id="preview" autoplay playsinline muted></video>
            </div>
            <div class="card">
                <h2>Recorded Output</h2>
                <video id="playback" controls playsinline></video>
            </div>
        </section>
    
        <section class="controls">
            <button id="btnStart" class="btn primary">Start Camera & Transform</button>
            <button id="btnRecord" class="btn accent" disabled>Start Recording</button>
            <button id="btnStop" class="btn danger" disabled>Stop Recording</button>
        </section>
    
        <div id="status" class="status-banner">Ready. Click "Start Camera" to initialize.</div>
    </main>
</body>

</html>

Key Gotchas for Production

• Frame Resource Leaks: Every incoming VideoFrame holds reference counters to underlying GPU or system memory buffers. Failing to invoke videoFrame.close() inside the transform cycle will cause memory exhaustion and browser tab termination within seconds.
• Timestamp Matching: Always pass { timestamp: videoFrame.timestamp } when producing the new VideoFrame. Stripping or mismatching timestamps will cause MediaRecorder or WebRTC pipelines to drop frames or produce out-of-sync playback.
• Worker Offloading: For high-resolution (1080p+) or complex workloads, transfer the processor.readable and generator.writable across a Worker thread using standard postMessage([stream], [stream]) to keep the main UI thread at a smooth 60 fps.

:root {
    --bg-primary: #0f172a;
    --bg-card: #1e293b;
    --text-primary: #f8fafc;
    --text-muted: #94a3b8;
    --accent: #0284c7;
    --accent-record: #e11d48;
    --border: #334155;
}

* {
    box-sizing: border-box;
    margin: 0;
    padding: 0;
    font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, Cantarell, sans-serif;
}

body {
    background-color: var(--bg-primary);
    color: var(--text-primary);
    min-height: 100vh;
    display: flex;
    align-items: center;
    justify-content: center;
    padding: 2rem 1rem;
}

.container {
    width: 100%;
    max-width: 900px;
    display: flex;
    flex-direction: column;
    gap: 1.5rem;
}

header {
    text-align: center;
}

header h1 {
    font-size: 1.75rem;
    letter-spacing: -0.025em;
}

.subtitle {
    color: var(--text-muted);
    font-size: 0.95rem;
    margin-top: 0.25rem;
}

.viewport-grid {
    display: grid;
    grid-template-columns: repeat(auto-fit, minmax(320px, 1fr));
    gap: 1.5rem;
}

.card {
    background: var(--bg-card);
    border: 1px solid var(--border);
    border-radius: 8px;
    padding: 1rem;
    display: flex;
    flex-direction: column;
    gap: 0.75rem;
}

.card h2 {
    font-size: 1rem;
    color: var(--text-muted);
}

video {
    width: 100%;
    aspect-ratio: 16 / 9;
    background: #000;
    border-radius: 4px;
    object-fit: cover;
}

.controls {
    display: flex;
    justify-content: center;
    gap: 1rem;
    flex-wrap: wrap;
}

.btn {
    padding: 0.65rem 1.25rem;
    border-radius: 6px;
    font-weight: 600;
    border: none;
    cursor: pointer;
    transition: opacity 0.2s ease;
}

.btn:disabled {
    opacity: 0.4;
    cursor: not-allowed;
}

.btn.primary {
    background-color: var(--accent);
    color: #fff;
}

.btn.accent {
    background-color: var(--accent-record);
    color: #fff;
}

.btn.danger {
    background-color: #475569;
    color: #fff;
}

.status-banner {
    text-align: center;
    font-size: 0.85rem;
    color: var(--text-muted);
    background: rgba(30, 41, 59, 0.5);
    padding: 0.5rem;
    border-radius: 4px;
}

/**
 * Insertable Streams for MediaStreamTrack Demo
 * Hess WebTech (hesswebtech.com)
 *
 * Demonstrates consuming a raw video MediaStreamTrack via MediaStreamTrackProcessor,
 * piping frames through a custom TransformStream for real-time natural overlay processing,
 * and pushing transformed frames into a generator sink to record via MediaRecorder.
 */

// --- DOM References ---
const previewVideo = document.querySelector('#preview');
const playbackVideo = document.querySelector('#playback');
const btnStart = document.querySelector('#btnStart');
const btnRecord = document.querySelector('#btnRecord');
const btnStop = document.querySelector('#btnStop');
const statusBanner = document.querySelector('#status');

// --- State Management ---
let rawStream = null; // Captures direct webcam output from getUserMedia
let processedStream = null; // Holds the final synthesized track containing transformed frames
let mediaRecorder = null; // MediaRecorder instance attached to the processed stream
let recordedChunks = []; // Buffer array collecting encoded WebM data packets

/**
 * Normalizes browser differences between VideoTrackGenerator and MediaStreamTrackGenerator.
 * Returns an object containing the writable sink and standard MediaStreamTrack.
 */
const createTrackGenerator = () => {
	// Spec standard implementation (Chrome 94+)
	if ('VideoTrackGenerator' in window) {
		const generator = new VideoTrackGenerator();
		return {
			writable: generator.writable,
			track: generator.track
		};
	}

	// Pre-standard or alternative naming variant
	if ('MediaStreamTrackGenerator' in window) {
		const generator = new MediaStreamTrackGenerator({ kind: 'video' });
		const track = generator.track || generator;
		return {
			writable: generator.writable,
			track
		};
	}

	throw new Error(
		'Insertable Streams generator is not supported in this browser.'
	);
};

/**
 * Validates user agent support for both readable extraction and track generation.
 */
const verifySupport = () => {
	const hasProcessor = 'MediaStreamTrackProcessor' in window;
	const hasGenerator =
		'VideoTrackGenerator' in window || 'MediaStreamTrackGenerator' in window;

	if (!hasProcessor || !hasGenerator) {
		statusBanner.textContent =
			'Error: Insertable Streams API is not supported in this browser. Ensure Chrome/Edge 94+ or flags enabled.';
		statusBanner.style.color = '#f87171';
		btnStart.disabled = true;
		return false;
	}
	return true;
};

/**
 * Creates the TransformStream that intercepts, stamps with a watermark, and outputs VideoFrames.
 * Retains natural true-color camera rendering without inversion filters.
 */
const createFrameTransformer = () => {
	// Maintain an OffscreenCanvas across transform iterations to avoid per-frame GC allocations
	const offscreen = new OffscreenCanvas(640, 480);
	const ctx = offscreen.getContext('2d', { willReadFrequently: true });

	return new TransformStream({
		transform: async (videoFrame, controller) => {
			const width = videoFrame.displayWidth;
			const height = videoFrame.displayHeight;

			// Dynamically match offscreen canvas dimensions if incoming frame size changes
			if (offscreen.width !== width || offscreen.height !== height) {
				offscreen.width = width;
				offscreen.height = height;
			}

			// 1. Draw raw video frame in natural, true color
			ctx.drawImage(videoFrame, 0, 0, width, height);

			// 2. Render real-time watermark overlay with timestamp
			ctx.fillStyle = '#0284c7';
			ctx.font = 'bold 18px monospace';
			ctx.fillText('HESS WEBTECH | LIVE PIPELINE', 20, 36);

			// 3. Construct new VideoFrame with exact microsecond timestamp preservation
			const newFrame = new VideoFrame(offscreen, {
				timestamp: videoFrame.timestamp
			});

			// Explicitly close input frame handle to prevent memory and GPU leaks
			videoFrame.close();

			// Enqueue the new frame to the stream output controller
			controller.enqueue(newFrame);
		}
	});
};

/**
 * Requests camera access, connects the stream pipeline, and displays preview
 */
const initPipeline = async () => {
	try {
		statusBanner.textContent = 'Requesting camera stream...';

		// Acquire source video track from hardware webcam
		rawStream = await navigator.mediaDevices.getUserMedia({
			video: { width: 640, height: 480 },
			audio: false
		});

		const [videoTrack] = rawStream.getVideoTracks();

		// 1. Instantiate the source processor (turns track into a ReadableStream of VideoFrames)
		const processor = new MediaStreamTrackProcessor({ track: videoTrack });

		// 2. Instantiate the sink generator (provides WritableStream and target MediaStreamTrack)
		const { writable, track: outputTrack } = createTrackGenerator();

		// Guard check to confirm valid MediaStreamTrack extraction
		if (!outputTrack || !(outputTrack instanceof MediaStreamTrack)) {
			throw new Error(
				'Resolved generator object is not a valid MediaStreamTrack.'
			);
		}

		// 3. Connect: Processor -> Custom Transformer -> Generator Sink
		const transformer = createFrameTransformer();
		processor.readable
			.pipeThrough(transformer)
			.pipeTo(writable)
			.catch((err) => {
				console.error('Pipeline processing error:', err);
			});

		// 4. Wrap generated track into a standard MediaStream and assign to DOM
		processedStream = new MediaStream([outputTrack]);
		previewVideo.srcObject = processedStream;

		// Update UI controls
		btnStart.disabled = true;
		btnRecord.disabled = false;
		statusBanner.textContent =
			'Stream connected. Ready to record transformed track.';
		statusBanner.style.color = '#94a3b8';
	} catch (err) {
		statusBanner.textContent = `Initialization Failed: ${err.message}`;
		statusBanner.style.color = '#f87171';
		console.error(err);
	}
};

/**
 * Captures the processed stream via MediaRecorder
 */
const startRecording = () => {
	recordedChunks = [];

	// Prefer VP9 codec if supported; fallback to standard webm container
	const options = MediaRecorder.isTypeSupported('video/webm;codecs=vp9')
		? { mimeType: 'video/webm;codecs=vp9' }
		: { mimeType: 'video/webm' };

	mediaRecorder = new MediaRecorder(processedStream, options);

	// Buffer incoming compressed video packets
	mediaRecorder.ondataavailable = (event) => {
		if (event.data && event.data.size > 0) {
			recordedChunks.push(event.data);
		}
	};

	// Finalize recording and set video source for playback
	mediaRecorder.onstop = () => {
		const blob = new Blob(recordedChunks, { type: 'video/webm' });
		playbackVideo.src = URL.createObjectURL(blob);
		statusBanner.textContent = 'Recording finalized. Ready for playback.';
	};

	mediaRecorder.start();
	btnRecord.disabled = true;
	btnStop.disabled = false;
	statusBanner.textContent = 'Recording transformed stream...';
};

/**
 * Halts active recording and toggles button states
 */
const stopRecording = () => {
	if (mediaRecorder && mediaRecorder.state !== 'inactive') {
		mediaRecorder.stop();
		btnRecord.disabled = false;
		btnStop.disabled = true;
	}
};

// --- Event Listeners ---
btnStart.addEventListener('click', () => {
	if (verifySupport()) initPipeline();
});
btnRecord.addEventListener('click', () => startRecording());
btnStop.addEventListener('click', () => stopRecording());