Skip to content

Add transparent background rendering - #218

Open
bc-anam wants to merge 15 commits into
mainfrom
codex/transparent-background-poc
Open

Add transparent background rendering#218
bc-anam wants to merge 15 commits into
mainfrom
codex/transparent-background-poc

Conversation

@bc-anam

@bc-anam bc-anam commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add transparentBackground client/session option
  • request the server-side green-screen avatar rendition
  • composite chroma-keyed frames into an SDK-managed WebGL canvas
  • expose optional key tuning, renderer diagnostics telemetry, and canvas access

Preview integration

The Lab preview pins the attached deterministic SDK tarball rather than relying on an unpublished npm tag.

Verification

  • npm run build
  • targeted changed-file ESLint passed during artifact creation
  • clean consumer install verified for CommonJS, ESM/type declarations, and UMD

Known POC constraints

  • stream() returns the opaque green source; transparency is canvas-only
  • native video controls/PiP do not capture the composited canvas
  • WebGL is required; unsupported devices fail explicitly

Summary by cubic

Adds transparent avatar background rendering via a WebGL overlay and a packed, premultiplied color+alpha H.264 stream. The packed path uses server‑feathered alpha; the client only clears the decoder black pedestal, clamps RGB to alpha, and calibrates the legacy key to the decoded carrier to avoid a faint veil.

  • New Features

    • Auto-selects packed-alpha-v2 when MediaCapabilities reports supported+smooth in both orientations; otherwise falls back to an adaptive chroma key that auto‑detects green or blue and calibrates to the decoded border. Promotes H.264 to Main Level 4.0 on initial offers and ICE restarts; emits client_transparent_renderer telemetry.
    • WebGL overlay mirrors the video’s object-fit/position, border radius, clip path, transforms, and z-index; getTransparentBackgroundCanvas() returns the composited canvas. Supports vertical and horizontal packed layouts; shuts down the session if the local renderer can’t initialize.
  • Migration

    • Enable with transparentBackground: true, then call streamToVideoElement on a DOM‑attached <video> with autoplay and playsinline.
    • WebGL is required; unsupported devices fail explicitly.
    • The underlying MediaStream remains opaque (packed planes or green‑screen). Native controls/PiP and recordings capture the raw stream—use getTransparentBackgroundCanvas() when you need the transparent canvas.

Written for commit c629603. Summary will update on new commits.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 10 files

Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.

Fix all with cubic | Re-trigger cubic

Comment thread src/modules/TransparentBackgroundRenderer.ts
Comment thread src/modules/TransparentBackgroundRenderer.ts

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Comment thread src/modules/TransparentBackgroundRenderer.ts Outdated
Comment thread src/modules/TransparentBackgroundRenderer.ts Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.

Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

Comment thread src/modules/TransparentBackgroundRenderer.ts

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.

Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

Comment thread src/modules/TransparentBackgroundRenderer.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant