<!--
Visual QC with Gemini Vision
Part of the Bles Software agent skills repository:
https://www.bles-software.com/claude-skills

Install for Claude Code:
  mkdir -p ~/.claude/skills/visual-qc-with-gemini-vision
  mv visual-qc-with-gemini-vision.md ~/.claude/skills/visual-qc-with-gemini-vision/SKILL.md

Install for Codex:
  mkdir -p ~/.codex/skills/visual-qc-with-gemini-vision
  mv visual-qc-with-gemini-vision.md ~/.codex/skills/visual-qc-with-gemini-vision/SKILL.md

For Gemini CLI and other agents, drop the body of this file into the
project instruction file the agent already reads.
-->

---
name: visual-qc-with-gemini-vision
description: Automated visual QC for AI-generated graphics (Remotion compositions, banner ads, slides, branded cards) using Gemini Vision. Capture last frame post-render, ask the vision model to check for layout breakage (overlaps, cutoffs, font failures, missing logos), retry only on real issues. Pairs with the logo-from-settings pipeline (DB settings → planner → renderer → public/ static file).
---

# Automated Visual QC for AI-Generated Graphics

## When to use this skill

You're rendering branded graphics in a pipeline that mixes:
- Multiple text fields with different lengths (Hebrew, Arabic, English mixed)
- Variable assets (logos with different aspect ratios)
- Layout-driven absolute positioning (Remotion, Canvas, web→PNG)

These pipelines silently produce broken output:
- Phone number wraps to 3 lines and the last line is hidden
- Logo is huge or stretched
- Text lands behind a button
- Font fails to load and renders as boxes

A human-in-the-loop catches these on test runs but not in production at scale. **Gemini Vision can be the QC reviewer.**

## The pattern (4 steps)

### 1. Capture the last frame after render

```python
def extract_last_frame(video_path, out_jpg):
    dur = subprocess.run(["ffprobe", "-v", "quiet", "-show_entries",
                          "format=duration", "-of", "csv=p=0", str(video_path)],
                         capture_output=True, text=True)
    duration = float(dur.stdout.strip() or 5.0)
    last_t = max(0.1, duration - 0.3)  # 0.3s before end so we catch the final state
    subprocess.run([
        "ffmpeg", "-y", "-ss", str(last_t), "-i", str(video_path),
        "-vframes", "1", "-q:v", "2", str(out_jpg)
    ], capture_output=True, timeout=30)
```

For static images (PNG/JPG), skip this step and feed the file directly.

### 2. Build a checklist-style QC prompt

Generic "does this look good?" gets vague answers. **Enumerate the expected elements and known failure modes**:

```python
qc_prompt = f"""You are a visual QA reviewer for a Hebrew business advertising outro card.

Expected elements (these should ALL be visible and look clean):
- Business name (Hebrew, large): "{props['businessName']}"
- Tagline (Hebrew, smaller): "{props['tagline']}"
- Phone number (LTR digits, large): "{props['phoneNumber']}"
- Address (Hebrew with 📍 emoji): "{props['address']}"
- CTA pill button (bottom): "{props['ctaText']}"
- {'Logo image at top' if props.get('logoSrc') else 'A circle with the first letter at top'}

Common issues to look for:
- Text cut off, overlapping, or hidden behind other elements
- Phone number cropped or unreadable
- Logo broken (missing, distorted, wrong aspect ratio, way too big/small)
- Overflow off screen
- Background or animations covering critical text
- RTL/LTR mixing breaking layout
- Empty white/black space where elements should be
- Text rendered as boxes (font failed to load)

Return JSON ONLY:
{{
  "ok": true/false,
  "issues": "one short sentence describing problems (or empty string if ok)",
  "severity": "none" | "minor" | "major" | "broken"
}}"""
```

Two crucial details:
- **List the EXPECTED text content verbatim** — the model can verify each string actually appears
- **Enumerate failure modes** — primes the model to look for them specifically

### 3. Send to Gemini with temperature 0

```python
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
with open(last_frame, "rb") as f:
    img_bytes = f.read()
resp = client.models.generate_content(
    model="gemini-2.5-flash",  # Flash is enough for QC; cheap + fast
    contents=[types.Content(role="user", parts=[
        types.Part.from_text(text=qc_prompt),
        types.Part.from_bytes(data=img_bytes, mime_type="image/jpeg"),
    ])],
    config=types.GenerateContentConfig(temperature=0.0))

text = resp.candidates[0].content.parts[0].text.strip()
if text.startswith("```"):  # strip markdown fences defensively
    text = re.sub(r'^```(?:json)?\s*', '', text)
    text = re.sub(r'```\s*$', '', text)
result = json.loads(text)
# {"ok": true, "issues": "", "severity": "none"}
```

Temperature 0 + structured prompt = stable, deterministic JSON.

### 4. Threshold-based retry (the most important rule)

**Do NOT retry on every issue.** Gemini will find "minor" cosmetic issues even in clean output (e.g. flagging the user's actual logo as "the logo has small text on it"). If you retry on every issue, you'll throw away good output.

```python
severity = result.get("severity", "").lower()
issues = result.get("issues", "").lower()

# Only retry on MAJOR or BROKEN
if severity in ("major", "broken"):
    # Targeted retry: if issue is logo-specific, drop the logo and retry text-only
    if "logo" in issues and logo_present:
        props["logoSrc"] = None  # fallback
        re_render(props)
    else:
        # Generic retry — could try smaller font, different layout, etc.
        # OR just surface the issue to humans for review (don't auto-iterate forever)
        log_for_human_review(result)
elif severity == "minor":
    # Log it but ship it. Minor cosmetic complaints aren't worth a re-render.
    print(f"  [QC] minor issue (kept as-is): {issues}")
else:
    print(f"  [QC] ✓ clean")
```

The 4 severity levels:
- **`none`** — looks clean, ship it
- **`minor`** — cosmetic detail not worth re-rendering (small spacing, minor font weight). Log it, ship it.
- **`major`** — readability/comprehension affected (text obscured, logo broken). Retry with targeted fix.
- **`broken`** — feature missing entirely (no phone, no CTA, blank screen). Retry or surface to humans.

## Real example — Big Burger reel outro

**v1 render**: phone number was 130pt and wrapped to 3 lines. The last 4 digits ("1818") got hidden behind the CTA button below.

**Gemini QC verdict**:
```json
{
  "ok": false,
  "issues": "The last four digits of the phone number (1818) are obscured by the CTA button, making the full number unreadable.",
  "severity": "major"
}
```

The QC caught the EXACT issue. Retry with logo dropped wasn't appropriate (issue was layout, not logo), so I logged it and fixed the layout manually: phone font 130pt→90pt, added `whiteSpace: nowrap`, `maxWidth: 90vw`. Re-rendered.

**v2 render**: layout fixed but Gemini flagged the logo image's own internal text:
```json
{
  "ok": false,
  "issues": "The logo image contains small, unreadable text that duplicates the address and phone number, making the logo cluttered.",
  "severity": "minor"
}
```

This is a **false positive** — the user's actual logo design has small text in it. We can't fix the user's logo; that's just what it looks like. The fix: severity threshold = retry only on `major`/`broken`, ship `minor` as-is.

**v3 render**: `severity: none, issues: (none) — ✓ clean` ✅

## The companion: logo-from-settings pipeline

Many QC failures come from missing logos because the asset path didn't propagate from the database to the renderer. Wire it like this:

```
DB: settings.companyLogo = { url, diskPath, filename, ... }
       ↓
Service: pulls settings.companyLogo into business context
       ↓
Planner (LLM): produces creative content; the system fills in companyLogo
       ↓
Renderer (Python): copies diskPath into Remotion's public/ dir,
                   passes filename to logoSrc prop
       ↓
Remotion: <Img src={staticFile(logoSrc)} />
```

Gotcha: Remotion's `staticFile()` resolves relative to the project's `public/` directory. You can't pass a `/opt/...` absolute path. Always copy the asset into `public/` per render and pass just the filename.

Logo fallback in the component:
```tsx
{logoSrc ? (
  <Img src={staticFile(logoSrc)} style={{ maxWidth, maxHeight, objectFit: "contain" }} />
) : (
  <div className="circle-with-letter">{businessName.charAt(0)}</div>
)}
```

## Reference implementation

`/opt/autoposter/reels-engine/generate_from_plan.py`:
- `render_business_outro()` — copies logo to Remotion public/, passes filename
- `qc_check_outro()` — extracts last frame, sends to Gemini with checklist prompt
- threshold-based retry inside `render_business_outro()`

`/opt/autoposter/reels-engine/remotion/outro-card/src/BusinessOutro.tsx` — composition with logo+fallback pattern

## When to use Flash vs Pro for QC

- **Gemini 2.5 Flash** — perfect for QC. Fast (1-2s), cheap, accurate enough to catch obvious layout breakage. Use this by default.
- **Gemini 2.5 Pro / 3.x** — overkill for QC. Save for content generation, not validation.

The QC budget should be ~10% of the render budget. Don't spend more on checking than on rendering.

## Quick checklist before shipping a QC system

- [ ] Last-frame extraction is robust (handles short videos, missing duration)
- [ ] QC prompt enumerates EXPECTED text content verbatim (not just "check the layout")
- [ ] QC prompt enumerates KNOWN failure modes (overlaps, cutoffs, font fails)
- [ ] Output is structured JSON with `severity` field (4 levels)
- [ ] Retry threshold = `major`/`broken` only, NEVER `minor`
- [ ] Targeted retry strategies per issue type (logo issue → drop logo; layout issue → log, surface to humans)
- [ ] Defensive JSON parsing (strip markdown fences)
- [ ] Temperature 0 for QC calls (deterministic)
- [ ] Cheap model (Flash) for QC (<$0.001 per frame)

## Why this works

- **Vision models are good at "is this broken?"** even when generation models can't reliably produce non-broken output
- **Closed feedback loop** — render → QC → log/retry — catches issues that would otherwise hit users
- **Threshold-based retry** prevents the QC from killing good output (the most common failure mode of naïve auto-iteration)
- **Cheap to run** — Flash + 1 frame = ~$0.0005 per QC check

## Anti-pattern to avoid

**Don't ask Gemini "rate this from 1-10".** Numerical ratings are noisy and inconsistent. Use enum severity (`none|minor|major|broken`) — it's more stable across runs and easier to threshold.