# Fansifter Video Clipper

A Python-based video processing engine that converts music videos (any aspect ratio) to vertical 9:16 marketing clips using AI-powered scene detection and intelligent face-tracking reframing.

## Features

- **AI-Powered Scene Detection**: Uses Gemini Flash API to identify high-energy, marketing-relevant segments
- **MediaPipe Detection**: Face mesh + pose estimation for subject tracking; Haar cascade and Canny edge detection as fallbacks
- **Scene Equilibrium Reframing**: Splits each segment into scenes using PySceneDetect, computes a stable equilibrium crop position per scene, with clean cuts at scene boundaries — no panning
- **Frame-by-Frame Processing**: OpenCV + scipy for correct per-frame crop application without FFmpeg expression bugs
- **Temporal Consistency**: EMA + Gaussian multi-pass smoothing prevents crop window jumping within each scene
- **Any Input Format**: Handles videos of any aspect ratio, always outputs 9:16 vertical clips
- **Rotation-Correct Processing**: Reads EXIF/side-data rotation metadata (iPhone portrait MOVs, Android videos) and applies `cv2.rotate()` so all detection and cropping work in display space
- **Multi-Video Cut-to-Music**: Cut-to-music mode accepts additional video sources (files, YouTube URLs, Google Drive links); audio always comes from the main video while visual segments are pooled from all sources and ranked by Gemini score
- **Optimized Processing**: Smart downsampling balances Gemini API token cost with visual quality
- **Mock Mode**: Test cropping without using Gemini API quota (specify segments manually)
- **Debug Overlay Videos**: Generate full-frame videos with rectangle overlays showing exact crop positions
- **Debug Visualization**: Save annotated frames showing crop positions and detection methods
- **Dry-Run Mode**: Preview segment timestamps before processing
- **Timestamped Output**: Each run creates a dated subfolder for organized outputs
- **Caption Overlay**: Burn a static text phrase onto every frame in one of three TikTok-native styles

## Pipeline Architecture

The tool follows a 3-step pipeline:

1. **Intelligence**: Gemini API analyzes the video and identifies N high-energy segments (10-15 seconds each)
2. **Reframing**: Scene-equilibrium algorithm calculates per-scene static crop positions with clean cuts
3. **Execution**: Frame-by-frame OpenCV processing generates final clips with audio preservation

## Installation

### Prerequisites

- Python 3.11.4 or higher
- FFmpeg installed (already at `/opt/homebrew/bin/ffmpeg` on your system)
- Gemini API key (get one at https://ai.google.dev/)
- uv package manager

### Setup

1. All dependencies are already installed via uv

2. Configure your Gemini API key in `.env`:

```bash
GEMINI_API_KEY=your_actual_api_key_here
```

That's it! The project is ready to use.

## Using as a Library in Another Project

The package exposes a full public API importable via `from fansifter_clipper import ...`.

### Install as an editable path dependency (local development)

Add the library to your other project's `pyproject.toml` using a `[tool.uv.sources]` entry:

```toml
[tool.uv.sources]
fansifter-video-clipper = { path = "/absolute/path/to/fansifter-video-clipper/lib", editable = true }
```

Then declare it as a dependency and sync:

```bash
# pyproject.toml — add to [project] dependencies:
#   "fansifter-video-clipper"

uv sync   # or: uv add fansifter-video-clipper
```

Alternatively, from the consuming project's directory:

```bash
uv add --editable /absolute/path/to/fansifter-video-clipper/lib
```

With `editable = true`, uv writes a `.pth` pointer into the consuming project's `.venv` that points directly at this source tree. **No files are copied into the other project's `.venv`.** Any change you make to the library source is immediately visible — no reinstall needed.

### Configuration in the consuming project

The library reads configuration from environment variables or a `.env` file in the **consuming project's working directory** — not from the library's own source tree. Add a `.env` in your project root (or export env vars) with at minimum:

```bash
# your-project/.env
GEMINI_API_KEY=your_key_here

# FFmpeg is auto-discovered via PATH — only set these if ffmpeg is in a non-standard location
# FFMPEG_PATH=/usr/local/bin/ffmpeg
# FFPROBE_PATH=/usr/local/bin/ffprobe
```

All other settings have sensible defaults (see [Configuration](#configuration) for the full list). The library can be imported without any `.env` present; `GEMINI_API_KEY` only needs to be set when Gemini analysis is actually called.

### Importing the pipeline building blocks

```python
from fansifter_clipper import (
    # Top-level orchestrators
    VideoProcessor, ProcessingConfig,
    CutToMusicProcessor, CutToMusicConfig,
    # Data models
    VideoSegment, SyncedClip, ChorusWindow, VisualPeakSegment, AudioOnset,
    VideoSource, CaptionStyle, CaptionPosition,
    # Individual pipeline stages
    IntelligenceEngine,
    CutToMusicIntelligence, OnsetDetector, PeakSyncMapper,
    Reframer, FFmpegExecutor, CutToMusicAssembler,
    # Utilities
    GoogleDriveDownloader,
    probe_video, downsample_video,
)
```

### Driving the pipeline step by step (for a multi-step UI)

Each pipeline is split into named phase methods so an external orchestrator can
display intermediate results, let the user adjust them, and resume:

```python
# ── Clip pipeline ──────────────────────────────────────────────────────────
processor = VideoProcessor(config)

segments = processor.analyze()             # Phase 1: Gemini → [VideoSegment]
# ↑ show segments on a timeline UI, let user drag start/end handles

clips = processor.reframe_and_render(adjusted_segments)  # Phases 2+3: encode

# ── Cut-to-music pipeline ───────────────────────────────────────────────────
processor = CutToMusicProcessor(config)

chorus, visual_peaks, onsets = processor.analyze_only()  # Phases 1+2
# ↑ show chorus window + visual peaks + beat markers on a timeline UI

synced_clips = processor.sync(adjusted_peaks, onsets, chorus)  # Phase 3+3.5
output_path  = processor.render(synced_clips)                   # Phase 4
```

## Usage

### Basic Usage

```bash
# Dry run to preview segments (no processing)
uv run python cli.py harry.mp4 --dry-run

# Generate 3 clips with default settings
uv run python cli.py harry.mp4

# Generate 5 clips to custom directory
uv run python cli.py harry.mp4 -o ./my_clips -n 5

# Custom segment duration (8-12 seconds)
uv run python cli.py harry.mp4 -d 8-12

# Mock mode: Test specific segments without Gemini API
uv run python cli.py harry.mp4 --mock-segments "10-22,35-47,60-72"

# Debug mode: Save visualization frames showing crop positions
uv run python cli.py harry.mp4 --debug

# Debug overlay: Generate full-frame video with crop rectangles
uv run python cli.py harry.mp4 --mock-segments "134-145" --debug
```

### CLI Options

```
uv run python cli.py [OPTIONS] VIDEO_PATH

Options:
  -o, --output-dir PATH           Output directory for clips (default: ./output)
  -n, --num-clips INTEGER         Number of clips to generate (default: 3)
  -d, --duration TEXT             Duration range in seconds (default: 10-15)
  -m, --mock-segments TEXT        Mock segments (bypass Gemini API) in format: start1-end1,start2-end2
  --debug                         Enable debug mode (save visualization frames and overlay videos)
  --dry-run                       Only analyze, don't generate clips
  --caption TEXT                  Static text phrase to burn onto every frame
  --caption-style STYLE           Caption style: viral_hook, aesthetic_label, modern_minimal, tiktok (default: viral_hook)
  --caption-position POSITION     Vertical placement: top, center, or bottom (default: top)
  --help                          Show this help
```

### Example Workflow

1. **Test with dry-run first:**
   ```bash
   uv run python cli.py harry.mp4 --dry-run
   ```
   This will show you the identified segments without processing.

2. **Generate clips (uses Gemini API):**
   ```bash
   uv run python cli.py harry.mp4
   ```
   Clips will be saved to `./output/YYYY-MM-DD_HH-MM-SS/clip_1_s10.0-22.0.mp4`, etc.

3. **Test specific segments without Gemini API (mock mode):**
   ```bash
   uv run python cli.py harry.mp4 --mock-segments "10-22,35-47,60-72"
   ```
   Useful for testing cropping quality without using API quota.

4. **Debug cropping issues:**
   ```bash
   uv run python cli.py harry.mp4 --mock-segments "10-22" --debug
   ```
   Saves annotated frames to `debug_frames/` and generates overlay video `*_overlay.mp4` with crop rectangles.

5. **Verify outputs:**
   ```bash
   ls -lh output/2026-01-22_01-00-22/
   ffprobe output/2026-01-22_01-00-22/clip_1_s10.0-22.0.mp4  # Check dimensions (should be 9:16)
   ```

### Mock Mode (Testing Without Gemini API)

Mock mode allows you to test the cropping and reframing logic without consuming Gemini API quota:

1. **Find segment timestamps** (use dry-run or check output filenames):
   ```bash
   uv run python cli.py video.mp4 --dry-run
   ```

2. **Test specific segments**:
   ```bash
   # Single segment
   uv run python cli.py video.mp4 --mock-segments "10-22"

   # Multiple segments (comma-separated)
   uv run python cli.py video.mp4 --mock-segments "10-22,35-47,60-72"
   ```

3. **Combine with debug mode** for maximum insight:
   ```bash
   uv run python cli.py video.mp4 --mock-segments "10-22" --debug
   ```

### Caption Overlay

Burn a static text phrase onto every frame of every generated clip using `--caption`. Three styles based on the 2026 TikTok landscape are available:

| Style | Font | Appearance | Background |
|---|---|---|---|
| `viral_hook` | Bebas Neue | White, ALL CAPS, drop shadow | Opaque black rounded rectangle (~85%) |
| `aesthetic_label` | Courier Prime | Cream `#F5F5F5`, lowercase, wide letter-spacing | Full-width solid dark label-tape strip |
| `modern_minimal` | Lora Bold (serif) | White, sentence case | None (transparent) |
| `tiktok` | Inter Bold | White with black outline, as-typed, color emoji support | None (transparent) |

`top` and `bottom` positions land in the TikTok "Action Zone" (40–55% from top), safely above the UI chrome. `center` places the text block in the vertical middle of the frame — the default placement for the `tiktok` style.

```bash
# Viral hook style (default) — opaque box, Bebas Neue
uv run python cli.py video.mp4 --mock-segments "10-22" \
  --caption "NEW SINGLE OUT NOW"

# Aesthetic label — typewriter font, label-tape background
uv run python cli.py video.mp4 --mock-segments "10-22" \
  --caption "out now everywhere" --caption-style aesthetic_label

# Modern minimal — clean serif, no background
uv run python cli.py video.mp4 --mock-segments "10-22" \
  --caption "Available on all platforms" --caption-style modern_minimal

# TikTok native look — Inter Bold, white with black outline, color emoji, centered
uv run python cli.py video.mp4 --mock-segments "10-22" \
  --caption "out now everywhere 🔥" --caption-style tiktok --caption-position center

# Place text at bottom of the action zone instead of top
uv run python cli.py video.mp4 --mock-segments "10-22" \
  --caption "NEW SINGLE OUT NOW" --caption-position bottom
```

Long phrases wrap automatically to fit the safe zone (88% of frame width).

## Project Structure

```
fansifter-video-clipper/lib/
├── src/fansifter_clipper/
│   ├── __init__.py
│   ├── models.py                      # Pydantic data models (clip + cut-to-music)
│   ├── config.py                      # Settings management
│   ├── intelligence.py                # Gemini API integration (clip pipeline)
│   ├── cut_to_music_intelligence.py   # Gemini API integration (cut-to-music)
│   ├── onset_detection.py             # librosa percussive onset detection
│   ├── peak_sync.py                   # Visual peak ↔ audio onset mapping
│   ├── cut_to_music_assembler.py      # Per-clip rendering + FFmpeg assembly
│   ├── cut_to_music_processor.py      # Cut-to-music pipeline orchestrator
│   ├── reframing.py                   # Saliency detection & scene-equilibrium crop paths
│   ├── execution.py                   # Clip generation (frame-by-frame)
│   ├── frame_processor.py             # CropInterpolator + frame I/O
│   ├── ffmpeg_utils.py                # Video probing & downsampling utilities
│   ├── processor.py                   # Clip pipeline orchestrator
│   ├── captioning.py                  # Caption overlay renderer
│   └── downloader.py                  # YouTube download support
├── tests/
│   ├── test_ffmpeg_utils.py
│   ├── test_scene_detection.py
│   └── test_scene_integration.py
├── cli.py                  # CLI entry point (clip + cut-to-music subcommands)
├── .env                    # Your API keys (not committed)
├── .env.example            # Template for environment variables
├── pyproject.toml          # Project dependencies
├── PROJECT_STRUCTURE.md    # Detailed technical reference
└── README.md               # This file
```

## Cut-to-Music Mode

A second top-level mode that assembles a **beat-synced 14-18s highlight reel** from a music video. Unlike the `clip` pipeline which generates N independent clips, this mode assembles a single output video where each clip's most impactful frame lands exactly on a percussive audio beat — the editorial technique used in professional TikTok/Reels teasers.

### Usage

```bash
# Full run (4 clips, default settings)
uv run python cli.py cut-to-music harry.mp4

# Custom clip count
uv run python cli.py cut-to-music harry.mp4 -n 5 -o ./reels

# Dry run: see chorus + visual peaks without generating output
uv run python cli.py cut-to-music harry.mp4 --dry-run

# With text overlay
uv run python cli.py cut-to-music harry.mp4 --caption "OUT NOW"

# Debug: keep intermediate files, save overlay videos
uv run python cli.py cut-to-music harry.mp4 --debug

# Multi-video: use UGC footage for visuals, audio from harry.mp4
uv run python cli.py cut-to-music harry.mp4 -e backstage.mp4 -e crowd.mp4

# Multi-video: use only the extra sources (no visuals from the main video)
uv run python cli.py cut-to-music harry.mp4 -e backstage.mp4 --video-source extras

# Multi-video: extra source from YouTube
uv run python cli.py cut-to-music harry.mp4 -e "https://youtu.be/..."
```

### CLI Options

```
uv run python cli.py cut-to-music [OPTIONS] VIDEO_INPUT

Arguments:
  VIDEO_INPUT                     Path to video file or YouTube URL

Options:
  -o, --output-dir PATH           Output directory (default: ./output)
  -n, --num-segments INTEGER      Number of clips to sync to beats (default: 4, range: 2-8)
  -e, --extra-video TEXT          Additional video (file path, YouTube URL, or Google Drive URL)
                                  to draw visual segments from. Can be repeated for multiple sources.
                                  Audio is always taken from VIDEO_INPUT regardless.
  --video-source [all|main|extras]
                                  Which videos contribute visual segments:
                                  'all' = main + extras (default), 'main' = main video only,
                                  'extras' = extra videos only
  --dry-run                       Analyze video and show chorus + visual peaks, don't generate output
  --debug                         Keep intermediate files, save overlay videos
  --caption TEXT                  Static text phrase to overlay on every frame
  --caption-style STYLE           Caption style: viral_hook, aesthetic_label, modern_minimal, tiktok
  --caption-position POSITION     Vertical placement: top, center, or bottom
  --help                          Show this help
```

### Algorithm: Semantic Peak-Sync

The pipeline runs in 5 phases:

**Phase 1 — Gemini Analysis (parallel)**

All videos are downsampled and uploaded to the Gemini Files API concurrently. Then structured-output queries run in parallel:
- **Chorus query** (main video only): identifies the most energetic 15-20s section (`chorus_start`, `chorus_end`)
- **Visual peaks query** (one per video source): identifies N high-motion segments per video, each with a `visual_peak_timestamp` — the exact climax frame (dancer's foot landing, camera whip completing, flash peak)

Each `VisualPeakSegment` is tagged with its source video. After all queries complete, segments from all sources are merged, sorted by Gemini score, and the top N are selected. Distribution across videos is score-driven, not forced.

**Phase 2 — Local Onset Detection (librosa)**

Extracts the chorus audio window to WAV and detects percussive transients:
- `librosa.onset.onset_strength(aggregate='median')` — robust onset envelope
- `librosa.onset.onset_detect(backtrack=True)` — shifts each onset to the preceding energy trough so cuts land *on* the transient
- Greedy spacing filter: keeps only onsets ≥ 2s apart
- Returns N onsets sorted by time

**Phase 3 — Peak-Sync Mapping**

Pairs each visual peak segment to an audio onset (positional 1:1 match, both sorted ascending by time). For each pair:

```
slot_N          = onset_{N+1}.time − onset_N.time   (one inter-beat gap; last slot mirrors preceding)
gap_before_N    = slot_N / 2                          (half-slot before visual peak)
gap_after_N     = slot_N / 2                          (half-slot after visual peak)
source_start_N  = visual_peak_timestamp − gap_before
source_end_N    = visual_peak_timestamp + gap_after
output_position = onset_N.time − onset_0.time          (clip N starts at beat N in output)
speed_factor    = clip_duration / slot_N  [applied only if within ±10%]
```

Each clip is exactly one beat interval long. The cut (clip transition) lands on the beat;
the visual peak is centred within the slot, surrounded by build-up and aftermath.

**Phase 3.5 — Scene Boundary Snapping**

Gemini timestamps carry ±0.5–1s imprecision (it processes video at ~1fps). A PySceneDetect pass probes a ±2s window around each `source_start` and snaps it to the nearest frame-accurate scene cut, eliminating stray frames from the preceding shot at the start of a clip. The `source_end` and onset alignment are preserved.

**Phase 4 — Render + Assembly**

- Each clip is reframed to 9:16 using the same Scene Equilibrium algorithm as the `clip` pipeline, reading from its tagged source video (not necessarily the main input)
- Rotation metadata (iPhone portrait MOVs: `side_data rotation: -90`) is automatically corrected in OpenCV before detection and cropping
- Intermediate clips are muted and normalized to CFR 30fps/yuv420p; all clips are scaled to a common resolution before concat (required when sources have different native heights)
- A single FFmpeg concat command assembles all clips with audio from the main video's chorus section

### Output

A single `cut_to_music.mp4` file in a timestamped subfolder:
```
output/
└── 2026-01-22_15-30-00/
    └── cut_to_music.mp4   # ~14-18s, 9:16, chorus audio
```

---

## Technical Details

### Gemini API Configuration

- **Model**: gemini-3-flash-preview (latest, fastest, most cost-effective)
- **Structured Output**: Uses `response_schema` for type-safe JSON responses
- **Downsampling**: Intelligently reduces to 720p @ 24fps (balances quality vs tokens)
- **Token Savings**: ~60-70% reduction while maintaining visual fidelity

### Detection Backend

**MediaPipe** is the single detection backend:
- **Face Mesh**: Detects nose tip as center point; selects best face by 60% centrality + 40% size weighting
- **Pose Estimation**: Falls back to shoulder midpoint when no faces detected
- **Haar Cascade**: Falls back to frontal face detection when MediaPipe unavailable
- **Canny Edge Saliency**: Final fallback — Gaussian-blurred edge map with center-biased spatial weighting

### Reframing Algorithm: Scene Equilibrium

The only crop mode is `SCENE_EQUILIBRIUM`. Each video segment is processed as follows:

**Step 1: Scene Splitting**
- PySceneDetect (adaptive detector) splits the segment into independent scenes
- Minimum scene length: 1.0s (configurable via `PYSCENEDETECT_MIN_SCENE_LENGTH`)
- Falls back to a single equilibrium pass if no scene changes are detected

**Step 2: Single-Pass Detection**
- MediaPipe runs once across the entire segment, collecting `Detection` objects at regular intervals
- Each detection carries `(center_x, center_y, confidence, method)`

**Step 3: Per-Scene Equilibrium**
For each scene:
1. Filter detections to the scene's time range
2. Compute weighted median of subject positions (confidence-weighted) → equilibrium `(x, y)`
3. Generate keyframes at `SALIENCY_KEYFRAME_INTERVAL` intervals using the equilibrium position with a rubber-band safe-zone: crop stays at equilibrium unless the subject drifts outside the inner 60% of frame, then nudges toward subject
4. Insert clean-cut boundary keyframes at scene transitions (`scene_change=True`)

**Step 4: Smoothing**
- Per-scene temporal smoothing: EMA (α=0.3) → Gaussian filter (2s window) → cubic spline densification
- Scene boundaries are not smoothed across — each scene is processed independently

**Step 5: Frame-by-Frame Output**
- `CropInterpolator` maintains separate linear interpolators per scene segment
- At `scene_change` boundaries, a new interpolator segment begins (clean cut, no blending)
- Each frame is read, cropped at the computed position, and piped directly to FFmpeg stdin for single-pass encoding with audio mux — no intermediate temp file

### Key Architectural Properties

| Property | Value |
|---|---|
| Crop mode | `scene_equilibrium` (only) |
| Detection | MediaPipe → Haar → Canny edges |
| Interpolation within scene | Linear (constant per scene since all keyframes share equilibrium) |
| Cut at scene boundary | Clean (separate interpolator segment) |
| MotionStyle / motion analysis | Removed |
| ViNet / PyTorch | Removed |
| Lookahead refinement | Removed |

### Output Organization

```
output/
├── 2026-01-22_00-53-31/
│   └── clip_1_s5.0-17.0.mp4
└── 2026-01-22_01-00-22/
    ├── clip_1_s10.0-22.0.mp4
    ├── clip_2_s35.0-47.0.mp4
    └── clip_3_s60.0-72.0.mp4
```

**Filename Format**: `clip_{number}_s{start}-{end}.mp4`
- Example: `clip_1_s10.0-22.0.mp4` = Clip 1, from 10.0 to 22.0 seconds
- Makes it easy to reuse segments: `--mock-segments "10-22,35-47,60-72"`

### Debug Visualization

When using `--debug`:

**Debug Frames** (saved to `debug_frames/`):
- Green rectangle: crop window position
- Red crosshair: crop center
- Timestamp + crop coordinates label
- Detection method label

**Debug Overlay Video** (`*_overlay.mp4`):
- Full original frame with green rectangle overlay showing crop position
- Identical crop positions as the actual output clip
- Useful for validating equilibrium quality and scene boundaries

## Configuration

Edit `.env` to customize:

```bash
# Gemini API
GEMINI_API_KEY=your_key_here
GEMINI_MODEL=gemini-3-flash-preview

# Processing defaults
DEFAULT_OUTPUT_DIR=./output
DEFAULT_NUM_CLIPS=3
DEFAULT_DURATION_MIN=10.0
DEFAULT_DURATION_MAX=15.0

# Downsampling (for Gemini API submission only; 2fps + 480p = 5-10x smaller uploads)
MAX_RESOLUTION_HEIGHT=480
MAX_FPS=2

# FFmpeg paths
FFMPEG_PATH=/opt/homebrew/bin/ffmpeg
FFPROBE_PATH=/opt/homebrew/bin/ffprobe

# Reframing
SALIENCY_KEYFRAME_INTERVAL=1.0  # Sample keyframes every N seconds
SMOOTHING_WINDOW=2.0            # Gaussian smoothing window in seconds

# Scene Change Detection
ENABLE_SCENE_DETECTION=true
SCENE_HISTOGRAM_THRESHOLD=0.45
SCENE_CONFIDENCE_DROP_THRESHOLD=0.35
SCENE_PIXEL_DIFF_THRESHOLD=0.25
SCENE_DETECTION_MIN_INTERVAL=0.5
SCENE_DETECTION_DEBUG=false

# PySceneDetect
PYSCENEDETECT_DETECTOR=adaptive    # "adaptive" or "content"
PYSCENEDETECT_ADAPTIVE_THRESHOLD=3.0
PYSCENEDETECT_MIN_SCENE_LENGTH=1.0

# Equilibrium tuning
EQUILIBRIUM_SAFETY_MARGIN=0.2        # Inner safe zone (0.2 = 60% of frame)
EQUILIBRIUM_MAX_MOVEMENT=80          # Max px nudge per keyframe
EQUILIBRIUM_SAMPLE_INTERVAL=0.5      # Detection sample interval (seconds)
EQUILIBRIUM_RUBBER_BAND_STRENGTH=0.3 # Pull-back strength toward equilibrium
```

## Code Quality

```bash
uv run ruff check src/ cli.py
PYTHONPATH=src uv run pytest tests/ -v
```

## Troubleshooting

### "No video stream found"
- Ensure input video is a valid MP4 file
- Check with: `ffprobe harry.mp4`

### "Video processing failed"
- Gemini API may be rate-limited — check your API key in `.env`
- Use mock mode to test without API: `--mock-segments "10-22"`

### "Crop window is off or missing subject"
1. Enable debug mode:
   ```bash
   uv run python cli.py video.mp4 --mock-segments "10-22" --debug
   ```
2. Review the overlay video (`*_overlay.mp4`) to see crop positions with full context
3. Check detection method labels in debug frames — if showing `edges` frequently, MediaPipe may be struggling (extreme angles, heavy occlusion)
4. Tune smoothing in `.env`:
   ```bash
   SMOOTHING_WINDOW=3.0
   SALIENCY_KEYFRAME_INTERVAL=0.5
   ```
5. Adjust scene detection sensitivity: lower `PYSCENEDETECT_ADAPTIVE_THRESHOLD` for more scene splits, higher for fewer

### "Gemini API quota exceeded"
- Use mock mode: `--mock-segments "start1-end1,start2-end2"`
- Run dry-run first to identify segments: `--dry-run`

### "Audio out of sync"
- Shouldn't happen — audio is extracted from the original via FFmpeg and re-muxed
- Report if encountered

## Performance

Expected processing time (3-minute video):
- **Dry-run**: ~10-15 seconds (Gemini analysis only)
- **Mock mode**: ~30-45 seconds (frame-by-frame processing + encoding)
- **Full processing**: ~45-75 seconds
- **With --debug**: ~2x (generates both cropped and overlay videos)

## Current Limitations

- **Sequential Reframing**: MediaPipe detection is not thread-safe, so segments are reframed one at a time; clip encoding runs in parallel and overlaps with reframing via a producer-consumer pipeline
- **MediaPipe Face Detection**: Best for frontal/near-frontal faces; may miss extreme angles (>45°) or heavy occlusion
- **Scene Equilibrium Only**: Single crop strategy — no subject-chasing or segment-wide equilibrium modes
- **Rotation**: Only `side_data_list` rotation and legacy `tags.rotate` are read; non-standard containers that encode rotation differently may not be corrected automatically

## Docker / ECS Fargate Deployment

### Required system packages

```dockerfile
RUN apt-get update && apt-get install -y --no-install-recommends \
    ffmpeg \          
    libgl1 \          
    libglib2.0-0 \    
    libsm6 \          
    libxext6 \        
    libxrender1 \     
    libgomp1 \        
    libsndfile1 \     
    && rm -rf /var/lib/apt/lists/*
```

| Package | Required by |
|---|---|
| `ffmpeg` | FFmpeg + FFprobe (video processing, audio extraction, concat) |
| `libgl1` | OpenCV — `libGL.so.1` |
| `libglib2.0-0` | OpenCV — `libglib-2.0.so.0` |
| `libsm6`, `libxext6`, `libxrender1` | OpenCV X11 stubs (required even in headless mode) |
| `libgomp1` | OpenMP — MediaPipe and OpenCV parallel ops |
| `libsndfile1` | soundfile / librosa audio I/O |

### Environment variables

```bash
GEMINI_API_KEY=your_key_here      # required
FFMPEG_PATH=/usr/bin/ffmpeg       # Linux path — NOT /opt/homebrew/bin/ffmpeg
FFPROBE_PATH=/usr/bin/ffprobe
```

### Installing the library in Docker when the consuming project uses uv path dependency

Your frontend project's `uv.lock` encodes the absolute local path to this library
(e.g. `/Users/you/.../fansifter-video-clipper/lib`). That path doesn't exist inside a
container, so a bare `uv sync` will fail. The fix in both options below is the same:
install the library separately first, then tell uv to skip it when syncing the rest.

Font files are bundled inside the wheel (`fansifter_clipper/assets/fonts/`) so a regular
(non-editable) install works fine in Docker — no source copy required.

#### Option A — Copy library source into the image (simplest)

Use when you want the same pattern as local development or when you don't have a CI
wheel-build step. Set the Docker build context to the common parent that contains both
repos:

```dockerfile
# Assumes build context is the directory containing both projects
COPY fansifter-video-clipper/lib /opt/fansifter-clipper
RUN pip install /opt/fansifter-clipper   # non-editable is fine; fonts are inside the package

COPY my-frontend-app /app
WORKDIR /app
# Sync all remaining locked deps, skip the library (already installed above)
RUN uv sync --no-install-package fansifter-video-clipper
```

#### Option B — Pre-build a wheel and copy it (cleaner image)

Use in CI pipelines where you want no source code from the library in the final image.
Build the wheel once outside Docker:

```bash
cd fansifter-video-clipper/lib && uv build --wheel
# produces dist/fansifter_video_clipper-0.1.0-py3-none-any.whl
```

Then in the Dockerfile:

```dockerfile
COPY dist/fansifter_video_clipper-0.1.0-py3-none-any.whl /tmp/
RUN pip install /tmp/fansifter_video_clipper-0.1.0-py3-none-any.whl

COPY my-frontend-app /app
WORKDIR /app
RUN uv sync --no-install-package fansifter-video-clipper
```

### Sample Dockerfile (Option A)

```dockerfile
FROM python:3.11-slim

RUN apt-get update && apt-get install -y --no-install-recommends \
    ffmpeg \
    libgl1 \
    libglib2.0-0 \
    libsm6 \
    libxext6 \
    libxrender1 \
    libgomp1 \
    libsndfile1 \
    && rm -rf /var/lib/apt/lists/*

# Install the library (fonts are bundled in the package — non-editable works fine)
COPY fansifter-video-clipper/lib /opt/fansifter-clipper
RUN pip install /opt/fansifter-clipper

# Install the consuming application and its remaining locked deps
COPY my-frontend-app /app
WORKDIR /app
RUN uv sync --no-install-package fansifter-video-clipper

ENV GEMINI_API_KEY=""
ENV FFMPEG_PATH=/usr/bin/ffmpeg
ENV FFPROBE_PATH=/usr/bin/ffprobe
```

### Architecture note (x86_64 vs ARM64)

ECS Fargate supports both `x86_64` and `arm64` (Graviton). MediaPipe 0.10.21
ships prebuilt wheels for both architectures. Pin your base image architecture
to match your task definition (`--platform linux/amd64` or `linux/arm64`) so
Docker doesn't silently pull the wrong wheel during the build.

## License

POC project for The Orchard/DataScience team.