# HTML Publisher-Based Report Links

## Problem

Report link pills on the dashboard show incorrect stage names. All URLs are labeled with the first E2E stage name regardless of which stage the report belongs to, and skipped (NOT_EXECUTED) stages appear as labels.

### Root Cause

The current URL extraction flow:
1. For red builds, fetch per-stage console logs via `getStageLog` — these rarely contain the report URLs
2. When per-stage extraction finds nothing, fall back to `getConsoleText` (full build console)
3. Assign all extracted URLs to `dbStages[0]` — the first Playwright stage by insertion order, which may be skipped

The `dbStages` query includes all `isPlaywright=1` stages without filtering by status, so `dbStages[0]` can be a `NOT_EXECUTED` stage.

### Evidence from DB

Run 1407 (`frontend-audience-development` #1972):
- 5 E2E stages: `E2E Tests on PR Instance` (NOT_EXECUTED, id=76055), `E2E Permissions Tests on PR Instance` (NOT_EXECUTED), `E2E Tests` (FAILED), `E2E Permissions Tests` (FAILED), `E2E Prod Tests` (FAILED)
- URL assigned to stage 76055 (`E2E Tests on PR Instance`) — NOT_EXECUTED, first by insertion order

## Solution

Replace console text parsing with Jenkins HTML Publisher API as the primary URL source.

All pipelines use the HTML Publisher plugin to attach an HTML page per Playwright suite. Each page contains two links: a Playwright report URL and a Datadog test runs URL. The report name (e.g., `"Playwright Report - QA-Default-Suite"`) uniquely identifies the suite.

### New Data Flow

```
Jenkins /{build}/api/json
  -> filter HTML Publisher actions (reportName, urlName)
  -> fetch each report's HTML page
  -> extract <a href> URLs
  -> store with clean report name as label, stageId = null
```

### Changes

#### 1. JenkinsClient — new methods

**`getHtmlReports(jobPath, buildNumber)`**
- Calls `/{buildNumber}/api/json?tree=actions[_class,urlName,reportName]`
- Filters actions for HTML Publisher class (`htmlpublisher.HtmlPublisherTarget$HTMLBuildAction`)
- Returns `{ reportName: string, urlName: string }[]`

**`getReportHtml(jobPath, buildNumber, urlName)`**
- Fetches `/{buildNumber}/{urlName}/` (the HTML page)
- Note: the existing `request()` method sends `Accept: application/json`. This method must override the Accept header to `text/html` (either via an optional parameter on `request()` or a separate fetch).
- Returns the HTML body as string

#### 2. URL extraction — new function

**`extractUrlsFromHtml(html, reportName)`**
- Parses `<a href>` tags from the HTML using regex (no external dependency needed — the HTML is simple and controlled by us, always in the form `<a href="..." ...>`)
- Only extracts URLs from `<a>` tags with `href` starting with `http` (ignores relative/anchor links, navigation, etc.)
- Derives a clean label by stripping everything up to and including `" - "` from `reportName` (e.g., `"Playwright Report - QA-Default-Suite"` -> `"QA-Default-Suite"`)
- Labels by URL domain:
  - `playwright.*` URL -> `"QA-Default-Suite"`
  - `datadoghq.com` URL -> `"QA-Default-Suite (Datadog)"`
- Returns `{ url: string, label: string }[]`

#### 3. Schema — make stageId nullable

`extracted_urls.stageId`: change from `.notNull()` to nullable (keep the FK reference, just allow null).

This requires a Drizzle migration. Since SQLite does not support `ALTER COLUMN`, the migration will recreate the `extracted_urls` table. Existing rows with non-null `stageId` values do not need data migration — they are valid and will continue to work. The `onDelete: "cascade"` FK behavior is unchanged for rows with non-null stageId; rows with null stageId are unaffected by stage deletions.

#### 4. Poller — replace URL extraction logic

In `insertStagesAndUrls`:
- Remove: `getStageLog` loop for URL extraction, `getConsoleText` fallback, `seenUrls` set, `dbStageByName` map, `dbStages[0]` fallback
- Remove: `urlPatterns` parameter (now unused) from `insertStagesAndUrls`, `pollPipeline`, and `startPoller`
- Add: call `getHtmlReports` -> for each report, call `getReportHtml` -> `extractUrlsFromHtml` -> insert with `stageId: null`
- Only fetch HTML reports for red builds (same condition as current URL extraction)
- Green/yellow builds intentionally show no report links (same as current behavior)

API call count per red build: 1 (`api/json`) + N (`getReportHtml` per report). This is comparable to the current flow (N `getStageLog` calls + 1 `getConsoleText` fallback).

#### 5. API route — adapt queries

In `buildPipelinesResponse` (`pipelines.ts`):
- Change `innerJoin` on stages to `leftJoin` (stageId is now nullable)
- Remove `stageName` from the select and response
- Update `PipelineResponse` interface: remove `stageName` from `extractedUrls` type

In `buildRunsResponse` (`pipelines.ts`):
- No changes needed — it queries `extractedUrls` directly without joining stages. Null `stageId` values are acceptable in its response.

#### 6. Client — simplify rendering

In `PipelineCard.tsx`:
- Change `u.stageName || u.label || "Report"` to `u.label || "Report"`

In `types.ts`:
- Remove `stageName` from `ExtractedUrl` interface

### Tests

- `poller.test.ts`: Rewrite "fetches console text and extracts URLs for red runs" test to mock `getHtmlReports` and `getReportHtml` instead of `getStageLog`. Update `createMockJenkinsClient` with new method stubs.
- `pipelines.test.ts`: Update `extractedUrls` inserts to use `stageId: null`.
- `url-extractor.test.ts`: Add tests for `extractUrlsFromHtml`.
- `jenkins-client.test.ts`: Add tests for `getHtmlReports` and `getReportHtml`.

### What's removed

- Console text URL extraction for report links (`extractUrls` usage in poller)
- Stage-to-URL mapping logic (`dbStageByName`, `seenUrls`, `dbStages[0]` fallback)
- `getStageLog` calls for URL extraction purpose
- `urlPatterns` parameter from `insertStagesAndUrls`, `pollPipeline`, `startPoller`, and config usage in poller
- `stageName` in API response and client types

### What's unchanged

- `extractUrls` function and tests (kept for potential future use)
- `getStageLog` / `getConsoleText` methods on JenkinsClient (may be used elsewhere)
- Stage insertion logic
- `deriveStatus` logic
- All other poller, schema, and client behavior

### Error handling

- If `getHtmlReports` fails: skip URL extraction for this build (same as current behavior when console fetch fails)
- If `getReportHtml` fails for a specific report: skip that report, continue with others
- If HTML parsing finds no `<a href>` tags: store nothing for that report