---
name: dd-analyze-trace
description: Analyze a Datadog APM trace for performance problems, including time budget, OWS endpoint breakdown, and N+1 call detection. Use when given a Datadog trace ID or a trace JSON file path to investigate latency or performance issues.
---

# dd-analyze-trace skill

Analyze a Datadog APM trace for performance problems, including time budget,
OWS endpoint breakdown, and **N+1 call detection**.

## Trigger

Invoked as `/dd-analyze-trace` with either:
- A **trace ID** (numeric string): download then analyze
- A **trace JSON file path** (already saved): analyze directly

## Script Location

This skill bundles Python scripts in the `scripts/` subdirectory next to this SKILL.md file.
Set `<skill_dir>` to the directory containing this file (resolve from the SKILL.md path).
All `python <skill_dir>/scripts/...` commands below use this path.

## Prerequisites

For download mode, these env vars must be set:
- `DD_API_KEY`
- `DD_APP_KEY`
- `DD_SITE` (e.g. `datadoghq.com`)

## Workflow

### Step 0 — Ask for output folder

Use `AskUserQuestion`:

> "Where should I save the trace data? Enter a folder name (or press Enter to use `<trace_id>` as the default)."

Use the answer as `<output_dir>`. If the user gives no input, default to the trace ID (or the input file's basename without extension for file-mode).

All output will be organized under this folder:
```
<output_dir>/
├── full_<trace_id>.json      ← combined span tree
├── trace_analysis.txt        ← performance diagnosis
└── span_data/
    ├── page_1.json
    ├── page_2.json
    └── ...
```

---

### Step 1 — Acquire the trace file

**If given a trace ID** (looks like a long number, e.g. `7892843842668647916`):

```bash
python <skill_dir>/scripts/dd_trace.py <trace_id> --json --output-dir <output_dir> [--from <time>]
```

`--from` defaults to `7d`. Use `30d` for older traces.

This saves:
- `<output_dir>/span_data/page_N.json` — raw API pages
- `<output_dir>/full_<trace_id>.json` — combined span tree (use this for Step 2)

**If given a file path**: use it directly in Step 2. Still set `<output_dir>` from Step 0.

---

### Step 2 — Analyze

```bash
python <skill_dir>/scripts/dd_analyze.py <output_dir>/full_<trace_id>.json --output-dir <output_dir> [--top N] [--ows-only] [--n1-threshold N]
```

| Flag | Default | Purpose |
|------|---------|---------|
| `--top N` | 20 | Number of slowest spans to show |
| `--ows-only` | off | Skip header/global spans; show OWS analysis only |
| `--n1-threshold N` | 5 | Min call count to flag as N+1 candidate |

The script prints the analysis **and** saves it to `<output_dir>/trace_analysis.txt` automatically.

---

### Step 3 — Interpret and report

Read the four output sections:

#### TIME BUDGET BY SERVICE
- Identifies dominant services by total accumulated time and max single-span duration
- Flag any service consuming >50% of trace wall-clock time

#### OWS ENDPOINTS
- Lists every OWS HTTP endpoint with call count, total time, avg, max
- High call counts on single-record GET endpoints are N+1 suspects

#### N+1 CANDIDATES
```
SERVICE          RESOURCE                              CALLS    TOTAL      AVG   BATCH?  CALLER RESOLVER
ows-pdp          GET /identity/{uuid}/roles/             159   231.1s    1.45s      NO   tenantProfileRoles
ows-users        GET /users/identity/{param}             916   27.31s   29.81ms     NO   internalIdentities
```

For each candidate:
- **CALLS** ≥ threshold: repeated per-item call pattern confirmed
- **BATCH? NO**: no batch/dataloader endpoint found in the trace — fix needed
- **CALLER RESOLVER**: the GraphQL resolver driving the loop

Fix recommendation template:
> `<resolver>` in graphql-user calls `<service> <endpoint>` individually per item.
> Add a dataloader that batches by `[id list]` into a new `POST /endpoint/batch` or equivalent.
> Expected saving: `<count> × <avg>` = `<total>`.

#### TOP N SLOWEST SPANS
- Individual outlier spans (timeouts, slow DB queries, etc.)
- Look for spans where one instance is vastly slower than avg from the OWS section

---

### Step 4 — Output a structured diagnosis

Present findings in this format:

```
## Trace Analysis: <resource> (<service>)
**Trace duration**: X.Xs  |  **Total spans**: N

### N+1 Problems (fix these first)
1. **<resolver>** → `<service> <endpoint>` called **N times** (total Xs, avg Xms)
   - No batch endpoint exists. Add a dataloader.
   - Estimated saving: Xs

### Dominant Services
- `<service>`: X% of trace time (Xs total)

### Other Slow Spans
- ...
```

`trace_analysis.txt` was already written to `<output_dir>/` by the script in Step 2.

---

### Step 5 — Present interactive span selection (loop)

Run:

```bash
python <skill_dir>/scripts/dd_analyze.py <output_dir>/full_<trace_id>.json --spans-json --leaf-only --top 15
```

`--leaf-only` restricts results to spans with no children — the actual bottleneck calls, not GraphQL/Tornado wrappers that merely enclose them. The longest leaf span is always rank 1.

Parse the JSON and present a numbered menu:

```
## Choose a span to investigate

  1. [ows-grass-public  ]  7.37s   tornado.request   GET /graphql-router/graphql
  2. [ows-account       ]  6.80s   flask.request     GET /vendor/<vendor_id>
  3. [graphql-account   ]  6.82s   graphql.execute   vendor(vendorId: ...)
  ...

Which span would you like to dig into? (enter a number, or 0 to skip)
Ctrl-C to quit.
```

Use `AskUserQuestion` to collect the user's choice. If `0` or no valid selection, end the session. Otherwise proceed to Step 6.

---

### Step 6 — Identify service and framework

From the selected span:
- **Service**: `span.service` (e.g., `ows-account`)
- **Framework**: inferred from `span.operation`:
  - `flask.request` → Flask
  - `fastapi.request` → FastAPI
  - `tornado.request` → Tornado
  - `graphql.resolve` or `graphql.execute` → GraphQL resolver layer
- **Route**: `span.resource` (e.g., `GET /vendor/<vendor_id>`)

---

### Step 7 — Ask for repo path

Use `AskUserQuestion`:

> "Where is the source code for `<service>`? Please provide the absolute path to the repo."

---

### Step 8 — Code analysis

Using Read, Grep, and Glob on the provided path, perform the following:

**a) Find the route handler**

| Framework | Search pattern |
|-----------|---------------|
| Flask     | `@*.route` or `add_url_rule` matching the path |
| FastAPI   | `@router.get/post/put/delete` matching the path |
| Tornado   | `url_patterns` or `Application(` with the handler class |

Strip Flask path params (e.g., `<vendor_id>` → search for `/vendor/`) to find the right file.

```bash
# Example Grep
Grep pattern: "vendor" glob: "**/*.py"
# Then narrow by route decorator
```

**b) Read the handler and its callees**

Read the handler function, then follow imports and helper calls 1–2 levels deep if they seem relevant to the slow operation.

**c) Look for these anti-patterns**

- **N+1 queries**: a loop (`for ... in ...`) containing ORM queries or HTTP client calls
- **Missing ORM optimization**: Django ORM without `.select_related()` / `.prefetch_related()` when accessing related models
- **Sequential HTTP calls**: multiple `requests.get()` / `httpx.get()` calls that could be parallelized with `asyncio.gather` or a batch endpoint
- **Missing caching**: repeated calls to the same function/endpoint with the same ID — no `@cache`, `redis.get`, or memoization
- **Blocking I/O in async context**: `requests` (sync) called inside `async def`

**d) Output findings**

Report per finding:
```
### Issue: <pattern name>
File: <path>:<line>
Code:
  <relevant snippet (≤10 lines)>
Fix: <concrete recommendation>
Estimated saving: <derived from trace avg × count if N+1>
```

Conclude with a priority-ordered fix list.

**e) Save code analysis to file**

Derive a safe filename from the span resource (replace spaces and `/` with `_`, strip special chars):
- e.g. `GET /vendor/<vendor_id>` on `ows-account` → `ows-account_GET_vendor_code_analysis.md`

**IMPORTANT**: Save to the trace output directory — the same folder as `trace_analysis.txt` and `full_<trace_id>.json` — NOT inside the repo path.

```
<output_dir>/<span_filename>_code_analysis.md
```

For example, if `<output_dir>` is `~/traces/16378788727480980175`, the file must be written to:
```
~/traces/16378788727480980175/ows-users_GET_users_identity_code_analysis.md
```
Never write it inside the repo directory the user provided in Step 7.

Tell the user the file was saved, showing the full path under `<output_dir>`.

---

### Step 9 — Loop back to span selection

After saving the code analysis, go back to **Step 5** and present the span menu again:

```
Code analysis saved. Ready for next span.

## Choose a span to investigate
  1. ...
  2. ...

Which span would you like to dig into? (enter a number, or 0 to skip)
Ctrl-C to quit.
```

Continue until the user enters `0` or cancels.

---

## Examples

```bash
# Download trace 7892843842668647916 and analyze
/dd-analyze-trace 7892843842668647916

# Analyze already-saved file
/dd-analyze-trace my_trace.json

# Lower threshold to catch smaller N+1s
/dd-analyze-trace my_trace.json --n1-threshold 3

# Only OWS analysis (skip header and global slow spans)
/dd-analyze-trace my_trace.json --ows-only
```