---
name: dd-endpoint-root-services
description: Estimate the root entry services for each endpoint of a Datadog-instrumented service — the leftmost (upstream) column of the APM Resource-page Dependency Map. For every server endpoint (resource), sample traces and record the service at the root of each trace. Use when reconstructing per-endpoint upstream dependencies, finding which services ultimately originate traffic to an endpoint, or programmatically approximating the UI dependency graph.
---

# dd-endpoint-root-services skill

Reconstruct, per endpoint, the **root entry services** that originate traces
flowing through a Datadog APM service — i.e. the leftmost/upstream column of
the Resource-page **Dependency Map**.

For a service's `GET /item/<int:item_id>`, the map's start column shows the
services that *originate* the traces flowing through that endpoint — e.g. an
API gateway, a couple of upstream microservices, and an async consumer. This
skill produces a `{ endpoint: { root_service: count } }` map that approximates
that upstream column.

Useful for:
- Reconstructing the per-endpoint upstream dependency graph programmatically
- Finding which services ultimately *originate* traffic to an endpoint (full
  trace walk to the root, not just the immediate caller)
- Auditing upstream blast radius before a change to an endpoint

> **This is an estimate.** No API returns this data directly. Datadog's own map
> is UI-only and is itself built from a *sample* of ingested spans. We replicate
> that by sampling traces per endpoint and reading each trace's root span. Raise
> `--traces-per-endpoint` until the root set stops growing — that's the
> convergence / confidence signal.

> **Distinct from `dd-endpoint-callers`.** That skill answers "who calls this
> endpoint *directly*" (one hop up — the immediate caller). This skill walks all
> the way to the **trace root** to find the originating service.

## Trigger

Invoked as `/dd-plugins:dd-endpoint-root-services` with a service name and
optional flags, e.g.:

```
/dd-plugins:dd-endpoint-root-services <service>
/dd-plugins:dd-endpoint-root-services <service> --env prod --from now-7d
/dd-plugins:dd-endpoint-root-services <service> --traces-per-endpoint 200
/dd-plugins:dd-endpoint-root-services <service> --method GET --path "/item/<int:item_id>"
```

## Script Location

Scripts live in the `scripts/` subdirectory next to this SKILL.md.
Set `<skill_dir>` to the directory containing this file.

## Prerequisites

These env vars must be set:
- `DD_API_KEY`
- `DD_APP_KEY`
- `DD_SITE` (e.g. `datadoghq.com`)

## Workflow

### Step 0 — Resolve parameters

Collect from the user's invocation (or ask with `AskUserQuestion` if missing):
- **service**: the Datadog service name (e.g. `<service>`)
- **env**: environment tag (default `prod`)
- **window**: `--from` / `--to` (default `now-30d` → `now`). Accept `now-30d`,
  epoch-ms, or ISO 8601.
- **traces-per-endpoint**: sample size (default `240`)

Ask for an output folder:
> "Where should I save the results? (Press Enter to print to stdout only)"

Use the answer as `<output_dir>`. If blank, output goes to stdout only.

---

### Step 1 — Verify field paths (first run only)

Before trusting a large run, do a cheap smoke test that also dumps a raw event
so the API's field nesting is confirmed against a live response:

```bash
DD_DEBUG=1 python <skill_dir>/scripts/dd_root_entry_services.py <service> \
  --env <env> --from now-7d --to now --dry-run
```

Confirm the printed raw `data[0]` has `attributes.trace_id`, `attributes.span_id`,
`attributes.parent_id`, `attributes.service`, `attributes.resource_name` where
the script reads them. (These vary across orgs; the dump is the safeguard.)

---

### Step 2 — Run the full map

```bash
python <skill_dir>/scripts/dd_root_entry_services.py <service> \
  --env <env> \
  --from <from> --to <to> \
  --traces-per-endpoint <N> \
  --out <output_dir>/root_entry_services.json
```

Optional flags from the user's invocation:

| User intent | Script flag | Purpose |
|-------------|-------------|---------|
| One endpoint only | `--path "<path>" [--method GET]` | Look up a single endpoint, skip enumeration |
| Endpoints from source | `--handlers <file>` | Read routes from a Flask (`@app.route`) or FastAPI (`@router.get`) handlers file |
| Recent-only sampling | `--no-spread` | Disable spread; sample just the most recent traces |
| Tune spread | `--spread-buckets N` | Number of sub-windows to spread across (default 24) |
| Low-count warning | `--low-count-threshold N` | Warn to verify roots sampled ≤ N times (default 2; 0 off) |
| Limit scope | `--max-endpoints N` | Only the top N endpoints by volume |
| Cheaper / faster | `--batch-size N` | trace_ids per OR-query (default 10; `1` = per-trace) |
| Span-cap handling | `--no-auto-shrink` | Disable splitting a batch that overruns the span cap (default: split & retry) |
| Custom entry filter | `--entry-filter "<query>"` | Override the server-entry span filter |
| CSV output | `--csv` | `endpoint,root_service,count` instead of JSON |

If no `<output_dir>`, omit `--out` so results print to stdout only.

#### Endpoint sources

By default the script discovers endpoints by aggregating spans (subject to the
retention caveat). When you already know the endpoints, supply them and skip
that step entirely:

- **Single endpoint** — `--path "/item/<int:item_id>" --method GET`
  (method defaults to `GET`). The resource is built as `METHOD path` to match
  Datadog's `resource_name`.
- **From source** — `--handlers /path/to/handlers.py` parses route decorators
  into `METHOD /path` resources. Supports **Flask** (`@app.route(...)`, honoring
  `methods=[...]`) and **FastAPI/Starlette** (`@router.get(...)`, `@app.post(...)`,
  etc.). Flask `<int:id>` and FastAPI `{id}` param syntax both already match
  Datadog's `resource_name`. This is the most accurate endpoint list and avoids
  the aggregate/retention step. Point it at the service's router/handlers module
  (e.g. `/path/to/<service>/routers/foo.py`). Note: if a FastAPI router is mounted
  with a `prefix=`, the parsed paths won't include it — confirm against Datadog's
  resource names, or use `--path` for a single known endpoint.

`endpoint_source` in the JSON envelope records which path was used
(`aggregate` | `explicit` | `handlers:<file>`). If a supplied endpoint returns
0 traces, the script warns it may be a path/method/format mismatch.

#### Recency bias and spread sampling

**Spread sampling is on by default.** The window is divided into buckets
(default 24, tune with `--spread-buckets N`) and a share of traces is pulled
from each, so older or bursty upstream roots aren't crowded out by the dominant
recent caller. The `sampling` field in the envelope records the mode
(`spread/N buckets` vs `recent`).

Use **`--no-spread`** to instead sample only the **most recent**
`--traces-per-endpoint` traces (`sort:-timestamp`) — faster, but on a busy
endpoint low-rate roots may be *entirely absent* from the recent slice, and
raising `--traces-per-endpoint` alone won't surface them.

Note: very-low-rate roots (e.g. `<0.01 req/s`) may still need a large
`--traces-per-endpoint`, and some upstreams only reach the service across async
hops (kinesis, SQS) that break trace propagation — those surface as the async
consumer (e.g. `lambda-…`) rather than the true origin, regardless of sampling.

#### Sizing the sample

Match the sample to the endpoint's traffic so low-rate callers aren't missed:

| Endpoint traffic | `--traces-per-endpoint` | `--spread-buckets` |
|------------------|-------------------------|--------------------|
| High (a caller dominates) | 500–1000 | 24–48 |
| Low / steady | 50–100 | 8–12 (or `--no-spread`) |

The bigger the numbers, the more low-traffic callers you catch. The trade-off
is **cost and rate limits**: more traces × more buckets = more API calls, so the
run is slower and more likely to hit 429s (it backs off and retries
automatically, so it self-heals — just takes longer). If the high-confidence
root set stops growing when you raise the values, you've sampled enough.

---

### Step 3 — Present the results

The JSON envelope has metadata plus an `endpoints` map:

```json
{
  "estimate": true,
  "service": "<service>",
  "env": "prod",
  "from": "2026-05-10T...Z",
  "to": "2026-06-09T...Z",
  "traces_per_endpoint": 240,
  "sampling": "spread/24 buckets",
  "entry_filter_used": "(operation_name:flask.request OR ...)",
  "endpoints": {
    "GET /item/<int:item_id>": {
      "api-gateway": 71,
      "upstream-service-a": 12,
      "upstream-service-b": 9,
      "async-consumer": 5,
      "<root not in sample>": 3
    }
  }
}
```

Summarize per endpoint:

```
## Root Entry Services: <service> (<env>, <from> → <to>)
**Sample**: <N> traces/endpoint  |  **Endpoints**: M  |  *estimate*

### GET /item/<int:item_id>
Root services (by sampled share):
- api-gateway — 71%
- upstream-service-a — 12%
- upstream-service-b — 9%
- ...
```

Interpretation notes:
- **`<root not in sample>`** — the trace's true root span wasn't in the sampled
  spans; the origin couldn't be determined for those traces.
- **`<no spans>`** — no spans returned for that trace_id in the window. With
  `--auto-shrink` (on by default) a batch that overruns the span cap is split
  and retried, so cap-induced drops are eliminated — remaining `<no spans>`
  mean the trace genuinely had no spans in-window (e.g. a retention edge).
- **Low-count roots are real but low-confidence.** Entries sampled only a few
  times (the script warns on count ≤ `--low-count-threshold`, default 2) are
  genuine but under-sampled — surface them to the user and recommend confirming
  them against the Datadog dependency map before treating them as established.
- Inferred/integration names (`requests`, `kinesis`, `sqs`) and async consumers
  (`lambda-…`) indicate the true origin sits across an uninstrumented or async
  hop the trace can't cross.

---

### Step 4 — Convergence check / drill-down

After presenting, offer:

> "Want higher confidence? I can re-run with a larger `--traces-per-endpoint`
> (e.g. 200) and compare — if the root set stops growing, we've converged."

If the user agrees, go back to **Step 2** with a larger sample and diff the root
sets. Stop when new roots stop appearing.

---

## Caveats

- **Sampled, not exact.** Mirrors how Datadog's own map is built. `<root not in
  sample>` rows are inherent to sampling.
- **Retention.** The v2 Spans Search API only sees raw spans within ingestion
  retention (often ~15 days), unlike the UI map's long-retained aggregate. A
  `now-30d` window may return little/nothing for older data — the script warns
  when every endpoint comes back empty. Try `--from now-7d`.
- **Cost.** Roughly `endpoints × traces-per-endpoint ÷ batch-size` trace-fetch
  queries (plus 1 aggregate + per-endpoint search pages). The default
  `--traces-per-endpoint 240` with spread sampling balances coverage and cost
  for a single endpoint; for a full multi-endpoint sweep, lower it or use
  `--max-endpoints` to bound the run.

## Examples

```bash
# Default: all endpoints of <service>, prod, last 30d, 240 traces each,
# spread across 24 buckets
python dd_root_entry_services.py <service>

# Shorter, retention-safe window
python dd_root_entry_services.py <service> --env prod --from now-7d --to now

# Higher confidence for the busiest 5 endpoints, save JSON
python dd_root_entry_services.py <service> --max-endpoints 5 \
  --traces-per-endpoint 500 --out roots.json

# Single endpoint only (no enumeration; spread sampling applies by default)
python dd_root_entry_services.py <service> \
  --method GET --path "/item/<int:item_id>"

# Recent-only sampling (faster; may miss low-rate / older upstream roots)
python dd_root_entry_services.py <service> \
  --method GET --path "/item/<int:item_id>" --no-spread

# Endpoints read from the service's source (Flask or FastAPI)
python dd_root_entry_services.py <service> \
  --handlers /path/to/<service>/routers/foo.py

# CSV for one quick pass, simple per-trace fetches
python dd_root_entry_services.py <service> --batch-size 1 --csv
```