# dd-endpoint-root-services

Estimate the **root entry services per endpoint** for a Datadog APM service —
the leftmost (upstream) column of the APM Resource-page **Dependency Map**,
reconstructed programmatically.

For each of a service's server endpoints (resources), it samples traces and
records the service at the **root** of each trace, producing:

```json
{ "GET /product/<int:product_id>": { "graphql-switchboard": 71, "ows-coda": 12, ... } }
```

## Why it's an estimate

There is no API that returns this data. Datadog's own Dependency Map is UI-only,
is "only available for service-entry span resources," and is itself **built from
a sample of ingested spans**. This tool replicates that nature by sampling
traces per endpoint and reading each trace's root span — so the output is an
estimate, clearly labelled `"estimate": true`.

`--traces-per-endpoint` is the confidence knob: **raise it until the root set
stops growing.** That convergence is your signal that the sample is large enough.

**Sizing the sample** — match it to the endpoint's traffic so low-rate callers
aren't missed:

| Endpoint traffic | `--traces-per-endpoint` | `--spread-buckets` |
|------------------|-------------------------|--------------------|
| High (a caller dominates) | 500–1000 | 24–48 |
| Low / steady | 50–100 | 8–12 (or `--no-spread`) |

Bigger numbers catch more low-traffic callers; the trade-off is **cost and 429s**
— more traces × more buckets = more API calls, so the run is slower and more
likely to be rate-limited (it backs off and retries automatically).

**Spread sampling is on by default** (`--spread-buckets`, default 24): traces are
pulled evenly across the whole window rather than just the most recent, which is
what surfaces low-rate or bursty long-tail roots. Pass **`--no-spread`** for
faster recent-only sampling (it biases toward the dominant recent caller and can
miss those roots entirely). Some upstreams only reach the service across async
hops (kinesis/SQS) that break trace propagation; those appear as the async
consumer (e.g. `lambda-…`) rather than the true origin, no matter the sample
size.

## Requirements

**Python ≥ 3.9** (uses `argparse.BooleanOptionalAction`). No third-party
packages and no install step — the script uses only the standard library
(`urllib`, `argparse`, `json`, `datetime`, …), the same stdlib-only pattern as
the other skills in this repo. Just run it with your system Python:

```bash
python3 --version          # confirm 3.9+
```

(If you prefer an isolated interpreter, `python3 -m venv .venv && source
.venv/bin/activate` works, but there's nothing to `pip install`.)

The only inputs are the credential environment variables below.

## Usage

```bash
export DD_API_KEY=... DD_APP_KEY=... DD_SITE=datadoghq.com

# Default: all endpoints, prod, last 30d, 240 traces/endpoint, spread across 24 buckets
python scripts/dd_root_entry_services.py ows-product

# Verify field paths + cheap smoke test (first endpoint only, dumps a raw event)
DD_DEBUG=1 python scripts/dd_root_entry_services.py ows-product --from now-7d --dry-run

# Higher confidence, busiest 5 endpoints, save JSON
python scripts/dd_root_entry_services.py ows-product \
  --max-endpoints 5 --traces-per-endpoint 500 --out roots.json

# Just one endpoint — skip enumeration entirely (spread sampling applies by default)
python scripts/dd_root_entry_services.py ows-product \
  --method GET --path "/product/<int:product_id>"

# Faster recent-only sampling (may miss low-rate / older upstream roots)
python scripts/dd_root_entry_services.py ows-product \
  --method GET --path "/product/<int:product_id>" --no-spread

# Read the endpoint list straight from the source (Flask or FastAPI)
python scripts/dd_root_entry_services.py ows-product \
  --handlers /Users/tcalhoun/work/ows-product/product/handlers.py
```

### Flags

| Flag | Default | Purpose |
|------|---------|---------|
| `service` (positional) | — | Datadog service name (e.g. `ows-product`) |
| `--env` | `prod` | Environment tag |
| `--from` / `--to` | `now-30d` / `now` | Window: `now-30d`, epoch-ms, or ISO 8601 |
| `--traces-per-endpoint` | `240` | Sample size per endpoint (confidence knob) |
| `--path` (+ `--method`) | — | Look up a single endpoint; skips enumeration (method default `GET`) |
| `--handlers FILE` | — | Read endpoints from a Flask (`@app.route`) or FastAPI (`@router.get`) handlers file; skips enumeration |
| `--spread` / `--no-spread` | on | Sample across the window (default) vs recent-only |
| `--spread-buckets N` | `24` | Number of sub-windows to spread across |
| `--low-count-threshold N` | `2` | Warn to verify roots sampled ≤ N times (0 disables) |
| `--auto-shrink` / `--no-auto-shrink` | on | Split & retry a batch that overruns the span cap (avoids `<no spans>`) |
| `--max-endpoints` | all | Only the top N endpoints by volume |
| `--batch-size` | `10` | trace_ids per OR-query (`1` = simple per-trace fetch) |
| `--entry-filter` | auto | Override server-entry span filter |
| `--dry-run` | off | Process only the first endpoint |
| `--csv` | off | `endpoint,root_service,count` instead of JSON |
| `--out FILE` | — | Also write output to FILE |
| `DD_DEBUG=1` (env) | off | Verbose logs + dump one raw event for field-path checks |

## How it works (Option B — sampling)

1. **Enumerate endpoints** — `/api/v2/spans/analytics/aggregate`, grouped by
   `resource_name`, filtered to server-entry spans. Probes `span.kind:server`,
   falls back to `operation_name:flask.request OR fastapi.request OR tornado.request`.
   *Or skip this step* with `--handlers` (read route decorators from Flask
   `@app.route` or FastAPI `@router.get`/`@app.post` source) / `--path` (one
   endpoint) — handy when you know the endpoints and want to avoid the
   retention-limited aggregate query.
2. **Collect trace_ids** — page `/api/v2/spans/events/search` for each endpoint
   up to `--traces-per-endpoint` (spread evenly across the window by default to
   de-bias recency; `--no-spread` takes just the most recent).
3. **Resolve roots** — batch trace_ids into `trace_id:(a OR b …)` queries, group
   returned spans by trace_id, and pick the root (span whose `parent_id` is
   empty / `"0"` / not present among the trace's span IDs); record its `service`.
4. **Aggregate** — `{ endpoint: { root_service: count } }`, emitted as JSON.

## Cost

Roughly:

```
1 aggregate call
+ per-endpoint search pages (ceil(traces-per-endpoint / 1000))
+ endpoints × ceil(traces-per-endpoint / batch-size) trace-resolution queries
```

`--batch-size` is the main lever to cut the dominant trace-resolution cost.
When a batch's combined spans overrun the cap (`BATCH_SPAN_CAP`, 20k), `--auto-shrink`
(on by default) splits it and retries the halves — a few extra queries on the
rare cap trip, in exchange for never dropping a large trace as `<no spans>`.

## Caveats

- **Sampled, not exact** — same nature as Datadog's own map. `<root not in
  sample>` / `<no spans>` rows are expected at low sample sizes.
- **Retention** — the v2 Spans Search API only sees raw spans within ingestion
  retention (often ~15 days). A `now-30d` window may return little for older
  data; the script warns when every endpoint comes back empty. Try `now-7d`.
- **Root accuracy** — when a trace's true root isn't sampled, the topmost
  available span is treated as the root, so its service is reported instead.
- **Low-count roots are real but low-confidence** — they're kept in the output,
  but the script warns (count ≤ `--low-count-threshold`, default 2) that you
  should verify them against the Datadog dependency map. Inferred names
  (`requests`, `kinesis`, `sqs`) and `lambda-…` consumers mark uninstrumented or
  async boundaries the trace can't cross.