# Monitoring During an Accounting Run

What to watch, when, and what the signals mean.

## Validation by Phase

Quick reference mapping [end-to-end flow](../end-to-end.md) phases to validation queries in this document.

| Phase | Description | Validation Section |
|-------|-------------|-------------------|
| Phase 1.3 | Sales file ingested into Snowflake | [After Sales Ingest](#after-sales-ingest) |
| Phase 2.2 | Sales data added to accounting period | [After Sales File Created](#after-sales-file-created) |
| Phase 3.1 | Pre-calculation check | [Pre-Calculation: Contract Size Check](#pre-calculation-contract-size-check) |
| Phase 3.2 | Calculation running | [Airflow DAGs to Monitor](#airflow-dags-to-monitor) |
| Phase 3.2 (legacy) | Legacy data load | [Legacy Run: After Data Load](#legacy-run-after-data-load) |
| Phase 3.2 (legacy) | Legacy calculation | [Legacy Run: After Calculation](#legacy-run-after-calculation) |
| Phase 3.2 (legacy) | Currency validation | [Legacy Run: Currency Validation](#legacy-run-currency-validation) |
| Phase 4.1 | Post-run DBT refresh | [Dashboards: During Active Runs](#during-active-runs) |

## Dashboards

### Always-On (Check During On-Call Handoff)

| Dashboard | What It Shows | URL |
|-----------|--------------|-----|
| **Abacus Health & Performance** | Service health, request latency, error rates | [Datadog](https://sonymusic-pde.datadoghq.com/dashboard/zcg-zpw-r9z/abacus-health--performance-dashboard) |
| **Business Dashboard** | Executive-level accounting metrics | [Sigma](https://app.sigmacomputing.com/sony-music-entertainment/workbook/Abacus-Executive-Dashboard-6x1niJrHQxOB6UqHpGOBDm) |
| **SAP Contract Syncs** | OA → Abacus account sync status | [Snowflake](https://app.snowflake.com/sme/orchard/w27SZHF1Q3RW) |
| **Adjustment Files** | Manual adjustment file health | [Snowflake](https://app.snowflake.com/sme/orchard/#/abacus-adjustments-d3Oud3J7B) |

### During Active Runs

| Dashboard | When to Watch | URL |
|-----------|--------------|-----|
| **prod-art-relations RDS** | During legacy runs (load/unload, calculation) — CPU, IO, connections | [Datadog](https://app.datadoghq.com/dashboard/czh-59d-w7i/prod-art-relations-rds-timeboard) |
| **RDS Monitors** | Alerts for art-relations cluster | [Datadog](https://app.datadoghq.com/monitors/manage?q=prod-art-relations) |
| **Load/Unload Logs** | Legacy data load scripts (python-accounting-run-utils) | [Datadog Logs](https://app.datadoghq.com/logs?saved-view-id=3115880) |
| **Airflow UI** | DAG execution progress and task status | AWS Console → MWAA → `[env]-abacus-airflow` |

### During Legacy Runs (ActiveMQ)

| What to Check | Where | What's Normal |
|--------------|-------|---------------|
| **Messages Enqueued** | ActiveMQ web console | Increases after publishAccountingQueue runs |
| **Messages Dequeued** | ActiveMQ web console | Increases as processAccountingQueue workers consume |
| **Number of Consumers** | ActiveMQ web console | Should match number of running process scripts |
| **Memcached GET hit %** | [Elasticache console](https://us-east-1.console.aws.amazon.com/elasticache/home?region=us-east-1#/memcached/prod-cron-accounting) | Should rise over time as vendor data is cached; if flat, something is wrong |
| **Maxwell CPU** | [Datadog monitors](https://sonymusic-pde.datadoghq.com/monitors/manage?q=prod-maxwells%20cpu%20) | Mute for 12 hours before starting calculation |

## Airflow DAGs to Monitor

| DAG Name | Triggered By | Duration (Approx) |
|----------|-------------|-------------------|
| `sales_ingest` | Jenkins job: `ows-abacus-event-create-ingest-sales-event` | ~20 min per 100M rows |
| `sales_get_eligible` | Abacus UI: "Get Eligible Sales" | Minutes |
| `sales_approve` | Abacus UI: "Approve All Sales" | Minutes |
| `accounting_period_mechanicals` | Abacus UI: "Prep Mech Deductions" | Minutes |
| `accounting_run_calculate` | Abacus UI: "Create" on a run controller | Varies by data volume |
| `accounting_run_commit` | Abacus UI: "Approve" on a run controller | Minutes |

### How to View a DAG

1. Go to [AWS MWAA Console](https://us-east-1.console.aws.amazon.com/mwaa/home?region=us-east-1#environments)
2. Click `[env]-abacus-airflow`
3. Click **Airflow UI** in the Details section
4. Find the DAG in the list
5. Click into the DAG run to see task-level status
6. Click a failed task → **Logs** tab for error details

## Validation Queries

Run these to verify data integrity at key checkpoints.

### After Sales Ingest

```sql
-- Verify data loaded into staging
SELECT COUNT(*)
FROM ROYALTY_ACCOUNTING.<ENV>.STMT_DB_SALES_DISTRO_STAGING
WHERE BATCH_ID = '<batch_id>';
-- Expected: non-zero, matches Finance's row count

-- Verify match against a known run controller
SELECT COUNT(*)
FROM ROYALTY_ACCOUNTING.<ENV>.STMT_DB_SALES_DISTRO_STAGING AS sd
    INNER JOIN ORCHARD_APP_REPORTING_V2.PROD_ROYALTY_ACCOUNTING_ROYALTY_ACCOUNTING.ACCOUNT_CONTRACT AS ac
        ON sd.label_id = ac.account_id
    INNER JOIN ORCHARD_APP_REPORTING_V2.PROD_ROYALTY_ACCOUNTING_ROYALTY_ACCOUNTING.RUN_CONTROLLER_CONTRACT AS rcc
        ON rcc.contract_id = ac.contract_id
    INNER JOIN ORCHARD_APP_REPORTING_V2.PROD_ROYALTY_ACCOUNTING_ROYALTY_ACCOUNTING.RUN_CONTROLLER AS rc
        ON rcc.run_controller_id = rc.run_controller_id
WHERE sd.BATCH_ID = '<batch_id>'
    AND rc.run_controller_name = '<controller_name>';
-- Expected: high percentage of staged data matches
-- NOTE: The ORCHARD_APP_REPORTING_V2.PROD_* tables are replicated from prod.
-- For QA validation, this cross-environment join is intentional because contract data lives in prod.
```

### After Sales File Created

```sql
-- MySQL (royalty_accounting)
SELECT * FROM sales_file WHERE accounting_period_id = <period_id>;

-- Snowflake
SELECT * FROM orchard_app_reporting_v2.<ENV>_royalty_accounting_royalty_accounting.sales_file
WHERE file_name = '<file_display_name>';
```

### Legacy Run: After Data Load

```sql
-- Validate dig_sales_detail row counts (search Datadog logs for validate_dig_sales_detail.py)
-- Expected output format:
-- Result: {'total_qty': 345927020318, 'total_amt': 101537342.791003, 'total_rows': 443372803}
-- Cross-check total_qty/total_amt against checksum file, total_rows against StatementDB export
```

### Legacy Run: After Calculation

```sql
-- Check for unprocessed records
SELECT COUNT(*) FROM TEMP_dig_sales_processed WHERE processed = 'N';
-- Expected: 0 (after all batches processed)

-- Check max row_id vs total row count
SELECT MAX(row_id) FROM accountingflat.TEMP_dig_sales_statements;
-- If > total row count, need additional batches

-- Check for duplicate processing
SELECT statement_detail_id FROM FACTS.PROD.STAGING_FACT_SALES
WHERE period_id = <period_id>
GROUP BY statement_detail_id HAVING COUNT(*) > 1;
-- Expected: 0 results. If duplicates exist, STOP and ask in #accounting-run-priv

-- Check for manual adjustments since prep started
SELECT * FROM art_relations.manual_adjustment WHERE date_added IS NOT NULL ORDER BY date_added DESC;
-- If recent entries exist, ask in #accounting-calc before proceeding with vendor accounting
```

### Legacy Run: Currency Validation

```sql
-- Check for missing exchange rates
CALL art_relations.sp_get_missing_exchange_rates(<period_id>);
-- Check for NULLs in: orig_to_payout_rate, payout_to_usd_rate, usd_to_payout_rate
-- Any NULLs are blockers — stop and post in #accounting-run-priv
```

### Pre-Calculation: Contract Size Check

Use this to identify contracts that may cause timeout issues:

```sql
SELECT
  run_controller_id, rcc.contract_id, ct.contract_term_id,
  SUM(
    COALESCE(NULLIF(JSON_LENGTH(ctc.conditions->'$.stores'), 0), 1)
    * COALESCE(NULLIF(JSON_LENGTH(ctc.conditions->'$.countries'), 0), 1)
    * COALESCE(NULLIF(JSON_LENGTH(ctc.conditions->'$.transaction_types'), 0), 1)
    * COALESCE(NULLIF(JSON_LENGTH(ct.attachments), 0), 1)
  ) / 1000000 AS total_rows_millions
FROM run_controller_contract rcc
JOIN contract_term ct ON ct.contract_id = rcc.contract_id
JOIN contract_term_condition ctc ON ctc.contract_term_id = ct.contract_term_id
WHERE ct.deleted_at IS NULL AND ctc.deleted_at IS NULL
GROUP BY run_controller_id, rcc.contract_id, ct.contract_term_id
HAVING total_rows_millions > 5
ORDER BY 4 DESC;
-- Any contracts over 5M rows are high risk for timeout
```

## Alerts and PagerDuty

| Alert Source | Email | Slack |
|-------------|-------|-------|
| **Accounting PagerDuty** | `accounting-email.ulwvrgqs@theorchard.pagerduty.com` | Alerts route to `#accounting-tech` |

### What to Do When Paged

1. Check PagerDuty for the alert details
2. Look at the [Health Dashboard](https://sonymusic-pde.datadoghq.com/dashboard/zcg-zpw-r9z/abacus-health--performance-dashboard) for systemic issues
3. If during an active run, check the relevant Airflow DAG for failed tasks
4. If a legacy run, check ActiveMQ consumer count and RDS dashboard
5. Post status in `#accounting-tech`
6. See [Troubleshooting](troubleshooting.md) for common failure patterns