# Accounting Run — End-to-End Flow

This document tells the full chronological story of an accounting run: what happens, in what order, by whom, in which systems, and what depends on what.

## Three Systems

There are three accounting run systems operating in parallel:

| System | Used For | Cadence | Scale |
|--------|----------|---------|-------|
| **Abacus** (new) | Distribution and NR runs for Abacus-managed contracts | Monthly | Millions of rows per run controller |
| **Legacy** (cron-accounting) | Full distribution run across all contracts | Monthly | Hundreds of millions of rows |
| **Publishing** (Snowflake-manual) | Publishing sales via manual adjustments | Quarterly | Smaller; manual Snowflake queries |

The legacy system uses Docker containers, ActiveMQ, PHP scripts, and Aurora MySQL. Abacus uses Airflow DAGs, Snowflake, lambdas, and a web UI. The publishing process is a hybrid of Snowflake SQL and manual coordination.

This document covers the **Abacus** flow primarily. See [Legacy Workflow](runbooks/legacy-workflow.md) and [Publishing Sales](runbooks/publishing-sales.md) for the other two.

---

## Monthly Timeline

```
Week 1      Finance delivers sales data → Sales Processing team prepares files
Week 1-2    Exchange rates uploaded to statement period
Week 2      Sales ingested into Snowflake → Accounting periods opened
Week 2-3    Run controllers executed (calculate → review → approve), one at a time
Week 3      DBT refresh triggered → Customer Accounting reviews data
Week 3-4    "Show to Customer" → second DBT refresh → period goes live
```

Exact timing varies. The critical constraint is that **everything is sequential** — each step depends on the previous one completing.

---

## Phase 1: Pre-Run Setup

### 1.1 Finance Delivers Sales Data

| | |
|---|---|
| **Who** | Finance / Sales Processing team |
| **Where** | StatementDB → S3 |
| **What** | Raw sales transaction files (distribution or NR) are compiled and made available |
| **Depends on** | End of the reporting period |

For details on how sales flow from DSPs through Finance to Abacus: [Sales File Delivery (Whimsical)](https://whimsical.com/sales-file-delivery-overview-CtLQfs3u6Hm1A3jKBjvKMg)

### 1.2 Exchange Rates Uploaded

| | |
|---|---|
| **Who** | Finance |
| **Where** | MySQL `royalty_accounting.exchange_rate` table |
| **What** | Currency exchange rates for the statement period (e.g., CAD→USD, EUR→USD) |
| **Depends on** | Statement period must exist |

These rates are used during calculation to convert between sale currency, USD, and payee currency.

### 1.3 Sales File Ingested into Snowflake

| | |
|---|---|
| **Who** | Accounting engineer |
| **Where** | Jenkins → Airflow DAG `sales_ingest` → Snowflake |
| **What** | Trigger the [`ows-abacus-event-create-ingest-sales-event`](https://scheduler.theorchard.io/job/ows-abacus-event-create-ingest-sales-event/) Jenkins job |
| **Depends on** | Finance has delivered the sales data |
| **Duration** | ~20 minutes per 100M rows |
| **Output** | Data lands in `ROYALTY_ACCOUNTING.<env>.STMT_DB_SALES_DISTRO_STAGING` (or `_NR_STAGING`) |

Parameters: Environment, Sales Type (`distro` or `nr`), Batch ID (unique integer, typically the statement period ID).

---

## Phase 2: Accounting Period Setup (Abacus UI)

All Phase 2 steps happen in the **Abacus web UI** by an **accounting operator**.

### 2.1 Open an Accounting Period

| | |
|---|---|
| **Who** | Accounting operator |
| **Where** | Abacus UI → Accounting Periods |
| **What** | Create a new accounting period for the contract type (distribution or NR) |
| **Depends on** | Statement period exists; no other period open for this contract type |
| **Constraint** | Only one period can be open per contract type at a time |

### 2.2 Add Sales Data to the Period

This is a multi-step process with enforced ordering:

```
Step A: "Get Eligible Sales"     → DAG: sales_get_eligible
  ↓ (must complete first)
Step B: "Mark All Sales Delivered"
  ↓
Step C: "Approve All Sales"      → DAG: sales_approve
  ↓ (must complete first)
Step D: "Prep Mech Deductions"   → DAG: accounting_period_mechanicals
```

| | |
|---|---|
| **Who** | Accounting operator |
| **Where** | Abacus UI → Accounting Period detail page |
| **Depends on** | Sales ingest (Phase 1.3) must be complete before Step A |
| **Failure mode** | Triggering "Get Eligible Sales" before ingest completes will error and block the run. See [Troubleshooting: Get Eligible Sales before ingest](runbooks/troubleshooting.md#get-eligible-sales-triggered-before-ingest-completes). |

---

## Phase 3: Calculation (per Run Controller)

Run controllers are executed **one at a time**. Each goes through: Create → Review → Approve.

### 3.1 Trigger the Calculation

| | |
|---|---|
| **Who** | Accounting operator |
| **Where** | Abacus UI → Run Controller → click "Create" |
| **What** | Triggers Airflow DAG `accounting_run_calculate` |
| **Depends on** | Phase 2 complete (sales approved and mech deductions prepped) |

### 3.2 What the Calculation Does (Automated)

The DAG executes 8 steps. No human intervention needed, but monitoring is important.

```
                    MySQL (royalty_accounting)              S3                    Snowflake
                    ─────────────────────────              ──                    ─────────
Step 1: Set status ──→ accounting_run.run_status = "Running"

Step 2: Snapshot   ──→ Read contract/term/condition ──→ Write TSV files
                                                         contract.tsv
                                                         contract-terms.tsv

Step 3: Flatten    ←── Read TSV snapshots ──────────────────────────────────→ INSERT INTO
         (distro                                                               contract_denormalized_distro
          only)                                                                (cartesian join of terms × conditions)

Step 4: Match      ────────────────────────────────────────────────────────→ JOIN stmt_db_sales_distro
         (distro                                                               × contract_denormalized_distro
          only)                                                              → INSERT INTO contract_transaction_staging

Step 5: Calculate  ────────────────────────────── exchange_rate ───────────→ Apply term_rate, currency conversion
         (distro                                                             → INSERT INTO accounting_run_results_distro_staging
          only)

Step 6: Summary    ←── INSERT INTO                ←───────────────────────── SUM results by contract
                       ledger_accounting_run_balance                         (lambda: ledger_accounting_run_balance)

Step 7: Export     ─────────────────────────────── Write TSV ←──────────────  (from ledger_accounting_run_balance)
                   Update accounting_run.summary_export_url

Step 8: Set status ──→ accounting_run.run_status = "Complete"
```

**Key dependencies within the DAG:**
- Step 3 reads the TSV snapshot from Step 2
- Step 4 reads the denormalized output from Step 3 AND the sales data (from Phase 1)
- Step 5 reads matched transactions from Step 4 AND exchange rates (from Phase 1.2)
- Step 6 reads calculated results from Step 5

**Monitor via:** Airflow UI at `[env]-abacus-airflow` → DAG `accounting_run_calculate`

For full detail on each step (matching logic, calculation formulas, table schemas): see [Calculation Pipeline](architecture/calculation.md).

### 3.3 Review Results

| | |
|---|---|
| **Who** | Accounting operator + stakeholders |
| **Where** | Abacus UI (run summary) + Snowflake (BI queries) + downloaded TSV |
| **What** | Review the run summary: total gross/net revenue per contract, distribution fees |
| **Depends on** | Calculation complete (status = "Complete") |
| **Decision** | Approve (commit results) or Invalidate (discard and start over) |

### 3.4 Approve the Run

| | |
|---|---|
| **Who** | Accounting operator |
| **Where** | Abacus UI → Run Controller → click "Approve" |
| **What** | Triggers the commit pipeline |
| **Depends on** | Stakeholder review/sign-off |

The commit pipeline (automated):

```
1. Copy accounting_run_results_distro_staging → accounting_run_results_distro  (Snowflake)
2. Copy contract_transaction_staging → contract_transaction_distro             (Snowflake)
3. Invoke commit_royalties lambda → INSERT INTO ledger_account_contract        (MySQL)
   (rounding remainders → ledger_deposit)
4. [Distro only] Invoke reserves_take lambda → INSERT INTO ledger_reserve_taken
5. [Distro only] Debit reserves from ledger_account_contract
6. [Distro only] Invoke reserves_schedule lambda → schedule reserve releases
```

After approval, **repeat Phase 3 for each remaining run controller**, one at a time.

For full detail: see [Approval and Commit](architecture/approval-and-commit.md).

---

## Phase 4: Post-Run

### 4.1 DBT Refresh (First)

| | |
|---|---|
| **Who** | Accounting engineer |
| **Where** | Jenkins → [`dbt-accounting-scheduler`](https://scheduler.theorchard.io/job/dbt-accounting-scheduler/) |
| **What** | Refresh materialized views so Customer Accounting sees the latest data |
| **Depends on** | All run controllers approved |
| **Notify** | Post in `#abacus-x-ca` before and after |

Must explicitly include `abacus_fact_sales_unified_dbt` (excluded by default because it's large/expensive).

### 4.2 Customer Accounting Review

| | |
|---|---|
| **Who** | Customer Accounting team (Ciara's team) |
| **Where** | Abacus UI, Looker reports |
| **What** | Review revenue data, make manual adjustments if needed |
| **Depends on** | DBT refresh complete |

### 4.3 "Show to Customer"

| | |
|---|---|
| **Who** | Accounting operator |
| **Where** | Abacus UI |
| **What** | Makes the period's data visible to external clients |
| **Depends on** | Customer Accounting sign-off |

### 4.4 DBT Refresh (Second)

| | |
|---|---|
| **Who** | Accounting engineer (or automated via [WAR-2815](https://theorchard.atlassian.net/browse/WAR-2815)) |
| **Where** | Jenkins → `dbt-accounting-scheduler` |
| **What** | Second refresh needed because `moneyhub_unified_statement_periods` won't display the new statement until DBT runs again after "Show to Customer" |
| **Depends on** | "Show to Customer" clicked |

---

## Dependency Graph (Summary)

```
Finance delivers sales ─────────┐
                                 │
Exchange rates uploaded ────────┐│
                                ││
                                ▼▼
                    Sales ingested into Snowflake
                                │
                                ▼
                    Accounting period opened
                                │
                                ▼
            ┌── Get Eligible Sales ──→ Mark Delivered ──→ Approve Sales ──→ Prep Mech Deductions
            │
            ▼
    For each run controller (sequential):
        Create (calculate) ──→ Review ──→ Approve (commit)
            │
            ▼
    DBT refresh #1
            │
            ▼
    Customer Accounting review + adjustments
            │
            ▼
    "Show to Customer"
            │
            ▼
    DBT refresh #2
            │
            ▼
    Period is live to clients
```

---

## Actors and Their Responsibilities

| Role | Responsibilities |
|------|-----------------|
| **Finance / Sales Processing** | Deliver sales files, upload exchange rates |
| **Data matching contact** | Provides data packages for legacy runs; transaction matching for publishing sales |
| **Accounting operator** | Trigger ingest, manage periods, execute run controllers, trigger DBT refreshes |
| **Customer Accounting team** | Review post-run data, make manual adjustments, approve "Show to Customer" |
| **Publishing team lead** | Review publishing contract rates, approve publishing run results |
| **Engineering (on-call)** | Monitor DAGs, handle failures, environment refreshes |

> For current names mapped to each role, see the [Publishing Sales contacts table](runbooks/publishing-sales.md#contacts) and the [Accounting & Royalties team page (Notion)](https://www.notion.so/6bc9dbc46f5b44688cfe9192825f156c).

## Systems Involved

| System | Role in the Run |
|--------|----------------|
| **Abacus UI** | Period management, run controller execution, approval |
| **Jenkins (Scheduler)** | Sales ingest trigger, DBT refresh trigger, E2E pipeline |
| **Airflow (MWAA)** | DAG orchestration for all automated steps |
| **Snowflake** | Sales data, contract denormalization, transaction matching, calculation |
| **MySQL (RDS)** | Contract source-of-truth, ledger tables, run metadata |
| **S3** | Contract snapshots, sales file storage, data exchange |
| **Lambda** | Run summary, commit royalties, reserves |
| **Datadog** | Monitoring dashboards and log views |

## When Things Go Wrong

See the [Runbooks](runbooks/) for operational reference:
- [Monitoring](runbooks/monitoring.md) — what dashboards to watch during each phase, validation queries
- [Troubleshooting](runbooks/troubleshooting.md) — common failures with diagnosis and recovery steps
- [Environment Refresh](runbooks/environment-refresh.md) — resetting QA/dev/UAT from prod