# Payment Allocation

This Lambda function creates payment allocation records from flowthrough ledger adjustments. It groups adjustments by contract, payee, and currency, then creates `payment_allocation` and `payment_allocation_ledger_adjustment` records.

## Overview

Triggered by `abacus_outbox` when a `close_balance.completed` event occurs for a statement period payment entity.

**Workflow Steps:**
1.  **Event Validation:** Validates the EventBridge event structure using Pydantic.
2.  **SPPE Lookup:** Queries `statement_period_payment_entity` to derive `statement_period_id` and `reference_payment_entity_id`.
3.  **Close Balance Validation:** Queries `abacus_state` to verify close_balance is complete.
4.  **Batch Processing:** Discovers contracts with unlinked adjustments, bin-packs them into batches, and processes each batch (see [Processing Pipeline](#processing-pipeline) below).
5.  **Grouping:** Groups adjustments by `(contract_id, account_payee_id, currency_code, payee_currency_code)`.
6.  **Allocation Creation:** Creates one `payment_allocation` record per group with summed amounts, and links each adjustment via `payment_allocation_ledger_adjustment`. Commits after each batch.

## Performance Characteristics

- **Typical Duration:** < 10 seconds
- **Memory Requirements:** Low
- **Expected Volume:** Low frequency (per close_balance completion event)

## Event Schema

**Input Event (from EventBridge via outbox):**
```json
{
  "detail-type": "close_balance.completed",
  "detail": {
    "metadata": {
      "correlation_id": "550e8400-e29b-41d4-a716-446655440000",
      "target_id": 123,
      "target_type": "statement_period_payment_entity"
    }
  }
}
```

**Simple Input Event (for manual triggering / testing):**
```json
{
  "statement_period_payment_entity_id": 123
}
```

**Output Response:**
```json
{
  "statement_period_id": 456,
  "statement_period_payment_entity_id": 123,
  "allocations_created": 3,
  "ledger_adjustments_linked": 12
}
```

## Processing Pipeline

```
process
  → _validate_balances_closed
  → _process_adjustments
      → _process_batches → _process_adjustment_batch
                               → _group_adjustments
                               → _create_allocations
```

The outer loop in `_process_adjustments` discovers contracts with unlinked adjustments (up to `contract_batch_size` per round). Contracts are bin-packed into batches under `batch_size` using First Fit Decreasing. Oversized contracts (adjustment count > `batch_size`) are split into a full-portion batch and a remainder that is FFD-packed with regular contracts. All batches use LIMIT and query until empty, guaranteeing all adjustments are processed regardless of count accuracy.

Before processing begins, `batch_size` is clamped against the MySQL server's `max_allowed_packet` to prevent oversized `IN (...)` clauses. A warning is logged if clamping occurs.

## Architecture & Design

*   **Hexagonal Architecture:** Separation of concerns between the core logic (`processor.py`), data access (`connectors/`), utilities (`utils/`), and schemas (`schemas/`).
*   **Type Safety:** Uses **Pydantic** models for strict typing of database records and events, ensuring data integrity throughout the application.
*   **Idempotency:** The `NOT EXISTS` clause in the adjustment query excludes records already linked via `payment_allocation_ledger_adjustment`, making the lambda safe to re-run.
*   **Batch Resilience:** Commits after each batch so progress is preserved if the lambda times out or fails mid-run. Re-invocation picks up where it left off.

## Database Dependencies

**Tables:**
- `statement_period_payment_entity` (read) - Derives `statement_period_id` and `reference_payment_entity_id` from the target ID
- `ledger_adjustment_applied` (read) - Source of flowthrough adjustments
- `account_payment_term` (read) - Links accounts to payment entities
- `account_payee` (read) - Provides payee information for grouping
- `payment_allocation` (write) - Creates allocation records
- `payment_allocation_ledger_adjustment` (read/write) - Links allocations to adjustments; read to exclude already-allocated adjustments
- `abacus_state` (read) - Validates close_balance action is complete

## Edge Cases & Error Handling

| Scenario | Behavior | Retry Strategy |
| :--- | :--- | :--- |
| **SPPE Not Found** | Raises `StatementPeriodPaymentEntityNotFoundError` if the record doesn't exist in database. | **Non-Retriable** |
| **Balances Not Closed** | Raises `BalancesNotClosedError` if close_balance state is not complete. Defensive check since the event itself implies completion. | **Non-Retriable** |
| **No Adjustments Found** | Returns zero counts. No records are created. | **Handled Internally** |
| **Re-invocation** | Already-allocated adjustments are excluded from the query, preventing duplicate allocations. | **Handled Internally** |
| **DB Connection** | Raises `TransientError` for temporary database connection issues (connection lost, timeout, server gone away). | **Retriable** |
| **Event Validation** | Raises `PermanentError` if event structure is invalid (wrong detail-type, target_type, missing fields). | **Non-Retriable** |

## Design Decisions

**First Fit Decreasing (FFD) for contract batch packing.** Contracts are sorted by adjustment count descending, then each is placed into the first batch with enough remaining capacity. Sorting is done in-memory on the already-fetched contract counts (no added DB cost). Four approaches were considered:

| Algorithm | Time | Packing guarantee | Tradeoff |
|---|---|---|---|
| Next Fit (greedy sequential) | O(n) | up to 2·OPT bins | Fastest, but can waste ~50% of capacity — a large item leaves a gap that later small items could fill |
| First Fit (FF) | O(n·b) | at most 1.7·OPT + 1 bins | Scans existing batches for gaps, but input order can still produce poor packing |
| **First Fit Decreasing (FFD)** | O(n log n) | at most 1.22·OPT + 1 bins | One sort + same scan as FF. Near-optimal packing for negligible added cost |
| Exact bin-packing | NP-hard | OPT bins | Optimal, but exponential worst-case time. Impractical for unbounded input sizes |

FFD was chosen because each batch produces a DB round-trip (SELECT + INSERTs + COMMIT), each costing milliseconds of network and query time. The packing itself runs in microseconds on an in-memory list, so batch count dominates total runtime — making the O(n log n) sort cost of FFD negligible for a meaningful reduction in round-trips. Example: items [600, 600, 400, 350] with max 1000 pack into 2 batches with FFD vs 3 with Next Fit.

**DB-direct close_balance validation over OWS.** Close balance state is validated by querying `abacus_state` directly rather than calling the OWS states endpoint. The OWS call was timing out for large statement periods with many payment entities. The direct query is faster and avoids an external service dependency. The OWS-based method is retained in the codebase as a fallback reference.

**`max_allowed_packet` clamping on `batch_size` only.** Only `batch_size` affects query size (via the `IN (...)` clause in `get_adjustments_for_contracts`). `contract_batch_size` controls a `LIMIT %s` parameter — the query is fixed-size regardless of its value, so no packet-based clamping is needed for it.

**Inline SQL over separate query files.** Unlike `adjustment_file_prepare` which uses 260-line DuckDB queries loaded from `.sql` files, the queries here are short (5-15 lines) and tightly coupled to their repository methods. Separate files would add indirection without improving readability.

## Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `ENVIRONMENT` | Deployment environment | `dev` |
| `MYSQL_DB_HOST` | MySQL database host | — |
| `MYSQL_DB_NAME` | MySQL database name | — |
| `MYSQL_DB_USER` | MySQL username | — |
| `BATCH_SIZE` | Max adjustments per batch (clamped by `max_allowed_packet`). Range: [1,000 - 250,000] | `10000` |
| `CONTRACT_BATCH_SIZE` | Max contracts per discovery round. Range: [1,000 - 250,000] | `250000` |
| `PAYMENT_ALLOCATION_DESCRIPTION` | Description written to `payment_allocation` records. Must not be empty | `Flowthrough payment allocation` |

## Requirements

- Python 3.13
- [uv](https://docs.astral.sh/uv/) - Fast Python package manager
- Docker (for containerized testing)

## Local Development

### Setup
1.  Copy the environment file:
    ```bash
    cp .env.shadow .env
    ```
2.  Install dependencies:
    ```bash
    make env_dev
    ```

### Running Tests
*   **Unit Tests:** `make test` (runs pytest with coverage)
*   **Linting:** `make lint` (check) or `make lint_fix` (auto-fix)
*   **Formatting:** `make format` (check) or `make format_fix` (auto-fix)
*   **Docker Tests:** `make docker_test` (runs linting and tests inside a container)
*   **Integration Tests:** `make docker_test_local_integration` (runs integration tests in Docker with full service dependencies)

### Running Locally
Run the Lambda function handler locally as a script:
```bash
make run
```

Or run as a Docker container:
```bash
make docker_local_up
```

## Deployment

Deploys via the [Jenkins pipeline](https://pipeline.theorchard.io/job/lambda-abacus-pipeline/). A webhook triggers the deploy for changes in this directory.