sendgrid-webhooks-inbound
==================

## Overview
An AWS Lambda handler (invoked via an AWS Application Load Balancer - ALB) that processes and validates SendGrid inbound webhook requests. It handles two types of webhook events:

- **Unsubscribe emails** (`/inbound` and `/inbound-unsub` paths) - Processes unsubscribe requests from email recipients and communicates with the OWS Preference Center
- **Reply emails** (`/inbound-reply` path) - Processes email replies, forwards them to Kafka, and sends emails to appropriate recipients via SendGrid

The service validates incoming webhook requests with concrete models (`UnsubRequest` and `ReplyRequest`), parses email metadata (sender, recipient, subject, text, HTML content), and routes them to appropriate handlers for further processing. For reply emails, recipient metadata is validated using a reply-to secret key and then used to look up the forward email address from Snowflake.

## Getting Started

### Requirements
Minimum requirements and recommended tooling:

- Python: 3.13+
- UV (astral `uv` tool) for environment and task running (see `pyproject.toml` `tool.uv` configuration)
- Docker and Docker Compose (used for CI targets)
- Make (for provided Makefile targets)

Developer tooling (used in CI / recommended):

- `pytest` (unit tests)
- `pytight` and `ruff` (type checks / linting)
- Redis (required by the reply anti-spam cache in runtime environments)

### Installation
```bash
# Install uv (if not installed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install project dependencies
make env
```

### Configuration
For local development copy the provided example environment file to `.env`:

```bash
cp .env.example .env
```

After copying, edit `.env` and fill in any required values (for example AWS settings, Kafka configuration, or OWS Preference Center credentials) before running the service or tests.

For reply webhook processing, the service also expects Redis connection settings and anti-spam limits:

- `REDIS_HOST`, `REDIS_PORT`, `REDIS_DB`, `REDIS_SSL`
- `REPLY_REQUESTS_COUNT_LIMIT`
- `REPLY_REQUESTS_COUNT_RESET_SECONDS`

### Reply forwarding sender settings

When forwarding reply emails through SendGrid, the sender address is built from `FROM_EMAIL_DOMAIN` (default: `email.qa-fansifter.theorchard.io`).

- The effective sender template is derived as `fan-reply-{unique_part}@<FROM_EMAIL_DOMAIN>`.
- `{unique_part}` is generated per forwarded reply and currently uses an 8-character UUID prefix.
- The SendGrid `from.name` value is taken from the original sender email address in the inbound reply.
- Email forwarding is **disabled in production** — SendGrid emails are only sent in non-production environments.

### QA forwarding allowlist

On QA, reply forwarding is restricted by account allowlist.

- `ACCOUNTS_TO_FORWARD_ON_QA_ENV` controls which vendor IDs are allowed to receive forwarded reply emails.
- If a mapping is found but its `vendor_id` is not in this list, the service skips the SendGrid forward while still acknowledging and processing the event.
- Default value is `7123`.

### Reply anti-spam protection

Reply requests are deduplicated and rate-limited with Redis before any forwarding or Kafka publishing happens.

- The cache key is built from the inbound `email_id` and sender email hash.
- The first request starts a TTL window controlled by `REPLY_REQUESTS_COUNT_RESET_SECONDS`.
- Requests above `REPLY_REQUESTS_COUNT_LIMIT` are acknowledged with HTTP 200 but skipped to prevent repeated spam processing.

## Testing

### Unit tests
Run the project's unit tests with:

```bash
make test
```

## Linting and formatting
Run the linter with:

```bash
make lint
```

Run the code formatter with:

```bash
make fmt
```

## Project Structure

- `app/` - Main application code
  - `__init__.py` - Application setup (logging and Sentry initialisation)
  - `handlers.py` - Handler implementations for unsubscribe and reply events
  - `models.py` - Pydantic request models (`BaseRequest`, `UnsubRequest`, `ReplyRequest`) and webhook parsing logic
  - `config.py` - Configuration management (AWS, Kafka, Snowflake, SendGrid settings)
  - `main.py` - Lambda handler entry point and event routing
  - `modules.py` - Dependency injection configuration
  - `types.py` - Email type enumerations
  - `connectors/` - External service connectors
    - `ows_preference_center.py` - OWS Preference Center client
    - `sendgrid.py` - SendGrid email client for forwarding reply emails
    - `aws/s3.py` - AWS S3 client for failed event logging
    - `db/` - Database connectors and models
      - `models.py` - SQLAlchemy ORM models for Snowflake tables
      - `orm.py` - Base ORM model configuration
      - `repository.py` - Base repository class
      - `repositories/` - Repository implementations
        - `account_to_fan_response_email_address_mapping.py` - Repository for email forwarding mappings
- `tests/` - Test suite
  - `unit/` - Unit tests
    - `handlers/` - Handler tests
    - `connectors/` - Connector tests
    - `data/` - Test event data files

## Event Processing

### Recipient source

The request models use `envelope.to[0]` from the SendGrid inbound payload as the canonical recipient. This field reflects the SMTP recipient accepted by SendGrid and is more reliable than the human-readable `to` header, which may contain display names or multiple addresses.

### Webhook Paths

- `/inbound` - Unsubscribe handler (SendGrid inbound parse webhook)
- `/inbound-unsub` - Unsubscribe handler (alternative path)
- `/inbound-reply` - Reply handler

## Running locally
This AWS Lambda is not intended to be run locally — it is designed to handle real requests based on email webhooks from SendGrid. For local checks use the unit tests (`make test`) and the linter/formatter (`make lint`, `make fmt`).