# PP-1201: Expired JWT Warning Logs - Datadog UAT

This Docker Compose project triggers the expired M2M token warning (line 132-135 in `owsclient/m2m/base.py`) and sends logs to Datadog for dashboard verification.

## Overview

The project includes:
- **Python App Container**: Runs a UAT script that continuously fetches an expired M2M token from AWS Secrets Manager
- **Datadog Agent Container**: Collects and forwards logs to Datadog

## Prerequisites

1. **AWS Credentials**: Access to AWS Secrets Manager in the QA environment
2. **Datadog API Key**: Valid API key from your Datadog account
3. **Docker**: Docker and Docker Compose installed

## Setup Instructions

### 1. Configure AWS Credentials

Run `awsume` to get temporary AWS credentials:

```bash
awsume permissions-platform-qa
```

### 2. Create Environment File

Copy the example environment file and populate with your Datadog API key:

```bash
cd python
cp .env.example .env
```

Edit `.env` and add:
- **DD_API_KEY**: The `dev` API key from https://sonymusic-pde.datadoghq.com/organization-settings/api-keys

**Note**: AWS credentials are automatically picked up from your shell environment after running `awsume`.

Example `.env` file:
```bash
DD_API_KEY=abcd1234efgh5678ijkl
SERVICE_NAME=lambda-machine-1
ENVIRONMENT=qa
LOOP_INTERVAL_SECONDS=30
```

### 3. Start the Docker Compose Stack

**Using Make (recommended):**
```bash
make up       # Start in background
make logs     # View UAT logs (`make logs-all` to view datadog container logs)
make down     # Stop containers
```

**Using docker compose directly:**
```bash
docker compose up --build -d
docker compose logs -f python-app
docker compose down
```

### 4. Verify Logs in Datadog

1. Go to your Datadog Logs dashboard: [https://app.datadoghq.com/logs](https://sonymusic-pde.datadoghq.com/logs?query=env%3Aqa%20%40library.name%3Apython-owsclient%20%40level%3AWARNING&agg_m=count&agg_m_source=base&agg_t=count&fromUser=false&messageDisplay=inline&refresh_mode=sliding&storage=hot&stream_sort=time%2Cdesc&viz=stream&from_ts=1769454338000&to_ts=1770059138000&live=true)
2. Use this search query to find the warning logs:
   ```
   env:qa @library.name:python-owsclient @level:WARNING
   ```
3. Look for log messages containing:
   ```
   Fetched an expired token from secrets manager. Please manually rotate 'qa/lambda-machine-1/M2M_JWT_ACCESS_TOKEN'
   ```
   
4. View the owsclient Warning logs M2M Rotation lambda dashboard: [lambda-auth0-m2m-token-secret-rotation DASHBOARD](https://sonymusic-pde.datadoghq.com/dashboard/dpv-z5r-w5s/lambda-auth0-m2m-token-secret-rotation-dashboard?fromUser=false&refresh_mode=paused&from_ts=1769455165397&to_ts=1770059965397&live=false&tile_focus=885218743599520)
![warning log screenshot](./images/m2m_dashboard_owsclient_section.png)

### 5. Stop the Stack

```bash
docker compose down
```

## Configuration

### Environment Variables

| Variable                | Default            | Description                           |
|-------------------------|--------------------|---------------------------------------|
| `DD_API_KEY`            | *required*         | Datadog API key                       |
| `SERVICE_NAME`          | `lambda-machine-1` | Service name for the M2M token secret |
| `ENVIRONMENT`           | `qa`               | Environment name (qa, prod, etc.)     |
| `LOOP_INTERVAL_SECONDS` | `30`               | Time between token fetch attempts     |

### Secret Name

The script fetches from: `{ENVIRONMENT}/{SERVICE_NAME}/M2M_JWT_ACCESS_TOKEN`

Default: `qa/lambda-machine-1/M2M_JWT_ACCESS_TOKEN` (which contains an expired token)

## Troubleshooting

### No logs appearing in Datadog

1. Check that `DD_API_KEY` is correct
2. Verify the Datadog agent container is running: `docker compose ps`
3. Check Datadog agent logs: `docker compose logs datadog`
4. Wait a few minutes - logs may take time to appear in Datadog

### AWS authentication errors

1. Ensure you've run `awsume permissions-platform-qa` in your current shell
2. Docker Compose will automatically use AWS credentials from your environment
3. Check that session token hasn't expired (they expire after 1 hour - just re-run `awsume`)

### Container fails to build

1. Ensure you're running from the project root or the PP-1201 directory
2. Check Docker daemon is running
3. Verify the build context includes the owsclient package

### View container logs

```bash
# UAT script logs only
make logs

# See all logs
make logs-all
```

Or with docker compose:
```bash
docker compose logs -f
docker compose logs -f python-app
docker compose logs -f datadog
```

## Testing Different Secrets

To test with a different secret, modify the `.env` file:

```bash
SERVICE_NAME=your-service-name
ENVIRONMENT=qa  # or dev, prod, etc.
```

The script will fetch: `{ENVIRONMENT}/{SERVICE_NAME}/M2M_JWT_ACCESS_TOKEN`

## Architecture

```
┌─────────────────┐      ┌──────────────────┐
│  Python App     │      │  Datadog Agent   │
│  Container      │─────▶│  Container       │
│                 │ logs │                  │
│ - Fetches token │      │ - Collects logs  │
│ - Logs warning  │      │ - Forwards to DD │
└─────────────────┘      └──────────────────┘
        │                         │
        │                         │
        ▼                         ▼
┌─────────────────┐      ┌──────────────────┐
│  AWS Secrets    │      │  Datadog Cloud   │
│  Manager (QA)   │      │  Logs Platform   │
└─────────────────┘      └──────────────────┘
```

## Expected Output

The Python app will continuously log output like:

```
pp1201-python-app  | {"message": "Fetched an expired token from secrets manager. Please manually rotate 'qa/lambda-machine-1/M2M_JWT_ACCESS_TOKEN'", "timestamp": "2026-01-30T20:51:02.582569+00:00", "level": "WARNING", "levelname": "WARNING", "service": "owsclient-uat", "environment": "qa", "thread_id": 140737471067968, "process_id": 1, "log_type": "application", "logger_name": "owsclient-uat", "logger_version": "audience-common/logger/0.40.0", "tag": "ows1", "tags": ["log_type:application"], "dd": {"version": "0.1", "env": "qa", "service": "owsclient-uat"}, "logger": {"name": "owsclient-uat", "pathname": "/usr/local/lib/python3.11/site-packages/owsclient/m2m/base.py", "file_name": "base.py", "method_name": "get_token_string", "lineno": 132, "thread_name": "MainThread"}}
```

The **WARNING** line is what you're looking for in the Datadog dashboard!