ows-dmp
=======

The Orchard Web Service for Audience Development Data Management Platform.

## Getting Started

### Installation

Make sure you have `uv` installed (https://docs.astral.sh/uv/getting-started/installation/).

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

or if `uv` already installed, ensure it is up to date:

```bash
uv self update
```

Prepare environment and install dependencies:

```bash
$ cd ows-dmp
$ make env
```

### Running

Copy environment variables:

```bash
cp .env.shadow .env
```

Here is an example of the `.env` file:

```bash
APP_DEBUG=true
Environment=dev

# Snowflake credentials
SNOWFLAKE_ROLE=FANSIFTER_ENGINEERING
SNOWFLAKE_ACCOUNT=delphi.us-east-1
SNOWFLAKE_USER={username}@sonymusic-pde.com
SNOWFLAKE_DATABASE=FANSIFTER_APP_REPORTING
SNOWFLAKE_WAREHOUSE=DEV_OWS_WH
SNOWFLAKE_KEY_PASSPHRASE=
SNOWFLAKE_PRIVATE_KEY="{inlined private key here}"
SNOWFLAKE_SCHEMA={DEV_USERNAME}
SNOWFLAKE_SHOPIFY_SCHEMA={DEV_USERNAME}_SHOPIFY

# Postgres credentials
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_NAME=dmp

# Redis
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_SSL=false

# Cache
CACHE_BACKEND=null
CACHE_REDIS_DB=1

# Logging
LOG_FORMAT=console

# Split.io
SPLIT_IO_API_KEY=localhost

SENTRY_DSN=

# Auth
JWT_AUTH_ENABLED=false
AUTH_ALLOW_FULL_ACCESS=true

# Audiences
AUDIENCE_EXPORT_BUCKET_NAME=qa-audience-export-files

FIVETRAN_API_KEY=
FIVETRAN_API_SECRET=

# Facebook
FACEBOOK_APP_ID=
FACEBOOK_APP_SECRET=
FACEBOOK_AD_ACCOUNT_ID=
FACEBOOK_APP_SYSTEM_USER_ACCESS_TOKEN=

# KMS
KMS_ENABLED=true
DMP_KMS_KEY_ID=arn:aws:kms:us-east-1:437795906767:alias/qa-ows-dmp
```

When all dependencies have been installed, ensure you have a local instances of `PostgreSQL` and `Redis` are up and running.

```bash
docker compose up -d fansifter-redis
docker compose up -d fansifter-postgres
```

Now you can run the FastAPI application on your local instance by running:

```bash
make dev
```

Navigate to http://127.0.0.1:8001/docs to view Swagger UI.

---

All make commands will automatically install all dependencies in a virtual environment folder called env.
You can activate a virtual environment by running `source .venv/bin/activate`.

By using the development server, you will have access to specific features that
are not necessarily available in production, such as the exception tracer.

### Testing

To run the tests, all you have to do is to run:

```bash
(.venv) $ pytest tests/unit/
(.venv) $ pytest tests/integration/
```

or

To run unit tests: `make test`

To run unit tests with coverage reporting: `make test_unit`

To run integration tests: `make test_integration`


### OpenAPI specification

OpenAPI specification available for dev environment only.

Go to http://127.0.0.1:8001/docs to view Swagger UI.

To generate `OpenAPI YAML`:

```bash
$ make openapi
```

### Linting

To lint: `make lint`

To format: `make fmt`

### Scheduler

Automated daily maintenance jobs that ensure data pipeline health and geographic reference data accuracy:

- **Jenkins Scheduler**: [https://scheduler.theorchard.io/job/ows-dmp-scheduler](https://scheduler.theorchard.io/job/ows-dmp-scheduler)

**Daily Jobs:**
- **Fivetran Tables Sync**: Maintains data pipeline configurations across all advertising platforms (Meta, TikTok, Google, Shopify), ensuring critical tables for audience development and campaign reporting remain enabled, custom reports stay synchronized, and unnecessary tables are disabled to accelerate sync performance and reduce costs
- **GeoNames Data Sync** (Mondays): Updates geographical reference data from GeoNames.org for accurate location-based audience targeting and geographic analytics

**Configuration**: See `Jenkinsfile.scheduler` for pipeline details and environment-specific parameters

### Third-Party Integrations for Reporting

This boilerplate comes with support for Sentry, and Datadog.