# Norway Jobs CEA

A data pipeline that extracts data from a reporting database, converts it to Tableau Hyper files, and publishes to Tableau datasources.

## Install

```bash
uv sync
```

## Prerequisites

Fill the following variables in your `.env` file:

```bash
LOGDIR=/path/to/logs
DATADIR=/path/to/data

#Reporting-DB
hostReportingDB
portReportingDB
dbnameReportingDB
usernameReportingDB
passwordReportingDB

#Tableau
 tableauServer
 tableauUsername
 tableauPassword
```

## Usage

Run all jobs:
```bash
uv run src/main.py
```

Run a specific job:
```bash
uv run src/main.py <job_name>
```

## Available Jobs

| Job | Description |
|-----|-------------|
| `marketshares` | Market share data (last 7 days, totals, charts, vendor codes) |
| `skip_saves` | Skip/save data (top 100 for 7 days and 365 days) |
| `playlist_top_200` | Top 200 playlists |
| `search_terms` | Search term analytics |
| `spotify_top200artists` | Top 200 Spotify artists |

## Project Structure

```
├── src/
│   ├── config.py      # Job definitions and logging configuration
│   └── main.py        # Main pipeline logic
└── sql/               # SQL query files referenced by jobs
```

## How It Works

1. Reads SQL queries from `../sql/` directory
2. Executes queries against the reporting database
3. Writes results to Hyper files in `DATADIR`
4. Publishes Hyper files to Tableau datasources
5. Cleans up temporary Hyper files