# dbt-luminate

A dbt (data build tool) project for transforming and modeling Luminate data in Snowflake.

## Overview

This project uses dbt to build data models and transformations for Luminate data within the Sony Music data platform. It leverages dbt-core with Snowflake as the data warehouse backend.

## Prerequisites

- Python 3.12.13 or higher
- Poetry (package management)
- Access to Snowflake account (`delphi.us-east-1`)
- Docker (for containerized development and CI/CD)

## Project Structure

TBD

## Setup

### Local Development

1. **Install dependencies:**
   ```bash
   make env
   ```
   This will create a virtual environment with Python 3.12 and install all required packages using Poetry.

2. **Install dbt packages:**
   ```bash
   make packages
   ```
   This installs dbt package dependencies defined in `packages.yml` (including dbt_utils).

3. **Configure Snowflake credentials:**
   - Set up your environment variables (see `.env.shadow` for template)

4. **Export environment variables:**
   ```bash
   export $(grep -v ^# .env | xargs)
   ```


## Available Commands

Use `make help` to see all available commands:

- `make env` - Install development requirements
- `make packages` - Install dbt packages
- `make clean` - Clean generated files and caches

## Code Quality

This project uses **sqlfluff** for SQL linting and formatting to maintain consistent code quality:

## Dependencies

### Python Packages
- **dbt-core** (>=1.11.7) - Core dbt functionality
- **dbt-snowflake** (>=1.11.3) - Snowflake adapter for dbt
- **sqlfluff** (>=4.0.0) - SQL linter
- **sqlfluff-templater-dbt** (>=4.0.0) - dbt templating support for sqlfluff

### dbt Packages
- **dbt_utils** (1.3.3) - Utility macros for dbt projects

## CI/CD

The project includes Docker-based CI/CD workflows:

- tbd Automated linting and testing via `docker-compose`
- Consistent environments between local development and CI/CD
- Snowflake authentication via SSH key pairs

## Contributing

1. Create a feature branch
2. Make your changes
3. tbd Run linting: `make lint`
4. tbd Run tests: `make dbt_test`
5. Submit a pull request

## Team

**Authors:** Sony Music PDE - Data Squad  
**Contact:** data-squad@sonymusic-pde.com

## Application Family

`data-platform`

## Version

- **Project Version:** 0.1.0
- **dbt Version:** >=1.11.7