# TADAS ETL

## Architecture

See [ARCH.md](ARCH.md) for how the code is laid out, what each layer is
responsible for, and where new code belongs. The layering rule is enforced
in CI via `make check_imports`.

## Development: run it locally

### Pre-requirements
- Connectivity to ReportingDB (requires SME VPN)
- Copy .env.shadow to .env and edit it to set your settings

### Local MainDB

We have local version of MainDB which is used for development purposes.
It set in docker-compose.yml file as service `maindb`.
It exposes port `5434` to host machine.

ATTENTION: the port is different from default Postgres port `5432`! 
It is done in order to avoid conflicts with other Postgres instances running on your machine.


So you can connect from your favourite DB UI with url like:
```
jdbc:postgresql://localhost:5434/maindb?user=tadas_etl&password=devpassword
```

### Connection to prod MainDB

If you plan to use prod maindb.
Normally in DEV environment it can be used provided by 

Use this settings if you want to connect to production MainDB.

.env
```
MAINDB_HOST_PORT=host.docker.internal:8015
```

The only way to connect to MainDB is SSH tunnel.
So first, you need access to SME bastion server.
MainDB is a postgres database. We use tunnel to local port 8015.

Below is the sample `~/.ssh/config` which can do the job:

```
Host cloudops-bastion
    HostName <cloudops-give-you-bastion-hostname>
    User <your-username>
    IdentityFile ~/.ssh/cloudops-given-key.pem
    ForwardAgent yes

Host tadas-vm
    HostName <datascience-vm-instance>
    User <your-username>
    ProxyJump cloudops-bastion
    LocalForward 8015 main-pg-db-dev.gdbdatascience.com:5432
    LocalForward 8016 reporting-db.smeanalyticsapps.com:5439
    ForwardAgent yes
```

When you have this config, you can run the following command to start the tunnel:
It is recommended to run some command because overwise shell will close after timeout of 600-900 seconds.

```bash
$ ssh tadas-vm
[user@hostname ~]$ htop
```


### Build and run locally

`cp .env.shadow .env`

Edit `.env` file to set your local environment variables.

There are many supported models. Each model has its own folder and can be run independently.

```bash
cd tadas/models/<model_version>

# build the model on the most recent data
make inference

# build the model on the given date
make inference REPORT_DATE=2026-12-31

# publish the model to snowflake tadas tables
make publish

# run monitoring and output to stdout
make monitor 
```