# dapd_db_seed

## Make commands
- `make venv` - create local virtual env(base.pip dependencies)
- `make devenv` - create local virtual env(dev.pip dependencies)
- `make test` - run tests
- `make lint` - run pylint check
- `make clean` - clean unnecessary files
- `make docker/login` - aws configuration need to be set first
- `make docker/build` - build Docker images from a Dockerfile

(when run `make test`, `make lint`  command, all dependencies from requirements/test will be installed.
When run `make venv` base.pip dependencies will be installed)

## Developers guide

Use `pre-commit` for git hooks managing (https://pre-commit.com/#installation). Hook configuration
can be found at .pre-commit-config.yaml. `isort` and `yapf` tools are included to run as a part
of precommit hook.

To register a new hook run:
```shell script
pre-commit install
```

To run hooks without code committing:
```shell script
pre-commit run
```

Sometimes these tools can have a conflict regarding import sorting. In this case consider to
enable `include_trailing_comma` option at `setup.cfg` file.

Don’t forget to run `pylint` and `mypy` before code committing manually. It will save you from
wasting time at reading CI logs for failed build.

In order to check everything is fine (and have a dry run for formatting and import sorting)
consider following commands, e.g. for `dapd_db_seed`:
```shell script
cd dapd_db_seed
workon dapd_db_seed # or './venv/bin/activate'
isort --check-only -rc . && yapf --recursive --verbose --parallel --diff . \
&& pylint --rcfile=pylintrc ./${PWD##*/} ./tests && mypy ${PWD##*/} && echo 'Exit code:' $?
```

It will run all checks inside a current directory.


## How to refresh list of playlists

While ApolloDB is our source of playlists we may need to download list of available playlists and ingest them into WorkflowDB.
There is a `refresh` script:

    usage: refresh [-h] -e ENV [-l LIMIT] -a {spotify,apple_music}
                [--data-dir DATA_DIR]

    optional arguments:
    -h, --help            show this help message and exit
    -e ENV, --env ENV     Environment
    -l LIMIT, --limit LIMIT
                            Limit number of playlist to fetch. Empty means no
                            limit
    -a {spotify,apple_music}, --action {spotify,apple_music}
                            Available actions
    --data-dir DATA_DIR   Path to directory where all files are landed on a
                            local FS

Download Apple Music playlists:

    refresh --env dev --data-dir ./data -a apple_music

    ...

    INFO:dapd_db_seed.refresh:{"filename": "./data/apple_music/all/playlist_ids.ndjson", "rows_count": 48458, "event": "Refreshed Apple Music playlist ids from Apollo DB", "logger": "dapd_db_seed.refresh", "level": "info", "timestamp": "2020-11-22T23:15:28.400799Z"}

Download Spotify playlists:

    refresh --env dev --data-dir ./data -a spotify

    ...

    INFO:dapd_db_seed.refresh:{"filename": "./data/spotify/all/playlist_ids.ndjson", "rows_count": 1270017, "event": "Refreshed Spotify playlist ids from Apollo DB", "logger": "dapd_db_seed.refresh", "level": "info", "timestamp": "2020-11-22T23:16:50.770135Z"}


## Seed WorkflowDB with downloaded Playlists

Seeding script ingest initial data into WorkflowDB. When record exist this will not be rewritten.

# FLOW

1) refresh `playlist_ids.ndjson` file
2) move this file from the `./data/<dsp>/<size>/playlist_ids.ndjson` to `./data/<dsp>/playlist_ids.ndjson`
3) run seeding script


    seed --help
    usage: seed [-h] [-c CONCURRENCY] [-a {all,base,spotify,apple_music}] [-e ENV]
                [--data-dir DATA_DIR] [--postgres-secret-key POSTGRES_SECRET_KEY]
                [--postgres-host POSTGRES_HOST] [--postgres-port POSTGRES_PORT]
                [--postgres-db POSTGRES_DB] [--postgres-user POSTGRES_USER]
                [--postgres-password POSTGRES_PASSWORD]

    optional arguments:
    -h, --help            show this help message and exit
    -c CONCURRENCY, --concurrency CONCURRENCY
                            Concurrency
    -a {all,base,spotify,apple_music}, --action {all,base,spotify,apple_music}
                            seed script
    -e ENV, --env ENV     environment
    --data-dir DATA_DIR   directory with sources
    --postgres-secret-key POSTGRES_SECRET_KEY
                            postgres secret key
    --postgres-host POSTGRES_HOST
                            postgres host
    --postgres-port POSTGRES_PORT
                            postgres port
    --postgres-db POSTGRES_DB
                            postgres database
    --postgres-user POSTGRES_USER
                            postgres user
    --postgres-password POSTGRES_PASSWORD
                            postgres password

Create Spotify Playlists

    seed -a spotify -c 10 --data-dir ./data -e dev

Create Apple Music Playlists

    seed -a apple_music -c 10 --data-dir ./data -e dev