# delphi-dapd-api-scraper
## Description
This service stores public data obtained from Apple Music/Spotify API's
into DB (ETLDB) and S3 using WorkflowDB as meta db for DAPD project

## Links
* [Source](https://github.com/filtr/delphi-dapd-api-scraper)
* [JIRA](https://jenkins.apollo.stream/job/Delphi/job/delphi-dapd-api-scraper/)

## Usage
Prerequisites
* Installed python3.10
* Installed [aws_cli](https://aws.amazon.com/cli/)
* Working credentials in ~/.aws/credentials
* delphi dev or prod vpn connection estabilished for dependencies install

1) Install poetry for package management
```console
curl -sSL https://install.python-poetry.org | python3 -
```
2) Add poetry to PATH
```console
export PATH=$HOME/.local/bin:$PATH
```
3) Install project dependencies
```console
poetry install
```
4) Activate venv
```console
. .venv/bin/activate
```
5) Set mandatory environment variable (or use .env file)
```console
# example values
export ENVIRONMENT=dev
export SPOTIFY_CREDENTIALS=delphi/dev/apd/spotify
export APPLE_MUSIC_CREDENTIALS=delphi/dev/apd/apple-music
export WORKFLOWDB_SECRET_NAME=delphi/dev/workflowdb/pg/user
export ETLDB_SECRET_NAME=delphi/dev/pd_etldb/pg/user
export DEBUG=false   # true to enable debug mode
export SENTRY_SECRET_KEY={working_sentry_key}
export RAW_BUCKET=dev-delphi-public-data-raw
export STORE_RAW=false
export ENTITIES=AppleMusicPlaylist,SpotifyTrack  # one or more comma separated entity, to enable all do not set this variable
export IS_LOCAL=true
```
6) Run service
```console
dapd-api-scraper
```
or
```console
python3.10 dapd_api_scraper/main.py
```
this entrypoint could be used for ide debugging