# Scraper tool

## Preparation

- create `.env` file from `.env.template` for local execution
- enable VPN to get access to AWS Secrets (credentials can be placed in `.env`. In this case scraper will not try to get them from AWS Secrets)
- `make venv` - setup scraper to the virtual environment for local execution
- set SPOTIFY_CLIENT_ID and SPOTIFY_CLIENT_SECRET via environment variables
- set APPLE_MUSIC_TEAM_ID, APPLE_MUSIC_KEY_ID, and APPLE_MUSIC_SECRET_KEY via environment variables

## Generic command

    usage: scrape [-h] [--tasks TASKS] [--tasks-file TASKS_FILE]
                [--tasks-file-delete] --env ENV --data-source
                {spotify,apple_music}
                [--validation-schema-storage VALIDATION_SCHEMA_STORAGE] --entity
                ENTITY [--postgres-host POSTGRES_HOST]
                [--postgres-port POSTGRES_PORT] [--postgres-db POSTGRES_DB]
                [--postgres-user POSTGRES_USER]
                [--postgres-password POSTGRES_PASSWORD]
                [--aws-region AWS_REGION]
                [--aws-access-key-id AWS_ACCESS_KEY_ID]
                [--aws-secret-access-key AWS_SECRET_ACCESS_KEY]
                [--aws-endpoint-url AWS_ENDPOINT_URL] --kinesis-transform-stream
                KINESIS_TRANSFORM_STREAM --s3-backup-bucket S3_BACKUP_BUCKET
                [--sync] [--concurrency CONCURRENCY]

    optional arguments:
    -h, --help            show this help message and exit
    --tasks TASKS         Pass tasks via STDIN or use --tasks-file
    --tasks-file TASKS_FILE
                            Path to tasks file that contains tasks
    --tasks-file-delete   Delete task file in the end
    --env ENV             environment
    --data-source {spotify,apple_music}
                            Name of a data source
    --validation-schema-storage VALIDATION_SCHEMA_STORAGE
                            Bucket with validation schemas
    --entity ENTITY       type of entity
    --postgres-host POSTGRES_HOST
                            postgres host
    --postgres-port POSTGRES_PORT
                            postgres port
    --postgres-db POSTGRES_DB
                            postgres database
    --postgres-user POSTGRES_USER
                            postgres user
    --postgres-password POSTGRES_PASSWORD
                            postgres password
    --aws-region AWS_REGION
                            AWS Region
    --aws-access-key-id AWS_ACCESS_KEY_ID
                            AWS acceess key id
    --aws-secret-access-key AWS_SECRET_ACCESS_KEY
                            AWS secret access key
    --aws-endpoint-url AWS_ENDPOINT_URL
                            AWS endpoint
    --kinesis-transform-stream KINESIS_TRANSFORM_STREAM
                            AWS Kinesis Stream name. Record will be transformed
    --s3-backup-bucket S3_BACKUP_BUCKET
                            AWS S3 bucket name. Storage for raw data
    --sync                Run scraper in sync mode
    --concurrency CONCURRENCY
                            Size of thread-poola


Examples scripts are in `tests/smoke`

## Smoke testing

Go to tests/smoke

    cd tests/smoke

Run apple_music and spotify ingestion

    ./all.sh

Run apple_music ingestion

    cd ./apple_music && all.sh

Run spotify ingestion

    cd ./spotify && all.sh

Or execute ingestion scripts one by one