# SLZ-Downloader Task Prepare environment make venv Activate virtual environment source venv/bin/activate Run slz-downloader ## slz-downloader params slz-downloader --help usage: slz-downloader [-h] --environment ENVIRONMENT --input INPUT [--download-concurrency DOWNLOAD_CONCURRENCY] [--max-download-retries MAX_DOWNLOAD_RETRIES] --rds-secrets-key RDS_SECRETS_KEY --dsp-settings-path DSP_SETTINGS_PATH --dsp-complete-criterias-path DSP_COMPLETE_CRITERIAS_PATH [--sqs-arns-meta SQS_ARNS_META] [--sqs-arns-split SQS_ARNS_SPLIT] [--debug] [--humanize] [--https-proxy-host HTTPS_PROXY_HOST] [--aws-region AWS_REGION] [--aws-endpoint AWS_ENDPOINT] [--sfn-task-token SFN_TASK_TOKEN] [--chunk-size CHUNK_SIZE] [--upload-concurrency UPLOAD_CONCURRENCY] [--upload-queue-size UPLOAD_QUEUE_SIZE] [--bucket-source BUCKET_SOURCE] [--bucket-source-key BUCKET_SOURCE_KEY] [--bucket-archive BUCKET_ARCHIVE] [--bucket-archive-quarantine BUCKET_ARCHIVE_QUARANTINE] [--bucket-decompressed BUCKET_DECOMPRESSED] [--bucket-decompressed-quarantine BUCKET_DECOMPRESSED_QUARANTINE] [--bucket-corrupted BUCKET_CORRUPTED] [--enable-historical-validation] [--validation-rows-number VALIDATION_ROWS_NUMBER] [--validation-rate VALIDATION_RATE] [--validation-threshold-min-file-size VALIDATION_THRESHOLD_MIN_FILE_SIZE] [--download-queue-size DOWNLOAD_QUEUE_SIZE] [--apollo-db-username APOLLO_DB_USERNAME] [--apollo-db-password APOLLO_DB_PASSWORD] [--apollo-db-host APOLLO_DB_HOST] [--apollo-db-port APOLLO_DB_PORT] [--apollo-db-name APOLLO_DB_NAME] [--apollo-worker-complete-days-offset APOLLO_WORKER_COMPLETE_DAYS_OFFSET] [--delphi-db-username DELPHI_DB_USERNAME] [--delphi-db-password DELPHI_DB_PASSWORD] [--delphi-db-host DELPHI_DB_HOST] [--delphi-db-port DELPHI_DB_PORT] [--delphi-db-name DELPHI_DB_NAME] [--delphi-bigtable-path DELPHI_BIGTABLE_PATH] [--optional-config OPTIONAL_CONFIG] optional arguments: -h, --help show this help message and exit --environment ENVIRONMENT, -e ENVIRONMENT name of environment (dev, prod, etc) --input INPUT, -i INPUT uow download parameters (json string) --download-concurrency DOWNLOAD_CONCURRENCY, -c DOWNLOAD_CONCURRENCY max number of worker for downloader --max-download-retries MAX_DOWNLOAD_RETRIES number of download retries --rds-secrets-key RDS_SECRETS_KEY RDS Postgres secrets key --dsp-settings-path DSP_SETTINGS_PATH S3 path to dsp settings --dsp-complete-criterias-path DSP_COMPLETE_CRITERIAS_PATH S3 path to dsp complete_criterias --sqs-arns-meta SQS_ARNS_META list of sqs arns for meta flow --sqs-arns-split SQS_ARNS_SPLIT list of sqs arns for split and compress flow --debug, -d debug mode --humanize human readable output (not in JSON) --https-proxy-host HTTPS_PROXY_HOST https proxy hostname, like dev-kong.delphiplatform.io --aws-region AWS_REGION AWS Region --aws-endpoint AWS_ENDPOINT AWS Endpoint --sfn-task-token SFN_TASK_TOKEN Step Function task token --chunk-size CHUNK_SIZE, -b CHUNK_SIZE chunk size --upload-concurrency UPLOAD_CONCURRENCY max number of workers for multipart uploader --upload-queue-size UPLOAD_QUEUE_SIZE max upload queue suze --bucket-source BUCKET_SOURCE S3 bucket with source files (if set, it substitutes downloading process from dsp with downloading from S3) --bucket-source-key BUCKET_SOURCE_KEY Path to file in S3 --bucket-archive BUCKET_ARCHIVE S3 bucket for raw untouched files --bucket-archive-quarantine BUCKET_ARCHIVE_QUARANTINE S3 quarantine bucket for raw untouched files --bucket-decompressed BUCKET_DECOMPRESSED S3 bucket for decompressed files --bucket-decompressed-quarantine BUCKET_DECOMPRESSED_QUARANTINE S3 quarantine bucket for decompressed files --bucket-corrupted BUCKET_CORRUPTED S3 bucket for corrupted files --enable-historical-validation Pass to enable report validation based on its historical sizes --validation-rows-number VALIDATION_ROWS_NUMBER number of rows to validate --validation-rate VALIDATION_RATE validation rate --validation-threshold-min-file-size VALIDATION_THRESHOLD_MIN_FILE_SIZE historical validation threshold min file size --download-queue-size DOWNLOAD_QUEUE_SIZE size of download queue --apollo-db-username APOLLO_DB_USERNAME --apollo-db-password APOLLO_DB_PASSWORD --apollo-db-host APOLLO_DB_HOST --apollo-db-port APOLLO_DB_PORT --apollo-db-name APOLLO_DB_NAME --apollo-worker-complete-days-offset APOLLO_WORKER_COMPLETE_DAYS_OFFSET --delphi-db-username DELPHI_DB_USERNAME --delphi-db-password DELPHI_DB_PASSWORD --delphi-db-host DELPHI_DB_HOST --delphi-db-port DELPHI_DB_PORT --delphi-db-name DELPHI_DB_NAME --delphi-bigtable-path DELPHI_BIGTABLE_PATH --optional-config OPTIONAL_CONFIG optional config (json string) # Examples slz-downloader -e dev -c 10 \ --content-status-table-name=dev-delphi-slz_content_status \ --bucket-archive=dev-sme-data-archive \ --bucket-decompressed=dev-sme-data-decompressed \ --bucket-corrupted=dev-delphi-sme-data-corrupted \ --bucket-archive-quarantine=dev-delphi-quarantine-sme-data-archive \ --bucket-decompressed-quarantine=dev-delphi-quarantine-sme-data-decompressed \ --dsp-settings-path=s3://stage-delphi-configs/dsp-specific-settings.json \ --dsp-complete-criterias-path=s3://stage-delphi-configs/dsp-complete-criterias.json \ --sqs-arns=https://sqs.us-east-1.amazonaws.com/475275892927/dev-delphi-reports-meta \ --rds-secrets-key=delphi/dev/slz/storage/pg_proxy/user \ --migration-stage=2 \ --debug --json \ -i "{\"dsp\":\"apple\"}" Job example: ``` { "dsp": "apple", "uow_id": "", "report_type": "", "subtype": "", "version": "", "licensor": "", "extension": "", "report_date": "2019-01-02", "config_bucket": "dev-delphi-configs", "contexts": [], "job_id": "test_job_id" } ``` # Scripts Default runner is an `slz-downloader`. If you need to create a special runner for your needs that relies on the `slz-downloader` package, you can add your own. Put it into `slz_downloader/scripts/`, so it will be a part of a package. Then add it into `setup.py::config.entry_points`. Now you can run script on your local machine with name that you've set in entry_points. Or you can use docker: docker run --rm -it -e PACKAGE= -v ~/.aws/:/app/.aws/ slz_downloader: ## apollo-backfill Command: + exec ./venv/bin/apollo-backfill usage: apollo-backfill [-h] -sd START_DATE -ed END_DATE -e {dev,qa,stage,prod} -d {apple,spotify} -r {playlists,playlist_ranking,charts,playlist_track_position} [--debug] [--humanize] apollo-backfill: error: the following arguments are required: -sd/--start-date, -ed/--end-date, -e/--env, -d/--dsp, -r/--report-type Example: docker run -it -e PACKAGE=apollo-backfill -v ~/.aws/:/app/.aws/ slz_downloader:DLP-3017 -sd 2020-05-10 -ed 2020-05-11 -r charts -d apple -e qa