# meta-force-takedown-via-srr

The current "Hybrid" feed sending data to Meta cannot perform a takedown to remove content from their system that we no longer want to claim rights for. Until the new "SRR" feed can be fully operational, we wish to use it to script takedowns in bulk using a file as input (one ISRC per line).

### Setup

#### Docker (Recommended)

1. Build the Docker image

```
$ docker compose build
```

2. Confirm you have access to read [this AWS secret value](https://us-east-1.console.aws.amazon.com/secretsmanager/secret?name=prod%2Flambda-sr-delivery-meta%2FMETA_SFTP_PKEY&region=us-east-1) (it is part of the "sound-recordings" application family). Only required for SFTP delivery.
3. Confirm you have access to read [this AWS secret value](https://us-east-1.console.aws.amazon.com/secretsmanager/secret?name=prod%2Flambda-sr-delivery-meta%2FMETA_SFTP_AWS_CREDENTIALS&region=us-east-1) (it is part of the "sound-recordings" application family). Only required for production S3 delivery.

#### Local (Without Docker)

1. Install virtual environment and packages

```
$ python -m venv env
$ source env/bin/activate
$ pip install -r requirements.txt
```

### Running

#### Docker

1. Activate your prod AWS profile (MFA required)

```
$ awsume prod-mfa
```

2. Place your ISRC file in the `data/` directory

```
$ cp /path/to/your/isrcs.txt data/
```

3. Run via Docker Compose

```
# Test S3 bucket (qa-sr-interm-delivery-bucket)
$ docker compose run meta-force-takedown-via-srr data/isrcs.txt --send-to-s3 True

# Production S3 bucket (orchard-audio-delivery)
$ docker compose run meta-force-takedown-via-srr data/isrcs.txt --send-to-s3 True --s3-test ""

# SFTP delivery
$ docker compose run meta-force-takedown-via-srr data/isrcs.txt
```

:bulb: The `data/` directory is mounted into the container via `docker-compose.yaml`, so any file placed in `data/` is accessible inside the container.

:warning: **Known issue:** The `--s3-test` flag uses `type=bool` in argparse, which means `--s3-test False` does **not** work as expected (`bool("False")` evaluates to `True` in Python). Use `--s3-test ""` to disable the safety catch for production S3 delivery.

#### Local (Without Docker)

1. Activate your prod AWS profile (MFA required)

```
$ awsume prod-mfa
```

2. Activate virtual environment

```
$ source env/bin/activate
```

3. Run script with filename of ISRCs (one per line) as positional argument.

```
$ python run.py data/isrcs.txt --send-to-s3 True
```

:bulb: Note the ISRCs being printed to stdout, you can restart from the middle of the file using `--since` if an error happens.

#### Delivery Modes

| Command | Destination | AWS Creds Used | Bucket / Server |
|---|---|---|---|
| `--send-to-s3 True` | Test S3 | Your local AWS creds | `qa-sr-interm-delivery-bucket` |
| `--send-to-s3 True --s3-test ""` | Production S3 | Meta creds from Secrets Manager | `orchard-audio-delivery` |
| _(no flag)_ | SFTP | SFTP key from Secrets Manager | `s-eaeb3091ded4453fa.server.transfer.us-west-2.amazonaws.com` |

#### Full Options
```
usage: run.py [-h] [--secret-arn SECRET_ARN] [--sftp-server SFTP_SERVER]
              [--sftp-user SFTP_USER] [--sftp-port SFTP_PORT]
              [--sftp-dir SFTP_DIR] [--batch-size BATCH_SIZE] [--sleep SLEEP]
              [--since SINCE] [--send-to-s3 SEND_TO_S3] [--s3-test S3_TEST]
              filename

positional arguments:
  filename              filename containing ISRCs

options:
  -h, --help            show this help message and exit
  --secret-arn SECRET_ARN
                        AWS secrets manager ARN of meta SFTP key (default: arn:aws:secretsmanager:us-east-1:437795906767:secret:prod/lambda-sr-delivery-meta/META_SFTP_PKEY-bNPPQU)
  --sftp-server SFTP_SERVER
                        Meta SFTP server (default: s-eaeb3091ded4453fa.server.transfer.us-west-2.amazonaws.com)
  --sftp-user SFTP_USER
                        Meta SFTP username (default: orchard)
  --sftp-port SFTP_PORT
                        Meta SFTP port (default: 22)
  --sftp-dir SFTP_DIR   Meta SFTP directory to upload into (default: fingerprinting)
  --batch-size BATCH_SIZE
                        Uploads to process before sleeping (default: 100)
  --sleep SLEEP         Seconds to sleep between batches (default: 10)
  --since SINCE         Start processing records after this ISRC (default: None)
  --send-to-s3 SEND_TO_S3
                        Use to send to S3, not SFTP (default: False)
  --s3-test S3_TEST     Use to send to PDE S3 as a test. Requires explicit override to send to production (default: True)
```

### Jenkins

This script can be run via Jenkins pipeline using the provided `Jenkinsfile`. The pipeline:

1. Accepts an ISRC file upload via the Jenkins UI
2. Builds the Docker image and runs the container
3. Supports three delivery modes: `s3_test`, `s3_production`, `sftp`
4. Injects AWS credentials at runtime via IAM role assumption
5. Sends Slack notifications on success/failure with delivery destination details

See the `Jenkinsfile` for full configuration and parameters.

#### Parallelization

##### Via Shell (Legacy, SFTP only)

:warning: This method is for local SFTP delivery only and is not supported in Docker or Jenkins.

A helper script exists to split up an input file so that parallel jobs can be run. Logs will be used to restart jobs that die before completing as long as the same exact parameters are used when restarting.

:warning: Assumes running on Mac and [coreutils](https://formulae.brew.sh/formula/coreutils) is installed so `gsplit` is available

##### Start
```
$ ./run_jobs.sh <params>
```
* $1 => input filename with complete data set
* $2 => integer line number offset to start processing at
* $3 => integer size of batch in total
* $4 => unique name of batch
* $5 => integer number of workers to spawn

##### Monitor

:warning: Assumes [watch](https://formulae.brew.sh/formula/watch) is installed
```
$ watch -n 15 -d "pgrep python | wc -l && wc -l batches/<BATCH_NAME>/*.log"
```