# Fingerprint Sweeper
Fingerprint Sweeper is a process that sweeps up tracks that were not yet fingerprinted and initiates their fingerprinting by pushing messages to a queue. It serves 2 purposes:

1. Ongoing Sweeper Process: fingerprint tracks that were missed by [Direct Delivery](https://github.com/theorchard/direct_delivery). This might happen if Direct Delivery experienced an intermittent problem writing to SQS. It can also happen if someone loads audio assets into Orchard systems using the [Ripper tool](https://github.com/theorchard/direct_delivery/blob/master/get_assets.php), which bypasses Direct Delivery logic.
2. Backfill Process: a one-time process to fingerprint tracks that already existed in The Orchard catalog prior to fingerprint capture going live.

## High-Level Flow
- [Flow Diagram](fingerprint_sweeper_flow.png)

1. Query art_relations.track table to get tuids (track.id) in given time range. This is the tuid population for a given run.
2. Query fingerprint_capture.track_fp and fingerprint_capture.codegen_error tables to get the tuids in the population that have not yet been fingerprinted.
3. Generate SQS message for each tuid that needs to be fingerprinted. This involves getting the filename (ip + full path of audio file) from Direct Delivery and getting the UPC from art_relations.track.
4. Push messages onto SQS queue called env-fp_capture_backfill, where env can be dev, qa, or prod.
5. Once messages are on fp_capture_backfill queue, they will be picked up by [fingerprint-capture](https://github.com/theorchard/fingerprint-capture/blob/master/doc/index.md) process to be fingerprinted.

## Time Window
- Sweeper looks for tuids in art_relations.track that were inserted or updated within a given time window. It uses the art_relations.track.last_updated field to determine this.
- The time window is defined by command line arguments --daterange_start_days and --daterange_end_seconds. For detailed usage, run ```python fpsweeper/logic/sweeper.py -h```

## Backfill Switch
- Sweeper takes a --backfill option. This option switches the query used to get the tuid population. If set, it will restrict the results to tracks where releases.release_status is 'in_content'.
- If the --backfill option is omitted, Sweeper will process all tracks in the time window regardless of releases.release_status. This is for the ongoing Sweeper process.
- See [track queries](https://github.com/theorchard/fingerprint-sweeper/blob/master/fpsweeper/model/query/track.py) for more detail.

## Chunking
- Sweeper processes tuids in batches of 10,000 to keep query sizes from exceeding a limit.
- Sweeper pushes 10 messages to SQS at a time, which is the maximum currently allowed by SQS.
- These values are configurable and are defined in [config.py](https://github.com/theorchard/fingerprint-sweeper/blob/master/fpsweeper/config.py) as TUID_CHUNK_SIZE and SQS_MAX_BATCH_SIZE.

## Deployment
- Is a Python 3 process that runs on Jenkins on a bam agent every 2 hours, 7 days per week.
- Time window of the scheduled job is midnight of 5 days ago to 3600 seconds ago.
- Detailed deployment info can be found in https://goo.gl/TnXrtv.

## Environment Variables
| Env Variable                    | Purpose                                  |
| ------------------------------  |------------------------------------------|
| FPC_DB_URL                      | fingerprint_capture db connection string |
| AR_DB_URL                       | art_relations db connection string       |
| DD_DB_URL                       | direct_delivery db connection string     |
| LOGGER_DSN                      | Loggly connection string                 |
| SENTRY_DSN                      | Sentry connection string                 |
| Environment                     | current environment such as qa or prod   |
| PYTHONPATH                      | path to fingerprint-sweeper              |
| DD_STORAGE_PHYSICAL_LOCATION_ID | 7 for QA; 1 for Prod Ashville;           |

## Logging and Alerting
- Logs are written to [Loggly](https://orchard.loggly.com/search)
- Exceptions are sent to [Sentry](https://sentry.io/the-orchard/fingerprint-sweeper-m4/)

## [Jenkins Jobs](http://jenkins.theorchard.com:8080/view/fingerprint)

### Deploy
- Deploy code from Github fingerprint-sweeper repo to Linux VM on target environment
- [QA](http://jenkins.theorchard.com:8080/view/fingerprint/job/qa-fingerprint_sweeper_deploy)
- [Prod](http://jenkins.theorchard.com:8080/view/fingerprint/job/prod-fingerprint_sweeper_deploy)

### Backfill
- Run fingerprint backfill (pushing messages to fp_capture_backfill queue) for given time window on target environment
- This applies to in_content tracks only
- [QA](http://jenkins.theorchard.com:8080/view/fingerprint/job/qa-fingerprint-backfill)
- [Prod](http://jenkins.theorchard.com:8080/view/fingerprint/job/prod-fingerprint-backfill)

### Backfill Monitor
- Monitor progress of backfill by printing 5 most recent rows of codegen_error & fp_sweeper_log tables; and min_tuid, max_tuid, count of codegen_error & track_fp tables
- [QA](http://jenkins.theorchard.com:8080/view/fingerprint/job/qa-fingerprint-backfill-monitor)
- [Prod](http://jenkins.theorchard.com:8080/view/fingerprint/job/prod-fingerprint-backfill-monitor)

## Tech Design
- For detailed background information, see tech design at https://goo.gl/uh1Dzf.

## Fingerprint Capture
- See [Fingerprint Capture Documentation](https://github.com/theorchard/fingerprint-capture/blob/master/doc/index.md) for details of process that consumes from fp_capture_backfill queue.