# Fingerprint Capture
Fingerprint Capture is a process that fingerprints The Orchard's audio assets and stores those fingerprints in a database on a track level.

## Fingerprinting Technology
- [Echoprint Codegen](https://github.com/spotify/echoprint-codegen) is used to generate fingerprints.
- Those fingerprints can be loaded into [Echoprint Server](https://github.com/spotify/echoprint-server) to match fingerprints of different audio files.

## High-Level Flow
- [Flow Diagram](fingerprint_capture_flow.png)

1. Read messages from SQS queues. These messages are generated by [Direct Delivery](https://github.com/theorchard/direct_delivery) or by [Fingerprint Sweeper](https://github.com/theorchard/fingerprint-sweeper)
2. Call Echoprint Codegen to generate the fingerprint for the filename specified in each message.
3. Load the fingerprint into the fingerprint_capture database. If Codegen succeeds, load the data into the track_fp table. Otherwise load codegen_error table.

## Queue Message Format
- The expected message format is defined in the [Validictory schema](https://github.com/theorchard/fingerprint-capture/blob/master/fpcapture/validation_schema/message_schema.py)

## SQS Queues
- Process reads from both env-fp_capture and env-fp_capture_backfill (where env can be dev, qa, or prod).
- Direct Delivery writes messages to fp_capture queue when customers upload new audio assets through Workstation.
- Fingerprint Sweeper writes messages to fp_capture_backfill queue to fingerprint existing audio assets.

## Environment Variables
| Env Variable  | Purpose                                  |
| ------------- |------------------------------------------|
| FPC_DB_URL    | fingerprint_capture db connection string |
| LOGGER_DSN    | Loggly connection string                 |
| SENTRY_DSN    | Sentry connection string                 |
| Environment   | current environment such as qa or prod   |

## Deployment
- Detailed deployment info can be found in https://goo.gl/TnXrtv.

### fingerprint_capture database
- The fingerprint_capture database is an RDS instance of MySQL.
- Dev server: dev-wpcorpsite.cb22xqmk0y0q.us-east-1.rds.amazonaws.com
- QA server: prod-compilations-wp.cb22xqmk0y0q.us-east-1.rds.amazonaws.com
- [DDL](https://github.com/theorchard/database/tree/master/fingerprint_capture/build/changelog/ddl)

### Fingerprint capture process
- Runs on Windows VM's. It was deployed to Windows to simplify access to the audio files, which are on other Windows servers (known as Mezzanine).
- Windows VM's are hosted in Ashville, NC using VMware ESX. They are hosted in Ashville because the audio assets are initially stored there (and later synced to New York).
- Fingerprint Capture is a Python 3 process that is started by a scheduled task (Windows equivalent of cron) every 60 seconds and exits after 60 seconds.

## Scaling
- This process can be scaled by either adding more VM's or increasing the number of processes per VM.
- Since requests come in through a queue, any number of processes can consume from the queue.

## Logging and Alerting
- Logs are written to [Loggly](https://orchard.loggly.com/search)
- Exceptions are sent to [Sentry](https://sentry.io/the-orchard/fingerprint-capture-2/)

## [Jenkins Jobs](http://jenkins.theorchard.com:8080/view/fingerprint)
### Deploy
- Deploy code from Github fingerprint-capture repo to pool of Windows VMs on target environment
- [QA](http://jenkins.theorchard.com:8080/view/fingerprint/job/qa-fingerprint_capture_deploy/)
- [Prod](http://jenkins.theorchard.com:8080/view/fingerprint/job/prod-fingerprint_capture_deploy/)

## Tech Design
- For detailed background information, see tech design at https://goo.gl/uh1Dzf.

## Fingerprint Sweeper
- See [Fingerprint Sweeper Documentation](https://github.com/theorchard/fingerprint-sweeper/blob/master/doc/index.md) for details of producer process that writes to fp_capture_backfill queue.

## Direct Delivery Integration Point
- See [DD Processwav.php module](https://github.com/theorchard/direct_delivery/blob/master/library/Orchard/Assetreceived/Processuploadedasset/Processwav.php) for point at which Direct Delivery pushes message to fp_capture queue.