## Backfill script parameters

    optional arguments:
      -h, --help            show this help message and exit
      --env {dev,qa,stage,prod}
      --start-date START_DATE
                            All units till this day are for backfill: in format YYYYMMDD
      --source-bucket SOURCE_BUCKET
                            [OPTIONAL] S3 bucket with decompressed files
      --dest-bucket DEST_BUCKET
                            [OPTIONAL] S3 bucket to transfer files of already completed units
      --uow-limit UOW_LIMIT
                            How many backfill units could be processed simultaneously
      --completeness-rate COMPLETENESS_RATE
                            [OPTIONAL] range [0; 1]. [DEFAULT: 0.7]
                            How many CS from all should be processed to count this unit as completed for unit-limit
      --completeness-threshold COMPLETENESS_THRESHOLD
                            [OPTIONAL] range [0; 1]. [DEFAULT: 1.0]
                            cs_complete = count(cs.completeness_status in (COMPLETE, COMPLETE_VOLATILE))
                            cs_total = count(cs)
                            completeness_rate:float = cs_complete/cs_total
                            As soon as completeness_rate >= completeness_threshold, unit of works will be moved to COMPLETE
      --debug               additional log level
      --json                json output



## Launch backfill (shortcuts based)

Create a job to be triggered by crontab on a schedule:

    touch jobs/<your_backfill_job>.sh
    chmod +x jobs/<your_backfill_job>.sh

Feel free to copy the existing job and modify for your needs.

Build and upload the docker image containing this script to use it in Production environment:

    make docker/image/build && make docker/image/upload

Create a link to your backfill job.

Note: `check_units` is a default job name baked in a Makefile:

    ln -s check_units.sh jobs/<your_backfill_job>.sh


Setup the crontab job to launch the script:

    make crontab/add

Stop the contact job:

    make crontab/clean


## Launch multiple backfill processes (custom-made)

manage your crontab schedule manually:

  crontab -l ; echo "*/2 * * * * sh $(PWD)/jobs/ae.sh
  crontab -l ; echo "*/2 * * * * sh $(PWD)/jobs/tiktok_theorchard.sh
  ...

or using the default editor:

  crontab -e

or using the vim:

  VISUAL="vim" crontab -e