# Backload tool

In case backfill/reiongestion is required you can use backload tool. 

## How to use:

Schedule backfill Job: 

https://scheduler.theorchard.io/job/swf-backload-scheduler-exec/build


## Monitor

Sample queries to monitor backfill process:
```sql

-- start soon
select * from facts.prod.backload_tasks where processed_at is null order by created_at desc;
select * from facts.prod.backload_tasks where processed_at is null and flow = 'pandora' order by created_at desc;

-- earliest dates to ingest
select * from facts.prod.backload_tasks where processed_at is null and flow = 'deezer' order by date;

-- pending tasks
select count(*) from facts.prod.backload_tasks where processed_at is null order by date;

-- recently processed
select count(*) from facts.prod.backload_tasks where processed_at is not null and processed_at > '2022-07-18' order by processed_at desc ;
select * from facts.prod.backload_tasks where processed_at is not null and flow = 'amazon_digital_services'  order by processed_at desc;

-- processed in the date range
select * from facts.prod.backload_tasks where processed_at between '2020-12-01' and '2020-12-02' order by processed_at desc ;

-- check the date
select * from facts.prod.backload_tasks where flow = 'amazon_music' and date = '2018-11-11' --and processed_at is null
order by created_at desc;

```

Please note that execution status of SWF job should be checked independently by Monty or
backloader's monitor.

### Monitoring tool

Monitoring job load statuses of SWF task and publishes it to Google Spreadsheet.
We used this to tool to monitor SME backfilling.
the tool can be adjusted to monitor other statuses.

* Spreadsheet https://docs.google.com/spreadsheets/d/10tVScKv6IbJ6yWA9CHWD5eNCfd8jtSCnV2hhpIfstwo/edit#gid=51186277
* Monitoring Job: https://scheduler.theorchard.io/job/swf-backload-monitoring-exec/
* Source code: https://github.com/theorchard/swf-feed-ingestion/blob/master/feed_ingestion/bin/backload_monitoring.py

## See also

* Automatic runner Job: https://scheduler.theorchard.io/job/swf-backload-runner-exec/
* Source code: https://github.com/theorchard/swf-feed-ingestion/blob/master/feed_ingestion/bin/backload_scheduler.py
* Limit for parallel execution: https://github.com/theorchard/swf-feed-ingestion/blob/master/feed_ingestion/bin/backload_executor.py#L18