# sync-cleanup

this is a script that's designed to clean up failed ownership ingest sync errors.

where 'clean up' means:
1. find the original SFN execution which created the error file
2. re-run it with exactly the same input
3. if it successfully runs, find the *new* error file that was created, and
4. IF the new error file has the same file size as the local file, delete it from S3

why step 4? because we know that re-running the sync process will regurgitate the same errors that were produced in the original run. If the new file has the same file size as the old one, then we know that no *new* errors were generated, and the new execution did its job. 

## caveat 1

sometimes the original execution may have failed quite early in the process, in which case re-running it may produce new rows in the error file (and thus cause the file size to be different). 

This isn't necessarily a problem - I've seen cases where a re-run of the same execution which had 504 Gateway Timeout errors in the first run, then went on to run successfully BUT produced some (legitimate) data validation issues in the error file.

This script will NOT delete the new error file in this case, and will leave it for manual review. Sorry.

## Usage

pre-requisites: `nvm use` (i think this script uses some node 22 stuff but i forget), `npm install`, `awsume prod` (or equivalent)

`node -r ts-node/register sync-cleanup.ts <list of input files>`

the way I suggest to run this is:

put all the error files you want to deal with in a specific folder, like `~/tmp/cleanup-gateway-timeout/`, then I can do something like (from this the sync-cleanup folder):

```bash
node -r ts-node/register sync-cleanup.ts $(find ~/tmp/cleanup-gateway-timeout/)
```

(note that this probably only scales to like ~100 or so files because of limits to command line argument length)

### Example output

```
  ✅ successfully re-ran step function and removed original error file for UUID: f3612709-dc5f-4549-a44b-7f8bdca90787
  ❌ re-run completed for UUID 22b7f566-32ce-4c58-ba77-020acdae65b3 but file sizes don't match - nothing was deleted
  ✅ successfully re-ran step function and removed original error file for UUID: a907627a-e4c1-4f7b-ad0f-b11e39e16b7f
  ❌ re-run completed for UUID fb2f1d86-fb74-4227-b395-2b1b6ef4895c but file sizes don't match - nothing was deleted
  ✅ successfully re-ran step function and removed original error file for UUID: 8f3d6144-d013-476d-83a4-ca708f070c15
  ✅ successfully re-ran step function and removed original error file for UUID: 484b6c55-36cf-4921-a8c7-e8a43bbddb14
```

these two failed UUIDs I would then need to review manually and check the file for new technical errors vs legitimate data validation errors.