# S3 copy

Python-based tool for data copy from S3 to S3 and from external URL to S3

## Dependencies
python 3.10


## Installation

Run the following commands:
```
make devenv
```

## How it works

    usage: s3-uploader [-h] [--source SOURCE --destination] DESTINATION
                            [--decompressed DECOMPRESSED] [--fin FILEINPUT] [--debug]

#### Examples

From external URL to S3
```
s3-uploader \
  --source https://ws.spotify.com/analytics/api/resource/path?oauth_token=my_token \
  --destination s3://my-s3-bucket/spotify/resource.gz
```

From external URL to S3 with simultaneous decompression
```
s3-uploader \
  --source https://ws.spotify.com/analytics/api/resource/path?oauth_token=my_token \
  --destination s3://my-s3-bucket/spotify/resource.gz \
  --decompressed s3://my-s3-bucket/spotify/resource.csv
```

From S3 to S3
```
s3-uploader \
  --source s3://other-s3-bucket/spotify/resource.zip \
  --destination s3://my-s3-bucket/spotify/resource.zip
```

From S3 to S3 with simultaneous decompression
```
s3-uploader \
  --source s3://other-s3-bucket/spotify/resource.gz \
  --destination s3://my-s3-bucket/spotify/resource.gz \
  --decompressed s3://my-s3-bucket/spotify/resource.csv
```

From Stream to S3
```
cat path_to_test_file.gz | s3-uploader \
  --fin - \
  --destination s3://my-s3-bucket/spotify/resource.gz
  --decompressed s3://dev-sme-stream-transfer/test_d.tsv
```

### Environment variables
    READ_CHUNK (default: 1024)
    UPLOAD_PART_LEVEL (default: 5 * 1024 ** 2)
    QUEUE_MAX_SIZE (default: 2)

### Linter
```
make lint
```

### Tests
```
make test
```