# sfpoc: simple url to s3 download Python module

# Local

For the local execution you need to have installed:

- pip
- python3.7+
- make

Create virtualenv:

    make venv

Now you can run command:

    python -m sfpoc --help
    usage: __main__.py [-h] --url URL
                    --s3-bucket S3_BUCKET_NAME
                    [--chunk-size CHUNK_SIZE]

    optional arguments:
    -h, --help            show this help message and exit
    --url                 the url to download file from
    --s3-bucket           name of the S3 bucket to files to
    --chunk-size          max size of a byte buffer used for streaming data

Example:

    python -m sfpoc \
        --url=http://sdm.lbl.gov/fastbit/data/star2000.csv.gz \
        --s3-bucket=stepf-poc

# AWS

Make sure you have installed `awscli` tool and set `AWS` credentials to push containers to `ECR`

Build container:

    make docker/build

Push container:

    make docker/push

How to start container:

    docker run -it 475275892927.dkr.ecr.us-east-1.amazonaws.com/sfpoc:latest \
        --url=http://sdm.lbl.gov/fastbit/data/star2000.csv.gz \
        --s3-bucket=stepf-poc

You should pass `AWS` credentials via environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`).