# Compressed formats comparison

In order to safe money on DynamoDB we might store main data in compressed format.

E.g.

    pk: string,
    sk: string,
    isrc: string,
    date: string,
    country_code: string,
    data: protobuf/snappy

The goal of the task is to check:

1) Which option gives the best compression rate: protobuf, snappy, gzip, smth else?

2) Can protobuf be efficiently compressed at all? Does double wrapping snappy(protobuf(data)) makes sense?

3) How much space will we save by compressing data? Does it make sense?

4) What would be compression/decompression overhead? What is the performance impact?

## Protocol:

- protobuf
- msgpack
- cbor
- json

## Compression:

- bz2
- zlib
- snappy
- none


# How it works:

I've generated a json message in an Apple Demographic record format.
Script reads file and builds object it can serialize/deserialize.
Protobuf requires to describe records explicitly, so `./proto/` contains data.proto file as well as generated python classes.
Python executes all combination of protocols and compression algorithms N times and repeats M times. I use median of serialize/deserialize time measures.


# Run locally

Create virtualenv:

    virtualenv -p python3 venv

Activate it:

    source ./venv/bin/activate

Generate report with default parameters

    make report

If you have updated proto-file, you should regenerate python representation.
`protoc` generator tool is required.

    make generate

Delete existing reports

    make clean
    
Analysis of the generated report lays in a `./comparison.ipynb` Jupyter notebook. 
You can open it with Jupyter or using GitHub

    jupyter notebook

Jupyter app will be opened in a browser and there you can open comparison notebook.

I've also added a copy of this report in `html` format. Please have a look at `./static/comparison.html`