# README for datalytics

## Repository

 - `git clone git@github.com:theorchard/datalytics.git`
 - `git checkout cleanup`  # TODO: adjust final branch name


## Usage

All scripts are available in the `datalytics/bin` directory and are also
"linked" by easy-install into Anaconda's virtualenv. So changes are available
immediately and you do not have to reinstall the package after
`pip install -i https://pypi.theorchard.io/pypi/ -e .`.

Given the number of dependencies, it might be is easier to use the project via
Docker.

Building:

    docker build -t local/datalytics .

Running scripts:

    docker run -v `pwd`/datalytics:/var/app/datalytics -e ORCHARD_SFTP_SPOTIFY_RO_PRIVATE_KEY --env-file .env.shadow -ti local/datalytics market_share_spotify.py

Note: only some rarely changing environment variables are specified in the
``.env.shadow` file and SSH key is a multi-line, which Docker does not like.
So, you'll have to specify all other vars on the command line with
`-e VAR="value"`.

Running Tests:

    docker run --env-file .env.shadow -ti local/datalytics make pip_dev lint test

If you run tests and/or change requirements frequently, consider building a
separate image based on the regular one (local/datalytics, that you should 
also keep up to date):

    docker build -f Dockerfile-dev -t local/datalytics-dev .
    docker run -v `pwd`/datalytics:/var/app/datalytics -v `pwd`/tests:/var/app/tests --env-file .env.shadow -ti local/datalytics-dev make lint test


### Known issues

#### Multiline environment variables

Multiline env vars are not supported by the `docker run --env-file` option.

    docker: poorly formatted environment: variable '-----END RSA PRIVATE KEY-----"' has white spaces.

To fix this:

 - put vars to your env
 - remove such vars from the env file
 - specify individual vars on the command line

Example:

    set -a; source .env.shadow; set +a
    docker run -e ORCHARD_SFTP_SPOTIFY_RO_PRIVATE_KEY -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY --env-file .env.shadow -ti local/datalytics python datalytics/bin/market_share_deezer.py


#### Requirements versions conflicts/mismatches

There could be version compatibility issues like this:

    pkg_resources.ContextualVersionConflict: (requests 2.12.4 (/opt/conda/lib/python3.6/site-packages), Requirement.parse('requests==2.10.0'), {'owslogger'})

if you run scripts by using just the
executable name. Our `requirements.txt` has versions these scripts are known to
work with.

Until all those minor incompatibilities are fixed, we would have
to run scripts specifying the full path to skip version conflict checks:

    python datalytics/bin/market_share_deezer.py

One thing that might help to build a good list of non-conflicting dependencies
is `pip-compile` from the [pip-tools](https://github.com/jazzband/pip-tools) package.


## External Services/Resources

### Orchard SFTP

This is being referenced as FTP in some scripts, but it is actually an SSH File
Transfer Protocol server.

Hostname: ftp.theorchard.com
Port: 22
Authentication type: public-key
Username is script-specific

See relevant vars in `.env.shadow`.