# AE members data direct load

This script loads report data for specified member IDs from AE API.
Only the following reports are supported:
- membersall
- memberslogin
- activityfeed
- membersextended

The script load data from API and save then to local file system or AWS S3
in decompressed bucket's like folder structure.
Members are specified in the CSV file in the following format:
- Text file with UTF-8 encoding
- No BOM
- No headers
- Unix-like line breaks (\\n)
- Number of columns from 1 to 3: Member ID, Licensor, Context
- Column separator - comma, can be omitted if the following columns are not specified:
    - Example 1: '1234\\n' - only data for member with ID = 1234 will be loaded for all licensor and contexts
    - Example 2: '1234,,US\\n' - only data for member with ID = 1234 and context 'US' for all licensors will be loaded
    - Example 3: '1234,sme,US\\n' - only data for member with ID = 1234, licensor = 'sme' and context 'US' will be loaded

The script load members data files partially by batches. Each batch is the set of member IDs.
When all data for specific set of member IDs are loaded the script write the data into file
and saves loaded member IDs into the log file in order not to load the same data more than once.
If the script fails it read loading log during initialization and continue loading members.

## Make commands

- `make venv` - create local virtual env(base.pip dependencies)
- `make testenv` - create local virtual env(test.pip dependencies)
- `make devenv` - create local virtual env(dev.pip dependencies)
- `make test` - run tests
- `make lint` - run pylint check
- `make build` - build package
- `make upload` - upload package on PyPI
- `make clean` - delete unnecessary files

- `make docker/login`  - login to AWS docker, aws configuration need to be set first
- `make docker/test/image` - pull image used for tests execution
- `make docker/test` - run tests in docker
- `make docker/lint` - run pylint check in docker
- `make docker/build` - build package in docker
- `make docker/upload` - upload package on PyPI from docker

- `make docker/image/build` - build Docker image
- `make docker/image/push` - push an image to a registry
- `make docker/image/pull` - pull an image from a registry

(when run `make test`, `make lint`  command, all dependencies from requirements/test will be installed.
When run `make venv` base.pip dependencies will be installed)

### Configure AWS:
- to check if aws configuration was already set: `aws configure list`
- to configure `https://docs.aws.amazon.com/en_us/cli/latest/userguide/cli-chap-configure.html`

## Example usage:

### Execute in docker

    docker run -it --rm -v ~/.aws:/app/.aws ae_direct_load:DEXP-2866 <your params>

### Execute locally

    make venv
    source venv/bin/activate
    ae-direct-load <your params>

### Get help (options description)

    make venv
    source venv/bin/activate
    ae-direct-load --help


### How to execute in AWS

## Local host
Container should be pre-built (on your local machine possibly):

    make docker/image/build
    make docker/image/push

## Remote host
Connect and create a separate `tmux` session:

    ssh ubuntu@10.32.14.76
    tmux new -s <yourname>

There is a GIT repository that contains all the util scripts. Pick your branch and proceed:

    cd ~/delphi-util/
    git fetch --all
    cd ./scripts/ae_direct_load
    make docker/image/pull
    exec/docker/<yourscript>&

Check logs. Logfile location should be defined in <yourscript>:

    tail -f ~/<yourscript log>

Stop processing:

    docker stop ae_memberslogin