# Data health config generator -  SLZ [gen_config](https://github.com/filtr/delphi-config/tree/master/src/gen_config) lib wrapper used for data health config generation

## Usage(run script)

    1. gen --help
    usage: gen [-h] --input INPUT [--debug]

	--yaml-dir-path INPUT		  Abs Path to dir with yaml files
	--config-dir-path INPUT		  Abs Path to dir with config file
	--config-name INPUT		  config file name(without extension) or "all" if you want to generate all configs at one time
	--environment -e INPUT		  env

    optional arguments:
    -h, --help            show this help message and exit
    --debug               additional log level

    2. check --help
    usage: check [-h] --input INPUT [--debug]

	--config-path INPUT		  Abs Path to config file
	--schema-path INPUT		  Abs Path to validation schema file


    optional arguments:
    -h, --help            show this help message and exit
    --debug               additional log level

## Examples

    1. gen --yaml-dir-path "/SONY/delphi-data-health-configs/files" \
    --config-dir-path "/SONY/delphi-data-health-configs/configs/dev" --config-name "dsp_config" \
    --environment "dev" --debug

    2. check  --schema-path "/SONY/delphi-data-health-configs/files/dsp_config/schema.json" \
    --config-path "/SONY/delphi-config/data_health_config.json" \
    --debug


## Make commands

- `make venv` - create local virtual env(base.pip dependencies)
- `make testenv` - create local virtual env(test.pip dependencies)
- `make devenv` - create local virtual env(dev.pip dependencies)
- `make test` - run tests
- `make lint` - run pylint check
- `make build` - build package
- `make upload` - upload package on PyPI
- `make clean` - delete unnecessary files
- `make gen/all` - generate set of configs for all envs
- `make gen/<env>` - generate set of configs for a specified environment

- `make docker/login`  - login to AWS docker, aws configuration need to be set first
- `make docker/test/image` - pull image used for tests execution
- `make docker/test` - run tests in docker
- `make docker/lint` - run pylint check in docker
- `make docker/build` - build package in docker
- `make docker/upload` - upload package on PyPI from docker

- `make docker/image/build` - build Docker image
- `make docker/image/push` - push an image to a registry
- `make docker/image/pull` - pull an image from a registry

- `make docker/gen/all` - generate set of configs for all envs inside of a docker container
- `make docker/gen/<env>` - generate set of configs for a specified environment inside of a docker container

(when run `make test`, `make lint`  command, all dependencies from requirements/test will be installed.
When run `make venv` base.pip dependencies will be installed)

### configure AWS:
- to check if aws configuration was already set: `aws configure list`
- to configure `https://docs.aws.amazon.com/en_us/cli/latest/userguide/cli-chap-configure.html`

## HOW TO ADD A NEW CONFIG:
1.  To add a new config, structure of files must be:

        files/     - folder for all initial config files
            <name_of_config>/   - this folder should be named exactly as desired result name of config
                params/
                    dev.yaml
                    qa.yaml
                    stage.yaml
                    prod.yaml
                templates/
                    main.yaml
                    ...   - here should be all templates in .yaml with initial data
                schema.json

    flow of making configs is:

        params_from_params/ folder per environement applying to ->
            all templates from templates/ folder except main.yaml, convert this to iterable ->
                apply iterable to main.yaml ->
                    validate result with schema.json

2. If your config is not environment specific, you should not create folder params/ so your file structure will be:

        files/
            <name_of_config>/
                templates/
                    main.yaml
                    ...
                schema.json


## Config yaml structure
Description of params see [here](https://github.com/filtr/delphi-data-health-configs)

#### delphi-data-health-configs.json yaml structure
    - data_type: ads_data
      dsps: [google, facebook]
      queries:
      - query: {{ ads_data_select }}
        db_type: postgresql
        db_secret_key: 'delphi/{env}/api/db_params'
        propagate_result: false
      - query: {{ ads_data_insert }}
        db_type: postgresql
        db_secret_key: 'delphi/{env}/api/db_params'
        propagate_result: false