# Template for Job

Job is piece of code that should take incoming parameters, do some work and die.
This template should be used for command-line tools, migration scripts, data processing scripts.

## Make commands

- `make venv` - create local virtual env(base.pip dependencies)
- `make testenv` - create local virtual env(test.pip dependencies)
- `make devenv` - create local virtual env(dev.pip dependencies)
- `make test` - run tests
- `make lint` - run pylint check
- `make build` - build package
- `make upload` - upload package on PyPI
- `make clean` - delete unnecessary files

- `make docker/login`  - login to AWS docker, aws configuration need to be set first
- `make docker/test/image` - pull image used for tests execution
- `make docker/test` - run tests in docker
- `make docker/lint` - run pylint check in docker
- `make docker/build` - build package in docker
- `make docker/upload` - upload package on PyPI from docker

- `make docker/image/build` - build Docker image
- `make docker/image/push` - push an image to a registry
- `make docker/image/pull` - pull an image from a registry

(when run `make test`, `make lint`  command, all dependencies from requirements/test will be installed.
When run `make venv` base.pip dependencies will be installed)

### configure AWS:
- to check if aws configuration was already set: `aws configure list`
- to configure `https://docs.aws.amazon.com/en_us/cli/latest/userguide/cli-chap-configure.html`

## Usage(run script)

    AppleMusicChartsScraper --help
    usage: AppleMusicChartsScraper [-h] --input INPUT [--debug] [--json]

	--input INPUT		  some text

    optional arguments:
    -h, --help            show this help message and exit
    --debug               additional log level
    --json                json output

## Examples

    AppleMusicChartsScraper --input 123 --json


## Developers guide

Use `pre-commit` for git hooks managing (https://pre-commit.com/#installation). Hook configuration
can be found at .pre-commit-config.yaml. `isort` and `yapf` tools are included to run as a part
of precommit hook.

To register a new hook run:
```shell script
pre-commit install
```

To run hooks without code committing:
```shell script
pre-commit run
```

Sometimes these tools can have a conflict regarding import sorting. In this case consider to
enable `include_trailing_comma` option at `setup.cfg` file.

Don’t forget to run `pylint` and `mypy` before code committing manually. It will save you from
wasting time at reading CI logs for failed build.

In order to check everything is fine (and have a dry run for formatting and import sorting)
consider following commands, e.g. for `slz_apple_music_charts_scrapper`:
```shell script
cd slz_apple_music_charts_scrapper
workon slz_apple_music_charts_scrapper # or './venv/bin/activate'
isort --check-only -rc . && yapf --recursive --verbose --parallel --diff . \
&& pylint --rcfile=pylintrc ./${PWD##*/} ./tests && mypy ${PWD##*/} && echo 'Exit code:' $?
```

It will run all checks inside a current directory.

## Improvements list

1. Use FilePath; Improve FileStorageService.
2. Improve Unit and Integration testing.
   1. Add validator service tests.
   2. Add more file_storage service tests.
   3. When logic of path creation will be removed from FileStorageService, perform refactoring for bucket_paths_stateless
3. ~~Validator should not write ContentStatus FailureLog. This should be handled by ApplicationManager.~~
4. Charts download process is a bit tangled and can be improved with refactoring. One fetching method should not call another.
5. Save raw charts data to quarantine bucket. Move them to corrupted bucket in case of fail, remove in case of success.