OWS Socials
=====================================

This service manages information socials networks data.

## Getting Started

### Installation

Before starting make sure you have installed the python version used in the project.
An easy way to manage python versions is with pyenv.

```
$ brew update
$ brew install pyenv
$ pyenv install `cat .python-version`
$ pyenv local `cat .python-version`
```
```bash
$ cd ows-socials
$ make install_with_dev_from_lock
$ pre-commit install
```

### Configuration

Let's set environment configuration

```bash
$ cp .env.shadow .env
```

Edit `.env` file and update configuration parameters.
You will definitely need to update `SNOWFLAKE_*` variables.
Others are optional and not normally required in dev environment.

### SQL Linter
This repository has SQL linter "sqlfluff" - https://github.com/sqlfluff/sqlfluff

#### Command line

pipenv run sqlfluff lint --ignore parsing

#### PyCharm integration  
  
You can make pycharm run sqlfluff on file change.
In order to configure it:

* Settings - Tools - File watchers
* Add new file watcher:
    * name: SQL linter
    * File type: SQL
    * scope: Project files
    * Program: $PyInterpreterDirectory$/sqlfluff
    * Arguments: lint --ignore parsing $FileDir$/$FileName$
    * Show console: On error
    
It's all set up.

Now go to any SQL file and start editing. 
You will see console popup with errors if any.
    

### Running

When all dependencies have been installed, you can run the flask application
on your local instance by running:

```bash
(env) $ python dev.py
```

All make commands will automatically install all dependencies in an virtual environment folder called env.
You can activate a virtual environment by running `. env/bin/activate` or `source env/bin/activate`.

By using the development server, you will have access to specific features that
are not necessarily available in production, such as the exception tracer.

### Testing

To run the tests, all you have to do is to run:

```bash
(env) $ py.test tests/unit/
(env) $ py.test tests/integration/
(env) $ flake8 socials/ tests/
```

or

To run unit tests: `make test`

To run unit tests with coverage reporting: `make test_unit`

To run integration tests: `make test_integration`

To lint: `make lint`

### Updating

To install new dependencies.

```bash
make install_with_dev
```

### Notes

Windows users will not be able to use the `make` commands as Make is a Unix util.
Windows users can attempt to install [GNUWin](http://gnuwin32.sourceforge.net/packages/make.htm) to get this functionality.

## Cache

Redis is used for caching. To use it, you must have a Redis server running. You can run one locally with Docker:

```bash
make docker/up-redis
```

By default, the application will look for a Redis server at `localhost:6379`. You can change this by setting the `REDIS_HOST` and `REDIS_PORT` environment variables in your `.env` file.

The cache will expire after 8 hour. You can change this by setting the `REDIS_CACHE_TTL` environment variable in your `.env` file (or config.py). The value should be in seconds.
On production, the cache is also invalidated when ingestion jobs run https://scheduler.theorchard.io/job/chartmetric-ingestion-pipeline/.

There is no warmup script. The cache is populated on demand.