# Integration Scripts (integration_scripts)
Collection of methods and patterns for use at The Orchard.

## Goal
To have a simple python package to import (available via internal PyPi) which scaffolds often needed, Orchard-specific subsystems.

## Features
|Module|Description|
|---|---|
|s3_backoff_utils|A collection of boto3 calls wrapped with `AWSretry` (N.B. Requires a specific commit of `AWSRetry` due to PyPi `AWSRetry` version not being up-to-date)|
|db_utils|A collection of handy db methods for rapidly accesing MySQL and Snowflake.|
|orch_utils|A collection of Orchard-specific methods for things like accessing ALW user_ID's et.al.|
|general_use|Handy general use methods.|
|logger|[Loguru](https://github.com/Delgan/loguru) based logging system with pre-configured sinks for console, text-file, CSV and JSON based logging. 
|xlsx_to_pandas_s3|Methods for processing CSV and XLSX via pandas dataframes. Allows for S3 access and S3->Snowflake data loading.|
|file_utils|Utility methods to assist with file operations.|
|service|Specialty methods for interacting with external services. (Currently only contains a IMAP email control-code parser.)

## Connectors
Connectors for S3, Snowflake, MySQL, and Sentry are included right out of the box.

## Logger Sink Examples
### CSV_LOG Example
```
from integration_scripts import logger

# CSV file output only
logger.bind(csv_only=True).info('csv_header_1, csv_header_2, etc.')
logger.bind(csv_only=True).info('val_row_1_col_1, val_row_1_col_2, etc.')
logger.bind(csv_only=True).info('val_row_2_col_1, val_row_2_col_2, etc.')

# CSV file output with mirror to console (or MASTER_LOG, if specified)_
logger.bind(csv=True).info('val_1', 'val_2', etc.)
```
### OTHER_LOG Example
```
from integration_scripts import logger

logger.debug('Message only in console (and MASTER_LOG, if specified)')
logger.bind(other=True).debug('Message in both console (and MASTER_LOG, if specified) & OTHER_LOG')
logger.bind(other_only=True).debug('Message in only OTHER_LOG')
```
### Combining Sinks
```
from integration_scripts import logger

#  Multiple sinks at once are allowed
logger.bind(csv=True, json=True).debug('val_2_1, val_2_2, etc.')```
``` 

### Multiprocessing / Threading
```
from integration_scripts import logger

logger.bind(multi=True).debug('Enqueued message for concurrency.'
```

## Pebble for Concurrency
Pebble is included in the requirements.txt file for easily wrapping methods in the wonderful `pebble.concurrent.thread` 
and `pebble.concurrent.process` decorators. The base functionality of pebble is inherited from `concurrent.futures` with 
the added ability to cancel enqueued but not yet started tasks. 

Read more [here](https://pythonhosted.org/Pebble/).


## WIP
### Loguru with python-owslogger 
Currently working on a PropagationHandler to pass logs to `owslogger` if a LOGGER_DSN is provided.|

## Improvements
I encourage others to add, fix, improve and submit to this package. 

## Environment:
[python-dotenv](https://github.com/theskumar/python-dotenv) based dynamic .env file loading is included out of the box.
Implementing dynamic .env loading requires the environment variable LOAD_ENV to be set to 'True'. To load a .env file aside from the default `.env` file, set the ENV_FILE environment variable to the new file name.
### Vars:
```
# Defaults are surrounded with parentheses

# True / False strings or 0 / 1 are acceptable for boolean values.

# Whether or not to use the python-dotenv based dynamic .env file loader
export LOAD_ENV=True/(False)

# Environment ----------------------------------------------------------------
export ENVIRONMENT=(dev)/qa/prod/test


# Orchard-Specific -----------------------------------------------------------
# OA user id with creds to access Bulk Upload VAPI endpoint and insert rows
export OA_USER_LOGIN=


# MySQL ----------------------------------------------------------------------
# Catch-all MySQL environment. Overrides any dev/qa/prod specific sets of values
export AR_MYSQL_USER=
export AR_MYSQL_PASSWORD=
export AR_MYSQL_HOST=
export AR_MYSQL_DB=

# QA-specific MySQL login values
export AR_QA_MYSQL_USER=
export AR_QA_MYSQL_PASSWORD=
export AR_QA_MYSQL_HOST=
export AR_QA_MYSQL_DB=

# PROD-specific MySQL login values
export AR_PROD_MYSQL_USER=
export AR_PROD_MYSQL_PASSWORD=
export AR_PROD_MYSQL_HOST=
export AR_PROD_MYSQL_DB=

# DEV-specific MySQL login values
export AR_DEV_MYSQL_USER=
export AR_DEV_MYSQL_PASSWORD=
export AR_DEV_MYSQL_HOST=
export AR_DEV_MYSQL_DB=

# RDS-specific login values
export RDS_MYSQL_USER=
export RDS_MYSQL_PASSWORD=
export RDS_MYSQL_HOST=
export RDS_MYSQL_DB=


# Snowflake ------------------------------------------------------------------
# Snowflake login credentials
export SNOWFLAKE_ACCOUNT=(orchard)
export SNOWFLAKE_USER=
export SNOWFLAKE_ROLE=(DEV_ENGINEERING)
export SNOWFLAKE_PASSWORD=
export SNOWFLAKE_WAREHOUSE=(DEV_OWS_WAREHOUSE)
export SNOWFLAKE_DATABASE=
export SNOWFLAKE_SCHEMA=

# PEM-key based authentication credentials
export SNOWFLAKE_KEY_PASSPHRASE=

# If SNOWFLAKE_PRIVATE_KEY_PATH is not provided, an attempt will be made to 
# determine the user ~/.ssh/snowflake directory (OSX/Linux/Windows specific) 
# and look for `rsa_key.p8`
export SNOWFLAKE_PRIVATE_KEY_PATH=


# AWS ------------------------------------------------------------------------
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export AWS_REGION=(us-east-1)


# S3 -------------------------------------------------------------------------
export S3_BUCKET=(dev-orchdbucket)


# Sentry ---------------------------------------------------------------------
# If value is present, Sentry will be enabled  
export SENTRY_DSN=


# Files and Folders ----------------------------------------------------------
# {date} is a placeholder for use with file_utils.create_path_with_todays_date()
# if a date is not provided to the method, the {date} element will be truncated 
# from the path.
export FILE_OUTPUT_PATH=('tmp/output/{date}')


# Enable Debug Logging -------------------------------------------------------
export DEBUG_LOG = True/(False)


# Logger DSN (WIP) -----------------------------------------------------------
# if provided, logs will be propagated to logger DSN
export LOGGER_DSN = 
export LOGGER_NAME = 


# Logging Targets ------------------------------------------------------------
# The directory to store file logs.
export LOG_DIR = (str(os.getcwd() + '/logs'))

# The presence of values in the following vars will enable the respective 
# Loguru log sinks. If they are not present, the sink will not be available.

# A filename to use for the log that catches all console log messges. You may 
# include '{timestamp}' in the log name to get an automatically generated 
# timestamp inserted into the log name. (e.g. 'master_log_{timestamp}_v1.log')
export MASTER_LOG = 

# A filename to be used with the JSON log sink. 
# Usage: log.bind(json=True).info('Test') - 'Test' is output to console and json log
# Usage: log.bind(json_only=True).info('Test') - 'Test' is output to json log only
export JSON_LOG = 

# A filename to be used with the CSV log sink
# Usage: log.bind(csv=True).info('Test') - 'Test' is output to console and csv log
# Usage: log.bind(csv_only=True).info('Test') - 'Test' is output to csv log only
export CSV_LOG = 

# A filename to be used with an uncategorized log
# Usage: log.bind(other=True).info('Test') - 'Test' is output to console and other log
# Usage: log.bind(other_only=True).info('Test') - 'Test' is output to other log only
export OTHER_LOG = 

```