# abacus-etl-tools

This repo is designed to provide access to data that is usually only available in a database.

It uses a Snowflake stage to copy the data into S3.  For example, the following is used for the transaction types:
`COPY INTO @abacus_static_data/transaction_types/transaction_types.csv`

`abacus_static_data` is the name of the stage in Snowflake, and it's configured to copy the data to `s3://qa-abacus-static-data`.

`transaction_types/transaction_types.csv` is the directory and filename within the S3 bucket.

More info here:
https://docs.snowflake.com/en/sql-reference/sql/copy-into-location.html

Local installation
------------------

If you make any changes to the library and want test it locally before publishing to the Orchard's pypi,
you can do that by navigating to the root of the package and running following command:
```bash
make local_install
```

If you want to include this library package to your project locally, then run either of the following commands to the root of the package:

```bash
make generate_whl
```
or
```bash
make generate_tar_gz
```

The above commands will create .whl or .tar.gz file in the /dist folder.  Just copy this .whl or .tar.gz file to root of the project and in requirements.txt of the project include library package as below:

```bash
file:Package-Name.whl or file:Package-Name.tar.gz
```