# Royalties Core Calculations This repository houses the code that performs the core calculations in the Royalties accounting run. We use Spark to run these calculations in a distributed manner. # Royalties Scala Spark Jobs This folder houses Scala Spark for the Royalties accounting run. ## Installation 1) In a terminal, run ```shell script javac -version ``` to see if you have a version of Java installed on your system. If it says `1.8.x` skip to step 3. 2) Install the Java Development Kit, [JDK 8](https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html). Note that it's important to choose JDK 8 here, and not anything newer. 3) Install sbt, the Scala Build Tool, by following the instructions on their [website](https://www.scala-sbt.org/) for your platform. On OS X, you can simply: ```shell script brew install sbt@0.13 ``` If running `sbt` doesn't work, you may need to link the executable: ```shell script brew link sbt@0.13 ``` ## Scala Built Tool (SBT) Commands These commands can be run in your command line or using IntelliJ's `sbt shell`. The `sbt shell` can be found at the bottom of the IDE. Once clicked, wait for the shell to initialize. To run the Scalatest specs: ```shell script > sbt "project ;" test ``` To lint using Scalastyle: ```shell script > sbt "project ;" scalastyle ``` To format the source using Scalafmt: ```shell script > sbt "project ;" scalafmt ``` Build a JAR: ```shell script > sbt "project ;" assembly ``` **NOTE:** If running sbt commands in the `sbt shell`, only the commands after the semi-colon are needed Example: ```shell script > test ``` instead of: ```shell script > sbt "project ;" test ``` ## Setting up Intellij IDEA When you open this project in Intellij, it will detect that it's based on sbt, and will offer to import settings from the [configuration file](build.sbt). Accept, and the IDE will download all the dependencies for you. ## Sub-Projects This repo contains multiple scala projects that function as processors for various abacus events: * **CalculateTotals** is used at the end of an accounting run to calculate the final transaction totals. * **MatchContracts** is used during an accounting run to match Abacus contracts to sales files in an accounting period. * **MatchContractTerms** is used during an accounting run to match Abacus contract *terms* to sales files in the accounting period. * **MatchMechanicals** is used during an accounting run to match Mechanical Deduction / Transactions to Abacus contracts. * **Mechanicals** is used to match Mechanical Deductions (exported from Snowflake and placed on S3) with sales files in an accounting period. * ~~**PreProcess** is used to prepare sales files and place them on S3. This makes them available for Mechanical and Contract matching.~~ TO BE DEPRECATED ### Add a New Sub-Project To create a new sub-project, navigate to the `Project Settings` menu in IntelliJ via File -> Project Structure. In the left menu select `Modules`. The other sub-projects should be listed here. Click the `+` icon to add a new module. Click `Apply` and the new module should appear in the IntelliJ file explorer. Create the following project structure: ```bash ProjectModuleName ├── src │   ├── main │   │   └── scala │   │   └── com │   │   └── theorchard │   │   └── │   │   ├── loaders *optional │   │   └── models *optional │   └── test │   ├── resources │   │   │   └── scala │   └── com │   └── theorchard │   └── └── target ``` In the repo's root [build.sbt](./build.sbt) file add the new project: ```scala lazy val NewProject = project .settings( assemblySettings, name := "", mainClass in assembly := Some("com.theorchard..") ) .dependsOn( abacus % "compile->compile;test->test" // ) ``` When saved a `reload sbt project` button should appear -- click it to correctly mark directories in the new project. (If the button does not pop up, it can be found in the `sbt` menu on the right side of the editor). In the repo's root [Makefile](./Makefile), add the new project's `output` and `target` directories to the `clean` command. ## Unit Tests The tests in this repo use [ScalaTest](https://www.scalatest.org/user_guide/using_matchers). A useful debugging command is `show`, which will print results to the console during tests. A (simplified) example: ```scala val transactions: Dataset[Transaction] = getTransactions("s3://path/to/sales.tsv") transactions.show() // +------+------------+ // | txnId| unitPrice| // +------+------------+ // | 11| 10.00| // | 22| 20.00| // | 33| 30.00| // +------+------------+ ``` Just remember to remove it before opening a PR! ## Integration Tests The integration tests depend on AWS in the following ways: 1. The tests run these jar files (AKA the compiled code) in EMR. * There should be an EMR cluster running in AWS specifically for these integration tests: `qa-abacus-compute-test-cluster`. * If the EMR cluster is not up and running, it will need to be [terraformed](https://github.com/theorchard/terraform-infra/blob/master/qa/abacus-compute-test/main.tf). * Once terraformed, make sure the `LIVY_URL` in your `.env` and/or `conftest.py` matches the new cluster's IP. 2. The code receives S3 file paths from which to read data and outputs results back to S3. * There is an S3 bucket with staged test data (`qa-abacus-compute-tests`) that includes: * a directory to config the EMR cluster * a directory for the test results * a directory of the jar files for each of the processors in this repo (to be run in EMR) * test data in a directory formatted with a date (i.e. `data-072221/`) ```bash ├── qa-abacus-compute-tests │   ├── abacus-compute-test │   │   └── emr-custom-action.sh │   ├── acc_run_testing_results │   │   └── * │   ├── code │   │   └── * │   ├── data-072221 │   │   └── * ``` ### Setting Up The Code When code changes are deployed, jar files are put in the S3 bucket responsible for processing royalties (ie: `qa-royalties-sales-files/code/`). The [jenkins job](https://pipeline.theorchard.io/view/Abacus%20Pipeline%20Dashboard/job/qa-abacus-compute-integration-tests/) that runs these integration tests will sync the two S3 buckets to ensure the latest jar files are also in the test bucket. :warning: However, if you are running these integration tests locally, you may need to manually copy the jar files from `qa-royalties-sales-files/code/` to `qa-abacus-compute-tests/code/`. ### Setting Up The Data The code in this repo is largely dependent on parquet files, which can be a pain to work with. Luckily, most of the data needed for these processors can be exported from snowflake directly to S3 as parquet or tsv files. This can be done using snowflake stages: `@abacus_parquet_file` and `@abacus_tsv_file`. Example: ```sql COPY INTO @abacus_parquet_file/dana-temp/exchange-rates FROM ( SELECT CAST(e.exchange_rate_id AS INT) AS exchangeRateId, CAST(c_from.ISO_NUMBER AS INT) AS exchangeFromCurrencyId, CAST(c_to.ISO_NUMBER AS INT) AS exchangeToCurrencyId, e.rate AS exchangeRate FROM ORCHARD_APP_REPORTING_V2.QA_ROYALTY_ACCOUNTING_ROYALTY_ACCOUNTING.exchange_rate AS e INNER JOIN ROYALTY_ACCOUNTING.QA.CURRENCY_MAP AS c_from ON e.from_currency_code = c_from.ISO_CODE INNER JOIN ROYALTY_ACCOUNTING.QA.CURRENCY_MAP AS c_to ON e.to_currency_code = c_to.ISO_CODE WHERE e.accounting_period_id = ) FILE_FORMAT = (TYPE = PARQUET) HEADER = TRUE; ``` One major drawback of using these stages is that they only write to the `qa-royalties-sales-file` bucket. A workaround is to create a temp folder in that bucket (as shown in the above example): * Make sure to delete the folder after all the data has been moved. * Export data from snowflake to that temp folder and then copy the files over to `qa-abacus-compute-tests/data-*`. Keeping the test files relatively small will make syncing the buckets much faster. It is important to keep in mind that scala/java is **_very_** strongly typed. It is worth being explicit when exporting data into a parquet file to avoid type errors. This is shown in the above example. ### Running The Tests There is one make command for running tests for all the processors, `make run_everything`. Make sure the file paths in the Makefile under `run_everything` are correct. As tests are running, batches will appear at `${LIVY_URL}/ui` in the browser, which can be helpful with debugging.