# Neo4j data transferring scripts

This folder contains scripts for generating code for transferring data from Neo4j to Aura.


## Generate data export Pull Request file

[generate_dump_files.py](generate_dump_files.py) script generates a PR with appropriate file format for the [Neo4j database catlog](https://github.com/theorchard/database/tree/master/neo4j/neo4j-orchard/build/changelog/dml). This PR can be later applied by this [pipeline](https://pipeline.theorchard.io/job/db-deploy-pipeline/).
The result of the queries execution is a set of JSON files that contain the NR data from the Neo4j database. These files can be found on one of the Neo4j nodes in `/var/lib/neo4j/import` catalog.

This script walks through the list of NR data labels and pulls the set of unique properties for each item. Using this information, it generates a set of queries that will be executed on the Neo4j database to dump the data.


## Generate indexes

[generate_indexes.py](generate_indexes.py) script generates a files with `cypher` DDL commands for creating indexes on the Aura database. Indexes are only created for the node labels that are used in relationships. 


## Generate data import Pull Request file

[generate_import_files.py](generate_import_files.py) script generates a PR with appropriate file format for the [Neo4j database catlog](https://github.com/theorchard/database/tree/master/neo4j-aura/neighbouring-rights/build/changelog/dml). This PR can be later applied by this [pipeline](https://pipeline.theorchard.io/job/db-deploy-pipeline/).
The queries from the generate file contain cypher code for creating nodes and relationships in the Aura database.
This script runs `aws s3 presign` command for each previously uploaded to S3 JSON file and put the result into the generated import file.
Important note: before uploading relationship data appropriate indexes on the related nodes should be created, otherwise upload process can be very slow.