# Neo4J migrations with liquigraph

## When to use this migration style

- Neo4j DDL statements.
- Minor data changes using specific record ids.
- Short running queries (less than 5 minutes)


### Oficial documentation

[Liquigraph documentation](http://www.liquigraph.org/latest/index.html)

### How can I create a migration

- Put your migration files under `/{neo4j-db-name}/build/changelog/dml` directory
- XML and cypher format is supported for migrations
- No special rollback sections/tags. To rollback a migration you will have 
to create a new migration with rollback queries manually.
- The changeset id must be unique across all changesets. Follow best practices below to ensure this.

### Best Practices

- Only one file per PR.
- Changeset id should match the file (minuse '.xml' filetype)
- If multiple changesets in a file, append the sequence number of the changeset
- Verify in `neo4j://dev-neo4j-cluster.dev.theorchard.io:7687` that the changeset can execute within the 5-minute Neo4j timeout.
- Changesets that create 100s of nodes and edges may exceed the 5-minute timeout. Break up large queries into multiple, smaller changesets. 
- Use [EXPLAIN](https://neo4j.com/docs/cypher-manual/current/execution-plans/) to verify the query plan does not include a [NodeByLabelScan](https://neo4j.com/docs/cypher-manual/current/execution-plans/operators/#query-plan-node-by-label-scan).

### Migration example
Filename: migration_example.xml
```
<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:neo4j="http://www.liquibase.org/xml/ns/dbchangelog-ext"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd">
    <changeSet id="migration_example:1" author="jdiaz85">
        <neo4j:cypher>CREATE (:Movie {title: 'My Life99'})</neo4j:cypher>
    </changeSet>
</databaseChangeLog>
``` 

Filename: migration_example.cypher
```
--liquibase formatted cypher

--changeset jdiaz85:migration_example
CREATE (:Movie {title: 'My Life'})
``` 

### Steps to Migrate XML from Liquigraph to Liquibase

Liquigraph file content:
```
<?xml version="1.0" encoding="UTF-8"?>
<changelog xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:noNamespaceSchemaLocation="http://www.liquigraph.org/schema/1.0/liquigraph.xsd">
    <changeset id="migration_example:1" author="jdiaz85">
        <query>CREATE (n:Element {text:'H', id:1}) RETURN n</query>
    </changeset>
    <changeset id="migration_example:2" author="jdiaz85">
        <query>CREATE (n:Element {text:'O', id:2}) RETURN n</query>
    </changeset>
</changelog>
``` 
#### Step 1 - Change changelog  
It is necessary to change the entire tag (changelog to databaseChangelog) and its content to look like this:  
Before
```
<changelog xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:noNamespaceSchemaLocation="http://www.liquigraph.org/schema/1.0/liquigraph.xsd">
</changelog>
```
After
```
<databaseChangeLog xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:neo4j="http://www.liquibase.org/xml/ns/dbchangelog-ext"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd">
                
</databaseChangeLog>   
```
#### Step 2 - Change changeset tag  
Change each changeset tag to changeSet  
Before
```
    <changeset id="migration_example:1" author="jdiaz85">
    </changeset>
    <changeset id="migration_example:2" author="jdiaz85">
    </changeset>
```
After
```
    <changeSet id="migration_example:1" author="jdiaz85">
    </changeSet>
    <changeSet id="migration_example:2" author="jdiaz85">
    </changeSet>
```
#### Step 3 - Change query tag  
Change each query tag to neo4j:cypher  
Before
```
        <query>CREATE (n:Element {text:'H', id:1}) RETURN n</query>
        <query>CREATE (n:Element {text:'O', id:2}) RETURN n</query>
```
After
```
        <neo4j:cypher>CREATE (n:Element {text:'H', id:1}) RETURN n</neo4j:cypher>
        <neo4j:cypher>CREATE (n:Element {text:'O', id:2}) RETURN n</neo4j:cypher>
```

Liquibase file content:
```
<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                   xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
                   xmlns:neo4j="http://www.liquibase.org/xml/ns/dbchangelog-ext"
                   xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd">
    <changeSet id="migration_example:1" author="jdiaz85">
        <neo4j:cypher>CREATE (n:Element {text:'H', id:1}) RETURN n</neo4j:cypher>
    </changeSet>
    <changeSet id="migration_example:2" author="jdiaz85">
        <neo4j:cypher>CREATE (n:Element {text:'O', id:2}) RETURN n</neo4j:cypher>
    </changeSet>
</databaseChangeLog>
``` 

### Running a Migration Locally
```
mvn -f build/pom.xml initialize package -DchangeLogFile=db-migration-you-want-to-run.xml
```

### Browsing Changsets
See all changesets in the changelog
```
MATCH (lcl:__LiquibaseChangeLog)-[r]-(lcs)
 return lcl,r,lcs;
```

See all changesets with id like PLATFORM-1624_cleanup_sample_data*
```
MATCH (lcl:__LiquibaseChangeLog)-[r]-(lcs)
WHERE lcs.id STARTS WITH 'PLATFORM-1624_cleanup_sample_data'
 return lcl,r,lcs;
```

See changeset exactly matching id=PLATFORM-1624_cleanup_sample_data:3
```
MATCH (lcl:__LiquibaseChangeLog)-[r]-(lcs)
WHERE lcs.id=PLATFORM-1624_cleanup_sample_data:3
 return lcl,r,lcs;
```

### Neo4j Graph Synchronization

Neo4j Graph is synchronized with art_relations database. The mechanism of the synchronization is described [here](https://docs.google.com/document/d/1j3kWU1XTTKoH6J7P38Fk2Bl9WCNfgi6pRQN8uH8dfsE/edit#heading=h.838uz6pwe1bf).

#### Neo4j Streams Sync Query

```cypher
WITH event AS event
MATCH(q:KafkaImportQuery {
  table: event.table, 
  database: event.database,
  type:  event.type})
WITH q, event
CALL apoc.cypher.doIt(q.query, event.data) YIELD value AS value
RETURN *
```

This query is executed every time when a new message appears in kafka topic. (Sync with art_relations db is defined [here](https://github.com/theorchard/irrigate-chef-repo/blob/master/roles/qa_neo4j_cluster.json#L57)).

The query retrieves an `KafkaImportQuery` node which contains another `cypher` query for processing the message. These queries are uniquely defined by `table`, `database` and `type` (`insert`, `update` or `delete`) attributes (corresponding `NODE KEY` is added to the `KafkaImportQuery` label).

#### Sync Nodes

Here is an example of sync node 
```cypher
WITH "CREATE (ai:ArtistInfo {id: toInteger(artist_id)})" AS query
MERGE (q:KafkaImportQuery {
    database: 'art_relations',
    table: 'artist_info',
    type: 'insert'})
ON CREATE SET q.query = query
ON MATCH SET q.query = query
```

This one creates a new ArtistInfo node each time `insert` records appears in Kafka topic.

#### Sync Nodes Migration

All migrations should be placed in this [file](https://github.com/theorchard/database/blob/master/neo4j/neo4j-orchard/build/dml/kafka_streams_import_queries.xml). It allows to see the latest version of the handlers in one place.
Please note how changeset id is defined
`id="kafka_streams_import_queries:<database>.<table>.on_<operation>"`
Also, add `run-on-change="true"` flag, it allows to change the handlers in the future.