# SPIKE PP-1107 ows-pdp is the source of truth for ows-permissions rap admin role Whimsical link: https://whimsical.com/permissions-platform-architecture-Q4W818RiJNAJSmiQjAZZcY@2bsEvpTYSt1HivBT1xDiQVhuacfzxWTctaP # Background Currently, ows-pdp depends on ows-permissions [endpoint](https://github.com/theorchard/ows-pdp/blob/9b2d5e2997b8bb46983755288ee232e21f39244d/pdp/connectors/ows_permissions.py#L83-L90) to determine who is an `ows_permissions_rap_admin`, someone who can administer users to have access to an account, subaccount, or label_participant. The data returned by the ows-permissions endpoint comes from Neo4J nodes for Identity, SettingsProfile, and Resource, and the edges between them. ## Current performance concerns / motivation ows-pdp consumes this data for two endpoints, critical for the performance of Settings (Manage Users) and SEAT apps.: - `GET /identity//roles/` - `POST /identity//tenant//attach-and-detach/roles/` ![Relevant current architecture](SPIKE%20PP-1107%20ows-pdp%20is%20the%20source%20of%20truth%20for%20o/Screenshot_2025-09-15_at_1.39.26_pm.png) Relevant current architecture For the GET endpoint, `ows-pdp` spends between 4-33% of its time waiting for `ows-permissions`: ![Screenshot 2025-09-15 at 1.28.25 pm.png](SPIKE%20PP-1107%20ows-pdp%20is%20the%20source%20of%20truth%20for%20o/Screenshot_2025-09-15_at_1.28.25_pm.png) For the POST endpoint, `ows-pdp` spends 6-18% of its time waiting for `ows-permissions`: ![Screenshot 2025-09-15 at 1.29.23 pm.png](SPIKE%20PP-1107%20ows-pdp%20is%20the%20source%20of%20truth%20for%20o/Screenshot_2025-09-15_at_1.29.23_pm.png) Notably, while `ows-pdp` has its own Redis cache, it does not cache the `ows-permissions` data because there is no cache invalidation for changes to data made via `ows-permissions`. Instead, `ows-pdp` expects `ows-permissions` to maintain/invalidate its own Redis cache for this data. ## Where does the data in ows-permissions come from? `ows-permissions` reads data from Neo4j to provide information to ows-pdp. This data gets into Neo4J in a few different ways: ![Screenshot 2025-09-15 at 6.30.44 pm.png](SPIKE%20PP-1107%20ows-pdp%20is%20the%20source%20of%20truth%20for%20o/Screenshot_2025-09-15_at_6.30.44_pm.png) Even though the A&I Team has made many strides to standardize calls to modify an identity’s access in Neo4J via starting from graphql-user, there are still other avenues of entry: - neo4j refresh (qa only, but a broken qa environment makes for grumpy devs) - database PRs - profile-users-scripts (a Jenkins job we would like to deprecate) Notably, all changes to Neo4J are captured in the `-cdc` Kafka cluster. Notably these topics capturing relationship/edges: ![Screenshot 2025-09-15 at 6.33.26 pm.png](SPIKE%20PP-1107%20ows-pdp%20is%20the%20source%20of%20truth%20for%20o/Screenshot_2025-09-15_at_6.33.26_pm.png) The topic mappings are described here: https://github.com/theorchard/terraform-infra/blob/62cbb5dc0ed2d71686296526cd6c3c5e2bfafbe3/qa/kafka-infra/neo4j_cdc_source/neo4j_v5/main.tf#L170 and configuration documentation here: [https://neo4j.com/docs/kafka/current/source/cdc/](https://neo4j.com/docs/kafka/current/source/cdc/) Here are some choice example messages (wip) - cdc.musicGraphV5.deleteHadAdminAccessTo # Goal This Spike describes approaches to make it possible for `ows-pdp` to fetch the `ows_permissions_rap_admin` role from the PDP Datastore, instead of from `ows-permissions` : ![Desired architecture - no calls to `ows-permissions`](SPIKE%20PP-1107%20ows-pdp%20is%20the%20source%20of%20truth%20for%20o/Screenshot_2025-09-15_at_1.41.49_pm.png) Desired architecture - no calls to `ows-permissions` We need: - some form of “dual-write” whereby we can capture the changes made to an identity’s “settings profile admin access” to `ows-permissions` into the `ows-pdp` datastore. - to support QA and Prod seamlessly - to support ad-hoc changes made by database PR - # Possible Approaches (diagrams forthcoming) ## Synchronous Writes ### Update graphql-user to dual-write Graphql-user would send a request to update roles for ows-permissions and a request to ows-pdp to attach/detach the ows_permissions_rap_admin role Pros: 1. This allows us to support from Settings v2 even after we get rid of SettingsProfile has admin access for an identity in Neo4j. 2. Would work if we eliminated DB PRs, profile-user-scripts. 3. This is currently where we dual-write Fansifter roles. Cons: 1. When we dual-write Fansifter roles, we’re actually receiving the fansifter role. Instead for Settings admin, we’d need to handle the SettingsProfile and the role etc. 2. The dual-write Fansifter roles code is already very hairy. 3. There are actually 4 different mutations that could be used atm - settings v1 invite/create, settings v2 invite/create, settings v1 edit, settings v2 edit 1. not to mention settings v1 deactivate one, settings v1 deactivate all 1. At least we’re set on settings v2 deactivate one/all! 4. This doesn’t support the neo4j refresh for folks added as an ows_permissions_rap_admin. Theoretically we could get around this by telling folks to set in Neo4j scripts and pdp-backfill but this workaround will likely lead to lots of questions/reminders/support messages in Slack 5. We’re not necessarily guaranteed that the operation will succeed in ows-permissions. This may result in inconsistent data in ows-pdp and ows-permissions ### Update ows-permissions to dual-write ows-permissions would send a request to attach/detach the ows_permissions_rap_admin role Pros: 1. Would work if we eliminated DP PRs, profile-user-scripts. Cons: 1. This doesn’t support the neo4j refresh for folks added as an ows_permissions_rap_admin. Theoretically we could get around this by telling folks to set in Neo4j scripts and pdp-backfill but this workaround will likely lead to lots of questions/reminders/support messages in Slack 2. There’s a scary circular loop atm… in order for the ows-pdp attach-and-detach endpoint to work, it makes a call to ows-permissions to check the identity is an ows_permissions_rap_admin. Thus, for this to work, ows-permissions would send a request to ows-pdp, get a different request from ows-pdp while it’s waiting for its request to be fulfilled. 3. Even without the scary loop, ows-permissions code is very hairy, branchy, etc. ## Asynchronous Writes ### Update ows-permissions to write to a Kafka topic ows-permissions would send a message to a Kafka topic indicating when an identity has become a Settings admin or been removed from being a Settings admin. Some machine would consume messages from this topic. Pros: 1. Would work if we eliminated DP PRs, profile-user-scripts. 2. This doesn’t support the neo4j refresh for folks added as an ows_permissions_rap_admin. Theoretically we could get around this by telling folks to set in Neo4j scripts and pdp-backfill but this workaround will likely lead to lots of questions/reminders/support messages in Slack 3. No scary circular loop Cons: 1. This is eventually consistent, as it’s an asychronous write 2. This doesn’t support the neo4j refresh for folks added as an ows_permissions_rap_admin. Theoretically we could get around this by telling folks to set in Neo4j scripts and pdp-backfill but this workaround will likely lead to lots of questions/reminders/support messages in Slack 3. The ows-permissions code can be quite hairy… we might not want to go through it all and find every path that might need to write a message? Or we write very general messages ### Use neo4j Kakfa Source Connector to write messages to a Kafka topic The neo4j Kafka Source Connector is already configured to write to Kafka topics, notably the `cdc.musicGraphV5.hasAdminAccessTo` topic might provide a lot of what we’re looking for. Some machine would consume messages from this topic, or alternatively a custom topic with specific patterns ([https://neo4j.com/docs/kafka/current/source/cdc/#_configuration](https://neo4j.com/docs/kafka/current/source/cdc/#_configuration)) to make attach/detach requests to ows-pdp. Pros: 1. Would work for all known avenues of identity/profile/relationships getting into Neo4J, including 2. Kafka topic messages are JSON. We are very good at parsing and wrangling JSON. Cons: 1. This is eventually consistent, as it’s an asychronous write 2. Well I’m not sure if every change can be represented in one topic. What if it can’t? 3. Well I’m not sure if we can get the Identity + Profile at the same time as an event related to a Profile + has admin access relationship change + Resource (lp, subaccount, account, etc) 1. Can KSQL help? If ksql can’t help, just knowing the profile uuid would be sufficient to query the identity uuid and then decide what to attach/detach to ows-pdp 2. What if the Machine just follows a topic containing `SettingsProfile + has admin access relationship change + Resource` and uses a Neo4j query to get the identity uuid that has/doesn’t have the SettingsProfile uuid? 1. We tried to make it do this: https://github.com/theorchard/terraform-infra/pull/26903, but it just never picked up any matches to create messages onto the topic. At the same time, we were getting messages onto [https://qa-akhq.theorchard.io/ui/qa-cdc/topic/cdc.musicGraphV5.hasAdminAccessTo/data?sort=Newest&partition=All](https://qa-akhq.theorchard.io/ui/qa-cdc/topic/cdc.musicGraphV5.hasAdminAccessTo/data?sort=Newest&partition=All), which would have been some of the messages I expected to land it the pp1107 topic.