{ "_meta": { "description": "Source of truth for the Kafka connector types this skill generates: Snowflake source + sink, and Neo4j CDC source. Each connector type defines (a) its required inputs, (b) its env vars, (c) its instance_path_pattern, (d) its template, and (e) module version overrides where they differ from the verified-latest defaults.", "module_versions_default": { "terraform_fargate_ref": "6.5.0", "terraform_default_tags_ref": "2.0.0", "terraform_vpc_info_ref": "3.1.0", "terraform_datadog_kafka_connector_ref": "6.18.2", "terraform_secrets_manager_ref": "1.5.1", "terraform_required_version": "1.14.3" }, "fargate_defaults": { "container_port": 8083, "service_platform_version": "1.4.0", "datadog_task_memory": 512, "datadog_custom_image": "086679231553.dkr.ecr.${var.aws_region}.amazonaws.com/orchard-datadog-agent-jmx:latest", "non_ecr_image_account_id": "086679231553", "blocking_waf_enabled": true, "propagate_tags": true }, "scope_note": "Scope is intentionally narrowed to Snowflake source + sink and Neo4j CDC source. To re-introduce jdbc_sink, s3_source/sink, or debezium_mysql_source, re-add entries here and matching templates under references/templates/.", "terraform_default_tags_module": { "repo": "https://github.com/theorchard/terraform-default-tags", "version_used_by_all_connectors": "2.0.0", "v2_0_0": { "used_by": ["snowflake_sink", "jdbc_source_snowflake", "neo4j_cdc_source"], "inputs_required": ["application_family", "environment", "team_name"], "inputs_optional": ["service_name", "additional_tags"], "validation": "team_name MUST match a Datadog team handle. The module pulls the valid list dynamically via `data \"datadog_teams\" \"pde\"` and validates at plan time. If invalid, terraform plan fails with an error listing all valid handles. application_family is NOT validated at the module level." }, "migration_note": "All three connectors now use terraform-default-tags 2.0.0 (snowflake_sink was previously pinned to 1.0.0, which had no team_name). team_name is therefore required for ALL connector types.", "emitted_tags": "environment, application_family, terraformed=true, terraform_github_repository (auto-detected from path), terraform_github_path (auto-detected), team, service_name (if non-empty), plus var.additional_tags." }, "application_family_discovery": { "description": "application_family is a user-provided value passed to terraform-default-tags. The plugin does NOT hardcode or validate it. Each terraform-infra application family corresponds to a directory under prod/iam/application_family/.", "available_options_dir": "/prod/iam/application_family/", "path_resolution_note": "NEVER hardcode an absolute path or a username. Resolve at runtime: it is the terraform-infra repo the connector is being written into (derive from the working directory / the confirmed output base path, or ask the user). The path is repo-relative: /prod/iam/application_family/.", "skill_behavior": "When asking the user for application_family, list that directory (after resolving the terraform-infra root) and offer the entries as suggestions (e.g. AskUserQuestion options). If the directory can't be located, just ask the user to type the application_family. Accept ANY user input — do NOT validate against the directory. The user may be adding a new application family that isn't yet there. terraform-default-tags itself does NOT validate application_family at plan time — the directory is org convention, not module-enforced." }, "team_name_input": { "description": "team_name is a user-provided value passed to terraform-default-tags 2.0.0, which ALL three connector types now use. Required for every connector.", "skill_behavior": "Ask the user; accept any string — the plugin does NOT validate. terraform-default-tags v2.0.0 validates at plan time against the live Datadog team-handle list (`data \"datadog_teams\" \"pde\"`); if the user provides an invalid handle, terraform plan will fail with an error message listing all valid handles. That's the canonical validation path — don't duplicate it in the plugin." }, "snowflake_user_and_key_setup": { "applies_to": ["snowflake_sink", "jdbc_source_snowflake"], "description": "These connectors authenticate to Snowflake with key-pair (RSA) auth. The plugin does NOT create the Snowflake service user, role, or key — it only references them (snowflake_user / snowflake_role) and creates the empty AWS Secrets Manager secret resource. The skill must emit the following as post-creation follow-ups.", "users_and_roles_terraform_path": "terraform-infra/prod/snowflake/delphi/ — define the service user under `service_users/` (RSA public key under `service_users/keys/`) and the role under `roles/`. The connector's snowflake_user/snowflake_role must match what is defined here.", "rsa_key_pair_guide": "https://app.notion.com/p/Snowflake-7c88cc17b0034e7db669a88fd2962bab#debbb574070e4059afd93520d392bca3", "secret_population": "The RSA PRIVATE key (and passphrase, for snowflake_sink) is stored in AWS Secrets Manager at `//SNOWFLAKE_PRIVATE_KEY` — the secret resource is terraformed by the plugin (terraform-secrets-manager), but the VALUE is populated manually post-apply. The PUBLIC key is registered on the Snowflake service user (in the delphi terraform above)." }, "jenkinsfile_update": { "description": "For a newly created connector to actually deploy, it must be registered in the kafka-connect repo's Jenkinsfile under the `SERVICES = [...]` Groovy map. This lives in a SEPARATE repo (`theorchard/kafka-connect`), not in `terraform-infra`. The plugin handles this conditionally: if the user provides a local path to the kafka-connect repo, the plugin edits the Jenkinsfile directly; otherwise it emits a precise follow-up the user must apply manually.", "file_path_in_repo": "Jenkinsfile", "what_to_insert": "A new line inside the SERVICES Groovy map: `'' : [project: '', deployToProd: ],`\n - `` is the AWS service name (no env prefix) — matches the value passed to terraform-fargate's `service_name` input.\n - `` comes from the connector's `jenkinsfile_project_key` field in the registry (must already exist in the Jenkinsfile's PROJECTS map; the plugin does not create new project entries).\n - `deployToProd` is DERIVED FROM the connector's environment: `true` when environment == 'prod', `false` when environment is 'qa' or 'dev'. Do not ask the user; set it from the env (user may override on explicit request).", "insertion_point": "Find the section comment matching the connector's `jenkinsfile_section_comment` (e.g. `// Neo4j CDC Source`). Append the new entry at the END of that section's existing entries, before the next section comment. Preserve the existing alignment/quoting style of nearby entries.", "skill_workflow": "Step 1 of the skill workflow should ASK the user (with AskUserQuestion) whether they have the kafka-connect repo checked out locally. If yes, ask them for its absolute path — do NOT propose a hardcoded path or assume a username. You MAY offer to locate it (commonly a sibling of the terraform-infra repo, i.e. `/kafka-connect`) and confirm the found path with the user. If they decline or it can't be found, skip the edit and emit the follow-up instead.", "skill_behavior_when_path_provided": "Verify the file `/Jenkinsfile` exists. Use the Edit tool to locate the section comment and append the new SERVICES entry. Preserve indentation/quoting. Do not modify any other SERVICES entries.", "skill_behavior_when_path_not_provided": "Add a follow-up to the validation report showing the exact line to add and where (file path + section comment to find). The user applies it manually before the connector image can be built/deployed." }, "akhq_update": { "description": "After generating the connector + topics, the plugin MUST also append the new connector to the AKHQ terraform config so it shows up in the AKHQ UI. AKHQ main.tf lives at `terraform-infra//kafka-infra/akhq/main.tf` (see msk_clusters.json#akhq_terraform_path) and contains a YAML heredoc with one entry per cluster under `akhq.connections..connect:`.", "what_to_insert": "A new YAML list item under the appropriate cluster's `connect:` block:\n - name: \n url: \"https://-.theorchard.io\"\nThe display name comes from the connector's `akhq_display_name_pattern` field (substitute `` / `` as applicable). The URL is always `https://-.theorchard.io` because terraform-fargate names the ALB `-`.", "which_cluster": "Look up `msk_clusters.json[].akhq_connection_key`. That's the YAML key to find inside the heredoc (e.g. `prod-cdc:`).", "insertion_point": "Append the new entry at the END of the existing `connect:` list for the chosen cluster, immediately before the next sibling (e.g. `ksqldb:` or the next cluster connection block). Preserve indentation (16-space leading for the `- name:` line inside the heredoc).", "skill_behavior": "Use the Edit tool to locate the cluster's `connect:` block and append the new two-line item. Do not reorder existing entries. Do not modify any other cluster's block. If the cluster key isn't found in akhq/main.tf, emit a warning telling the user to add it manually." }, "notification_endpoints_input": { "description": "notification_endpoints and escalation_notification_endpoints are user-provided values passed to the terraform-datadog/kafka_connector module. They are Datadog monitor mention strings (typically Slack handles like `@slack-` or PagerDuty handles like `@pagerduty-`, space-separated for multiple).", "skill_behavior": "Ask the user for BOTH values at creation time. Required for all three connector types. The plugin does NOT hardcode defaults or validate — accept any string. Common examples (do not enforce): `@slack-kafka-data-highway-alerts`, `@slack-data-alerts-`, `@slack-data-alarm-`, `@pagerduty-`. Suggest these as starting points only — the user picks what their team actually uses.", "skill_question_hint": "Use a single AskUserQuestion with two questions (one for routine notifications, one for escalations). Make it clear escalation_notification_endpoints is for higher-severity / paged alerts and may include a paging mention like @pagerduty-." } }, "connectors": { "snowflake_sink": { "direction": "sink", "kafka_connect_dir": "snowflake_sink", "ecr_image_name_pattern": "${var.service_name}-sfsink", "datadog_connector_type": "default_sink", "instance_path_pattern": "snowflake_sink/", "instance_path_pattern_note": "Nested under the shared `snowflake_sink/` catalog (matching existing instances like snowflake_sink/mg_fingerprint, snowflake_sink/neo4j_v5). The older flat `snowflake_sink_` sibling dirs are legacy — new instances go in the nested form.", "backend_state_key_pattern": "/kafka-infra/snowflake_sink//terraform.tfstate", "template": "snowflake_sink.tf.j2", "variables_template": "snowflake_sink_variables.tf.j2", "service_name_pattern": "${var.cluster_name}-${var.connector_type}-${var.connector_name}", "service_name_pattern_note": "Composed ONCE in variables.tf as local.service_name = \"${var.cluster_name}-${var.connector_type}-${var.connector_name}\" (cluster_name default 'kc', connector_type default 'sfsink', connector_name = hyphenated user input) → 'kc-sfsink-'. terraform-fargate prepends '-' for the deployed name. Every name reference in the template (fargate service_name, datadog agent SERVICE_NAME, CONNECT_GROUP_UNIQUE_IDENTIFIER, CONNECTOR_NAME, health-check path, secrets, datadog dashboard) uses local.service_name — do NOT repeat the composed string inline. Matches existing instances like kc-sfsink-mg-gsr / kc-sfsink-nrgraph / kc-sfsink-smf-fanresponse.", "akhq_display_name_pattern": "snowflake-sink-", "akhq_display_name_note": "This is the human-friendly AKHQ `name:` label (matching existing AKHQ entries like `snowflake-sink-mg-gsr`). It is NOT the service name. The AKHQ `url:` host is built from the deployed service name `-` = `-kc-sfsink-`.", "jenkinsfile_project_key": "snowflake_sink", "jenkinsfile_section_comment": "// Snowflake Sink", "name_length_limit": { "limit_chars": 32, "reason": "ALB / target group name limit on AWS Fargate (some resources truncate or fail above 32 chars).", "deployed_name_pattern": "-kc-sfsink-", "formula": "Static overhead = len(env) + len('-kc-sfsink-') = len(env) + 11. len(connector_name) <= 32 - 11 - len(env) = 21 - len(env).", "per_env_max_connector_name_length": { "prod": 17, "qa": 19, "dev": 18 }, "if_too_long": "Shorten connector_name. (The new kc-sfsink- scheme already gives far more room than the legacy kafka-connect-<...>-sfsink form.)", "validation_required": "HARD ERROR if exceeded — block generation and tell the user to shorten connector_name." }, "module_version_overrides": { "terraform_fargate_ref": "6.1.1", "terraform_default_tags_ref": "2.0.0", "terraform_datadog_kafka_connector_ref": "6.13.4", "terraform_required_version": "1.11.4" }, "default_task_cpu": 2048, "default_task_memory": 4096, "default_max_tasks": 1, "required_inputs": [ "topics", "snowflake_database", "snowflake_schema", "snowflake_user", "dlq_topic_name", "application_family", "team_name", "notification_endpoints", "escalation_notification_endpoints" ], "optional_inputs": [ "topic_table_overrides", "snowflake_role", "snowflake_ingestion_method", "snowflake_host" ], "snowflake_host_note": "SNOWFLAKE_HOST defaults to 'sme-delphi.snowflakecomputing.com:443' (the account the newer snowflake_sink instances use). Overridable per instance — existing instances also target 'orchard.snowflakecomputing.com:443' and 'delphi.us-east-1.snowflakecomputing.com:443'. Keep the ':443' port.", "snowflake_role_note": "snowflake_role is CONDITIONALLY REQUIRED, keyed on the ingestion method: REQUIRED when SNOWFLAKE_INGESTION_METHOD == 'SNOWPIPE_STREAMING' (the plugin default) — Snowpipe Streaming needs an explicit role; SKIP it for plain 'SNOWPIPE' (the role is never consumed there — e.g. running instances snowflake_sink/ddb_masters_active, /mg_fingerprint, /neo4j_v5 omit it and use plain SNOWPIPE). In config.template.json `snowflake.role.name` is only rendered under SNOWPIPE_STREAMING. Behavior: if streaming → ASK the user for the role and emit SNOWFLAKE_ROLE; if plain SNOWPIPE → do NOT ask and do NOT emit it.", "snowflake_ingestion_method_note": "SNOWFLAKE_INGESTION_METHOD is optional; plugin default is 'SNOWPIPE_STREAMING' (kafka-connect image Dockerfile default is plain 'SNOWPIPE'). It determines whether snowflake_role is required: SNOWPIPE_STREAMING ⇒ role required; SNOWPIPE ⇒ role skipped. Confirm the method with the user (default SNOWPIPE_STREAMING) so the skill knows whether to ask for the role.", "conditionally_required_inputs": { "snowflake_role": "required iff snowflake_ingestion_method == 'SNOWPIPE_STREAMING' (the default); skipped for plain SNOWPIPE" }, "topic_to_table_naming": { "description": "By DEFAULT the Snowflake target table for each topic is DERIVED from the topic name — the user does NOT need to provide a mapping. Only ask for overrides when a topic must land in a pre-existing table whose name doesn't match the transform.", "transform": "1) split the topic on '.' 2) convert each segment from camelCase to snake_case (insert '_' at each lowercase/digit→uppercase boundary) 3) uppercase everything 4) join the segments with '__' (double underscore).", "examples": { "cdc.musicGraphV5.acrid": "CDC__MUSIC_GRAPH_V5__ACRID", "event.musicEvent.socialMedia": "EVENT__MUSIC_EVENT__SOCIAL_MEDIA", "stream.nrGraph.soundRecording": "STREAM__NR_GRAPH__SOUND_RECORDING" }, "reversibility_note": "Dot becomes a DOUBLE underscore so it stays distinguishable from the single underscores introduced by camelCase splitting (i.e. the table name round-trips back to the topic).", "overrides": "topic_table_overrides is an optional dict { '': '' }. Any topic present there uses the explicit table; all other topics use the derived name. NOTE: some existing hand-written prod maps used a SINGLE underscore for the dot separator (e.g. event.musicEvent.socialMedia -> EVENT_MUSIC_EVENT_SOCIAL_MEDIA). To target one of those existing tables, add an explicit override — the new default would otherwise produce the double-underscore form." }, "default_env_vars": { "MAX_TASKS": 1, "BUFFER_COUNT_RECORDS": 10000, "BUFFER_SIZE_BYTES": 5000000, "BUFFER_FLUSH_TIME": 240, "SNOWFLAKE_INGESTION_METHOD": "SNOWPIPE_STREAMING", "SNOWFLAKE_HOST": "sme-delphi.snowflakecomputing.com:443" }, "secrets": { "manager_pattern": "fansifter", "secret_names": ["SNOWFLAKE_PRIVATE_KEY", "SNOWFLAKE_PRIVATE_KEY_PASSPHRASE"], "creation_module": "terraform-secrets-manager", "iteration": "for_each over secret_names" }, "kafka_connect_required_env_vars": [ "Environment", "CONNECT_BOOTSTRAP_SERVERS", "CONNECTOR_NAME", "MAX_TASKS", "BUFFER_COUNT_RECORDS", "BUFFER_SIZE_BYTES", "BUFFER_FLUSH_TIME", "KAFKA_TOPICS", "SNOWFLAKE_INGESTION_METHOD", "SNOWFLAKE_HOST", "SNOWFLAKE_USER", "SNOWFLAKE_DATABASE", "SNOWFLAKE_SCHEMA", "SNOWFLAKE_ROLE", "SNOWFLAKE_TOPIC_TABLE_MAP", "DLQ_TOPIC_NAME", "SERVICE_NAME" ], "notes": "Writes Kafka topics to Snowflake tables via Snowpipe Streaming. The SNOWFLAKE_TOPIC_TABLE_MAP is built automatically: each topic's table name is DERIVED from the topic via topic_to_table_naming (dot→'__', camelCase→snake, uppercase) unless the user supplies an explicit override in topic_table_overrides. Uses key-pair auth (private key in AWS Secrets Manager). Pinned to terraform-fargate 6.1.1 / datadog 6.13.4 to match the verified production snowflake_sink layout; terraform-default-tags bumped to 2.0.0 (so team_name is required like the other connectors)." }, "jdbc_source_snowflake": { "direction": "source", "kafka_connect_dir": "jdbc_source", "kafka_connect_subdir": "snowflake", "ecr_image_name_var": "var.connector_type", "ecr_image_name_default": "kc-jdbc-src", "datadog_connector_type": "default_source", "instance_path_pattern": "jdbc_source/snowflake/", "backend_state_key_pattern": "/kafka-infra/jdbc-source/snowflake//terraform.tfstate", "template": "jdbc_source_snowflake.tf.j2", "variables_template": "jdbc_source_snowflake_variables.tf.j2", "needs_query_sql": true, "service_name_pattern": "${var.connector_type}-", "akhq_display_name_pattern": "jdbc-snowflake-src-", "jenkinsfile_project_key": "jdbc_source", "jenkinsfile_section_comment": "// JDBC Source", "name_length_limit": { "limit_chars": 32, "reason": "ALB / target group name limit on AWS Fargate.", "deployed_name_pattern": "--", "formula": "Assumes var.connector_type defaults to 'kc-jdbc-src' (11 chars). Static overhead = 1+11+1 = 13. len(environment) + len(purpose-with-hyphens) <= 32 - 13 = 19.", "per_env_max_purpose_length": { "prod": 15, "qa": 17, "dev": 16 }, "validation_required": "HARD ERROR if exceeded — block generation and tell the user to shorten the purpose." }, "default_task_cpu": 1024, "default_task_memory": 3072, "default_max_tasks": 1, "required_inputs": [ "snowflake_database", "snowflake_schema", "snowflake_table", "snowflake_warehouse", "snowflake_role", "snowflake_user", "mode", "timestamp_column_name", "topic_prefix", "application_family", "team_name", "notification_endpoints", "escalation_notification_endpoints" ], "default_env_vars": { "MODE": "timestamp", "POLL_INTERVAL_MS": "300000", "BATCH_MAX_ROWS": "10000", "SNOWFLAKE_CONFIGURATION": "true", "SNOWFLAKE_SUBDOMAIN": "sme-delphi", "SNOWFLAKE_PRIVATE_KEY_FILENAME": "rsa_key.p8", "QUOTE_SQL_IDENTIFIERS": "never", "VALIDATE_NON_NULL": "false", "DEBUG_MODE": "false", "AWS_SECRETS_MANAGER": "true", "VALUE_CONVERTER_SCHEMAS_ENABLE": "true", "TRANSFORMS_VALUE_CONVERTER": "org.apache.kafka.connect.json.JsonConverter" }, "secrets": { "manager_pattern": "snowflake_private_key", "secret_names": ["SNOWFLAKE_PRIVATE_KEY"], "fargate_secrets_block": [ { "env_var": "SNOWFLAKE_PRIVATE_KEY_SECRET", "secret_path_pattern": "${var.environment}/${local.service_name}/SNOWFLAKE_PRIVATE_KEY" } ] }, "kafka_connect_required_env_vars": [ "Environment", "SERVICE_NAME", "CONNECT_GROUP_UNIQUE_IDENTIFIER", "CONNECT_BOOTSTRAP_SERVERS", "QUERY", "QUOTE_SQL_IDENTIFIERS", "MODE", "TIMESTAMP_COLUMN_NAME", "TOPIC_PREFIX", "POLL_INTERVAL_MS", "BATCH_MAX_ROWS", "SNOWFLAKE_CONFIGURATION", "SNOWFLAKE_SUBDOMAIN", "SNOWFLAKE_WAREHOUSE", "SNOWFLAKE_DATABASE", "SNOWFLAKE_ROLE", "SNOWFLAKE_SCHEMA", "SNOWFLAKE_USER", "SNOWFLAKE_PRIVATE_KEY_FILENAME", "VALIDATE_NON_NULL", "DEBUG_MODE", "AWS_SECRETS_MANAGER", "VALUE_CONVERTER_SCHEMAS_ENABLE", "TRANSFORMS_VALUE_CONVERTER" ], "requires_topic_creation_first": true, "notes": "Reads from a database table into a Kafka topic. The underlying JDBC source connector (`jdbc_source` image: Confluent JDBC + drivers) is GENERAL-PURPOSE and works with any JDBC-capable database (MySQL, Postgres, Sybase, Snowflake, …). THIS template is the Snowflake configuration: SNOWFLAKE_CONFIGURATION=true + Snowflake-native env vars, plus a query.sql file alongside main.tf that defines the SELECT (with TIMESTAMP_NTZ cast on the watermark column). A non-Snowflake JDBC variant would be a sibling template on the same image (DB connection URL + driver + table whitelist). Bootstrap servers come from data.aws_msk_cluster (not var)." }, "neo4j_cdc_source": { "direction": "source", "kafka_connect_dir": "neo4j_cdc_source", "ecr_image_name_default": "kc-neo-cdc-src", "datadog_connector_type": "default_source", "instance_path_pattern": "neo4j_cdc_source/", "backend_state_key_pattern": "/kafka-infra/neo4j_cdc_source//terraform.tfstate", "template": "neo4j_cdc_source.tf.j2", "variables_template": "neo4j_cdc_source_variables.tf.j2", "service_name_pattern": "kc-neo-src-${var.connector_name}", "service_name_pattern_note": "Final AWS-resource name after terraform-fargate prepends environment: `-kc-neo-src-` (e.g. `prod-kc-neo-src-fingerprinting`). The terraform-fargate module prefixes env automatically; do NOT include env in the locals.service_name expression.", "akhq_display_name_pattern": "neo-src-", "jenkinsfile_project_key": "neo4j_cdc_source", "jenkinsfile_section_comment": "// Neo4j CDC Source", "name_length_limit": { "limit_chars": 32, "reason": "ALB / target group name limit on AWS Fargate (some resources truncate or fail above 32 chars).", "formula": "len(environment) + 1 + len('kc-neo-src') + 1 + len(connector_name) <= 32 → len(connector_name) <= 20 - len(environment)", "per_env_max_connector_name_length": { "prod": 16, "qa": 18, "dev": 17 }, "validation_required": "HARD ERROR if exceeded — block generation and tell the user to shorten the user-provided name." }, "connector_class": "org.neo4j.connectors.kafka.source.Neo4jConnector", "neo4j_strategy": "CDC", "neo4j_strategy_note": "The kafka-connect/neo4j_cdc_source image hardcodes `neo4j.source-strategy: CDC` in config.template.json. QUERY-mode would require a different image. See docs_urls below.", "neo4j_version_requirement": "Neo4j 5 (on-prem) or Neo4j Aura, with CDC enabled on the target database.", "cdc_enablement_check": { "description": "CDC must be enabled on the target Neo4j database BEFORE the connector starts. The connector will fail to start if `txLogEnrichment` is not at least DIFF. Treat this as a HARD prerequisite — emit a confirmation step in the workflow and as a follow-up reminder in the validation report.", "verification_cypher": "SHOW DATABASES YIELD name, options WHERE name = $databaseName RETURN name, options.txLogEnrichment AS cdcMode", "expected_cdc_mode_values": ["DIFF", "FULL"], "if_off_value": "OFF (CDC not enabled — connector will fail)", "on_prem_enable_cypher": "ALTER DATABASE SET OPTION txLogEnrichment 'FULL'", "on_prem_enable_note": "Run as an admin against the system database. FULL is recommended for most use cases (captures both old + new property values); DIFF captures only changed properties. Requires a brief downtime on the database. See https://neo4j.com/docs/cdc/current/getting-started/", "aura_enable_instructions": "Aura: enable CDC via the Aura console for the specific database (Database settings → CDC → Enable). After enabling, the database briefly restarts. Docs: https://neo4j.com/docs/aura/auradb/managing-databases/cdc/", "aura_enable_via_api_hint": "Aura also exposes a REST API for CDC enablement: PATCH /v1beta3/instances/{instanceId}/cdc-enrichment-mode. Not used by this skill; manual console action recommended.", "what_skill_should_do": "1. Ask the user to confirm CDC has been enabled before generating the connector. 2. Provide the verification cypher above for the user to run. 3. Include the enablement instructions in the validation report follow-ups. 4. Do NOT block generation on this (the connector can be terraformed before CDC is enabled — it just won't start cleanly), but make it a top-line warning." }, "neo4j_instance_types": { "on_prem": { "description": "Self-hosted Neo4j Enterprise 5 cluster, typically reachable from inside the VPC.", "uri_pattern": "^(neo4j|neo4j\\+s|neo4j\\+ssc|bolt|bolt\\+s|bolt\\+ssc)://[^/]+(:\\d+)?$", "uri_examples": [ "neo4j+s://qa-neo4j-cluster.theorchard.io:7687", "neo4j+s://prod-neo4j-cluster.theorchard.io:7687", "neo4j://neo4j-internal.dev.theorchard.io:7687" ], "default_database_name": "graph.db", "default_encryption_enabled": "true" }, "aura": { "description": "Neo4j Aura (managed cloud).", "uri_pattern": "^neo4j\\+s://[a-z0-9]+\\.databases\\.neo4j\\.io$", "uri_format_note": "Aura URIs are always `neo4j+s://.databases.neo4j.io` — no explicit port, TLS mandatory.", "uri_examples": [ "neo4j+s://9e17b596.databases.neo4j.io", "neo4j+s://abcd1234.databases.neo4j.io" ], "default_database_name": "neo4j", "default_encryption_enabled": "true", "encryption_is_forced": true, "pause_resume_gotcha": "Aura instances that are paused-then-resumed (manually or due to inactivity on lower tiers) will have invalidated change identifiers. The connector must be re-deployed from scratch. Surface this prominently when emitting an Aura instance.", "recommended_overrides": { "neo4j_connection_timeout": "60s", "neo4j_connection_acquisition_timeout": "60s" } } }, "instance_type_auto_detect": "If the user-provided neo4j_server_uri matches `^neo4j\\+s://[a-z0-9]+\\.databases\\.neo4j\\.io$`, default neo4j_instance_type to `aura`; otherwise default to `on_prem`. Always confirm with the user.", "backup_restore_gotcha": "When the Neo4j DB is restored from backup (on-prem) or snapshot-restored / paused-resumed (Aura), existing change identifiers become invalid and the Source connector must be re-configured from scratch. Surface this as a follow-up warning whenever generating prod/qa instances. For Aura specifically, this is much easier to hit (pause-resume can happen unexpectedly).", "docs_urls": { "cdc_mode": "https://neo4j.com/docs/kafka/current/source/cdc/", "query_mode": "https://neo4j.com/docs/kafka/current/source/query/", "aura_cdc": "https://neo4j.com/docs/aura/auradb/managing-databases/cdc/" }, "module_version_overrides": { "terraform_fargate_ref": "6.3.0", "terraform_datadog_kafka_connector_ref": "6.15.3", "terraform_required_version": "1.14.0" }, "versions_template_args": { "omit_datadog_provider": true }, "default_task_cpu": 1024, "default_task_memory": 3072, "default_max_tasks": 1, "default_desired_task_count": 1, "default_minimum_capacity": 1, "default_maximum_capacity": 2, "required_inputs": [ "neo4j_server_uri", "neo4j_database_name", "neo4j_instance_type", "topic_mappings", "connector_name", "application_family", "team_name", "notification_endpoints", "escalation_notification_endpoints" ], "optional_inputs": [ "neo4j_connection_timeout", "neo4j_connection_acquisition_timeout", "neo4j_start_from_value", "topic_prefix" ], "input_shapes": { "neo4j_instance_type": "Either 'on_prem' or 'aura'. Drives URI validation, default database name, encryption enforcement, and Aura-specific follow-up warnings. Auto-detect from URI when possible; always confirm.", "neo4j_server_uri": "Full bolt/neo4j URI. On-prem typically `neo4j+s://:7687`; Aura always `neo4j+s://.databases.neo4j.io` (no port).", "neo4j_database_name": "Neo4j database name. On-prem default: `graph.db`. Aura default: `neo4j`.", "topic_mappings": "list of objects { mapping_key (UPPER_SNAKE), topic (full Kafka topic name), pattern (Cypher node/relationship pattern), key_strategy (optional, one of WHOLE_VALUE | ELEMENT_ID | ENTITY_KEYS — omit to use the global NEO4J_KEY_STRATEGY) }. The kafka-connect image's config.template.json renders these into `neo4j.cdc.topic..patterns` + `neo4j.cdc.topic..key-strategy` properties.", "connector_name": "User-provided short identifier for this instance. Used as-is in service_name (kc-neo-src-) and final deployed AWS name (-kc-neo-src-). Should be lowercase, hyphen-separated (no underscores), descriptive, and short enough for the env to keep total ≤ 32 chars. Examples: 'fingerprinting', 'fp', 'mg-v5', 'aura-graph'. Do NOT include 'kc', 'neo', 'src', or env in this value — those are added by the template.", "neo4j_connection_timeout": "Optional. Default unset (connector internal default ~30s). Recommended `60s` for Aura.", "neo4j_connection_acquisition_timeout": "Optional. Default unset. Recommended `60s` for Aura.", "application_family": "User-provided. Passed to terraform-default-tags as var.application_family. See _meta.application_family_discovery for the discovery directory. No validation against any list.", "team_name": "User-provided. Passed to terraform-default-tags v2.0.0 as var.team_name. No validation against any list." }, "pattern_syntax_reference": { "nodes": [ "(:Label) — all changes to nodes of one label", "(:Label1:Label2) — composite label", "(:User{name, surname}) — only include `name` and `surname` properties", "(:User{-address, -dob}) — exclude these properties; cannot mix include + exclude", "(:User{userId: 1001, name}) — key-based filter; requires NODE KEY constraint" ], "relationships": [ "(:Start)-[:RELTYPE]->(:End) — relationship between specific labels", "()-[:RELTYPE]->() — any-to-any relationship", "(:User)-[:BOUGHT{price}]->(:Product) — relationship with property include", "(:User)-[:BOUGHT{-card}]->(:Product) — relationship with property exclude" ], "operations_allowed": ["create", "update", "delete"], "relationship_direction_policy": "MANDATORY: when the user provides a relationship together with node labels, the skill must NOT infer the relationship direction or endpoint labels. Confirm with the user (via AskUserQuestion) the exact (:StartLabel)-[:REL]->(:EndLabel) shape — including which label is the start (arrow tail) vs end (arrow head), or `()` for any-to-any — before building topic_mappings. Relationship names alone (OWNS, BELONGS_TO, etc.) do not imply direction.", "indexed_pattern_advanced": "For per-pattern operation/changesTo/metadata filtering, use indexed form (not supported by current kafka-connect image template; would require image update): `neo4j.cdc.topic..patterns.0.{pattern,operation,changesTo,metadata.authenticatedUser,metadata.executingUser,metadata.txMetadata.}`." }, "advanced_options": { "neo4j_key_strategy_default": "Image default is WHOLE_VALUE (set via ENV in kafka-connect/neo4j_cdc_source/Dockerfile). Valid values per the kafka-connect README: WHOLE_VALUE | ELEMENT_ID | ENTITY_KEYS. The plugin does NOT emit NEO4J_KEY_STRATEGY unless explicitly provided — the image default applies. Override per-topic via the 3rd pipe-delimited segment of TOPIC_MAPPING_ (e.g. 'topic|pattern|ELEMENT_ID') or set NEO4J_KEY_STRATEGY env var globally.", "neo4j_start_from_user_provided": "Set NEO4J_START_FROM=USER_PROVIDED and NEO4J_START_FROM_VALUE= to resume from a specific point. Default NOW skips historical changes.", "neo4j_payload_mode": "Default: type-safe envelope. Set neo4j.payload-mode=COMPACT for flat JSON, or RAW_JSON_STRING for unstructured payloads. Not exposed via env var in the current image; would require config.template.json update.", "exactly_once": "Connector 5.2.0+ on Kafka 3.3.0+ clusters with KIP-618 supports exactly-once delivery (no extra config needed beyond cluster setup)." }, "default_env_vars": { "MAX_TASKS": "1", "NEO4J_ENCRYPTION_ENABLED": "true", "NEO4J_ENFORCE_SCHEMA": "false", "NEO4J_BATCH_SIZE": "1000", "NEO4J_POLL_INTERVAL": "10s", "NEO4J_START_FROM": "NOW", "NEO4J_IGNORE_STORED_OFFSET": "false", "NEO4J_KEY_CONVERTER": "org.apache.kafka.connect.storage.StringConverter", "NEO4J_VALUE_CONVERTER": "org.apache.kafka.connect.json.JsonConverter", "NEO4J_KEY_CONVERTER_SCHEMAS_ENABLE": "false", "NEO4J_VALUE_CONVERTER_SCHEMAS_ENABLE": "false", "AWS_SECRETS_MANAGER": "true", "DEBUG_MODE": "false", "CONNECT_LOG4J_ROOT_LOGLEVEL": "WARN" }, "secrets": { "manager_pattern": "neo4j_credentials", "secret_names": ["NEO4J_CREDENTIALS"], "creation_module": "terraform-secrets-manager", "secret_path_pattern": "${var.environment}/${local.service_name}/NEO4J_CREDENTIALS", "auth_via_lenses_aws_secret_provider": true }, "additional_security_group_rules": [ { "name": "allow_intra_cluster_communication", "purpose": "Allow kafka-connect tasks to communicate in distributed mode", "from_port": 8083, "to_port": 8083, "self": true, "target": "fargate_security_group_id" } ], "kafka_connect_required_env_vars": [ "Environment", "SERVICE_NAME", "CONNECT_GROUP_UNIQUE_IDENTIFIER", "CONNECT_BOOTSTRAP_SERVERS", "NEO4J_SERVER_URI", "NEO4J_DATABASE_NAME", "MAX_TASKS", "NEO4J_ENCRYPTION_ENABLED", "NEO4J_ENFORCE_SCHEMA", "NEO4J_BATCH_SIZE", "NEO4J_POLL_INTERVAL", "NEO4J_START_FROM", "NEO4J_IGNORE_STORED_OFFSET", "NEO4J_KEY_CONVERTER", "NEO4J_VALUE_CONVERTER", "NEO4J_KEY_CONVERTER_SCHEMAS_ENABLE", "NEO4J_VALUE_CONVERTER_SCHEMAS_ENABLE", "AWS_SECRETS_MANAGER", "DEBUG_MODE", "TOPIC_MAPPING_*" ], "kafka_connect_optional_env_vars": [ "NEO4J_START_FROM_VALUE", "NEO4J_KEY_STRATEGY", "TOPIC_PREFIX", "CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL", "NEO4J_CONNECTION_TIMEOUT", "NEO4J_CONNECTION_ACQUISITION_TIMEOUT", "DLQ_TOPIC_NAME", "OVERRIDE_CONFIGS" ], "requires_topic_creation_first": true, "notes": "Streams Neo4j CDC events to Kafka topics. Each topic_mapping declares one Kafka topic + a Cypher node/relationship pattern. Bootstrap servers come from data.aws_msk_cluster. Auth via NEO4J_CREDENTIALS secret in AWS Secrets Manager (read at connector startup by the Lenses AWS secret provider; not bound via Fargate secrets block). Only `allow_intra_cluster_communication` is emitted — the Jenkins pipeline-agent and shared-private prefix-list SG rules are intentionally excluded from this plugin for now." } } }