# Platform Service Troubleshooting ## Permission Denied — Debugging by Pipeline Step When a user reports "access denied," the `CheckResponse` includes a `step` number indicating where the pipeline rejected the request. Use this to diagnose: | Step | Name | Denied Reason | Investigation | | ---- | -------------------------- | -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | | 0 | Input validation | Invalid permission name | Check the permission string format (must match `^[a-z][a-z0-9_.]*\.[a-z][a-z0-9_]*$`, max 255 chars) | | 0 | Internal error | Pipeline threw unexpectedly | Check platform service logs for stack trace. Fail-closed: any exception → DENIED | | 3 | Tenant status | Tenant suspended/deactivated/not found | `GetTenant` RPC to check status. If suspended, only a super admin can reactivate | | 4 | User status | User not active in tenant | `GetUser` RPC to check TenantUser status. May be invited/suspended/deactivated | | 5 | Consent (stub) | Consent/DPA not recorded | Phase 2 — currently a no-op stub | | 6 | IP allowlist (stub) | IP not in allowlist | Phase 2 — currently a no-op stub | | 7 | Plan feature gating | Module not in tenant's plan | Check which modules are seeded for the tenant. The feature key is the module slug (e.g., `tools.snowflake` from permission `tools.snowflake.query`) | | 8 | Deny override | Explicit deny for this permission | Check `UserPermission` rows where `type=deny` for this user. Deny overrides beat everything, including super admin | | 9 | Super admin | Analytics/product level restrictions | Analytics admins must use aggregation layer. Product admins can only access test tenants | | 11 | Role permissions | Permission not in any assigned role | Check `UserRole` assignments and role permission sets. Use `GetEffective` RPC for the full picture | | 12 | Resource scope (stub) | Resource-level restriction | Phase 2 — currently a no-op stub | | 13 | Policy conditions | ABAC condition failed | Check `PolicyCondition` rows for this permission. The reason string includes the failing condition type | | 14 | Data classification (stub) | Data classification block | Phase 2 — currently a no-op stub | | 15 | Step-up auth (stub) | Step-up authentication required | Phase 2 — currently a no-op stub | ### Quick Diagnosis via GetEffective RPC ``` GetEffective({ tenantId: "...", userId: "..." }) → Returns: grantedPermissions[], deniedPermissions[], roles[], isSuperAdmin, superAdminLevel ``` This gives the full effective permission set for a user — the fastest way to see what they can and can't do. --- ## Cache Issues ### Stale Permissions After Role Change **Symptom:** User's permissions don't reflect a recent role assignment/removal. **Cause:** Cached permissions haven't expired yet (TTL-based). **Fix:** 1. Check if the role change handler called cache invalidation (it should be synchronous) 2. Manual Redis cache clear: ```bash # Clear specific user's cached permissions redis-cli DEL "perm:{tenantId}:{userId}:effective" redis-cli DEL "perm:{tenantId}:{userId}:deny" redis-cli DEL "perm:{tenantId}:{userId}:user_status" # Clear specific role's cached permissions redis-cli DEL "perm:{tenantId}:role:{roleId}" ``` 3. Wait for TTL expiry (effective permissions: 30s, role permissions: 15min) ### Cache Key Reference | Pattern | TTL | Invalidated By | | ----------------------------------------- | ----- | ----------------------------- | | `perm:{tenantId}:tenant_status` | 30s | `invalidateTenantStatus` | | `perm:{tenantId}:{userId}:user_status` | 30s | `invalidateUserStatus` | | `perm:superadmin:{userId}` | 5min | `invalidateSuperAdminStatus` | | `perm:{tenantId}:{userId}:deny` | 30s | `invalidateUserDenyOverrides` | | `perm:{tenantId}:{userId}:effective` | 30s | `invalidateUserPermissions` | | `perm:{tenantId}:role:{roleId}` | 15min | `invalidateRole` | | `perm:{tenantId}:plan` | 1hr | `invalidatePlan` | | `perm:{tenantId}:conditions:{permission}` | 15min | `invalidatePolicyConditions` | ### Nuclear Option: Clear All Tenant Cache ```bash redis-cli --scan --pattern "perm:{tenantId}:*" | xargs redis-cli DEL redis-cli --scan --pattern "access:{tenantId}:*" | xargs redis-cli DEL ``` Or via the service: `cache.invalidateTenant(tenantId)` (uses SCAN + pipeline DEL). **Note:** This does NOT clear super admin keys (`perm:superadmin:*`) — those are cross-tenant by design. --- ## Audit Buffer Issues ### Buffer Growing Unbounded **Symptom:** `audit:buffer` LLEN keeps increasing. **Cause:** Drain worker not running, or DB inserts failing. **Investigation:** ```bash # Check buffer depth redis-cli LLEN "audit:buffer" # Check drain worker logs # Look for: "Audit drainer: drain cycle failed" or "batch inserted" ``` **Fix:** 1. If drain worker is stopped: restart the platform service 2. If DB inserts are failing: check Aurora MySQL connectivity and `AuditLog` table permissions 3. If buffer > 10,000: the drain worker logs a warning. It processes up to 1,000/cycle (1s interval), so it will catch up if the underlying issue is resolved ### Plaintext PII in Buffer (Security Incident) **This should never happen.** The audit writer encrypts PII before `LPUSH`. If plaintext is found: 1. **Immediately:** Flush the buffer to prevent further exposure 2. **Investigate:** Check if the `encryptEntry` function was bypassed or if the master key is misconfigured 3. **Verify:** `AUDIT_MASTER_KEY` env var is set (if absent, a random key is used which is correct but not recoverable across restarts) --- ## Tenant Lifecycle Issues ### Suspending a Tenant **When:** Contract violation, non-payment, security incident. **Procedure:** ``` 1. SuspendTenant RPC (requires super admin) ├── tenantId: "..." └── reason: "Non-payment — invoice #1234 overdue 30 days" → Updates status to "suspended" → Purges ALL cached data for tenant (synchronous) → Emits tenant.status.changed event → All users immediately lose access (step 3 denies) ``` **Impact:** All users in the tenant are immediately blocked. No data is deleted — the tenant can be reactivated. ### Reactivating a Tenant **Procedure:** ``` 1. ReactivateTenant RPC (requires super admin) ├── tenantId: "..." → Updates status to "active" → Purges ALL cached data (symmetric with suspend) → Users regain access on next request ``` ### Deactivating a User (Permanent) **Procedure:** ``` 1. DeactivateUser RPC ├── tenantId: "..." ├── userId: "..." └── reason: "Employee termination" → Updates TenantUser status to "deactivated" → Invalidates user status cache → User cannot be reactivated (permanent) ``` **Note:** Deactivation is irreversible. For temporary blocks, use `SuspendUser` instead. --- ## Super Admin Issues ### Granting Super Admin Access ``` GrantSuperAdmin RPC ├── userId: "..." ├── level: FULL | READ_ONLY | SUPPORT | PRODUCT | ANALYTICS └── grantedBy: "" ``` **Level restrictions:** - `FULL` — unrestricted access (bypasses steps 10-13) - `READ_ONLY` — same as full in Phase 1 (RPC-layer enforcement in Phase 2) - `SUPPORT` — same as full in Phase 1 - `PRODUCT` — can only access test tenants (non-test → denied at step 9) - `ANALYTICS` — must use aggregation layer (always denied at step 9) **Important:** A user can only have ONE active super admin grant. Attempting to grant when one exists returns `AlreadyExists`. ### Revoking Super Admin Access ``` RevokeSuperAdmin RPC ├── userId: "..." └── revokedBy: "" → Sets revokedAt timestamp (soft delete) → Invalidates super admin cache (5min TTL) → User loses elevated access on next request (or within 5min if cached) ``` ### Multiple Active Grants (Should Not Happen) If `findFirst` for a user returns unexpected results, check for multiple non-revoked rows: ```sql SELECT * FROM super_admins WHERE user_id = '...' AND revoked_at IS NULL; ``` The resolver uses `ORDER BY granted_at DESC` to pick the most recent grant. If duplicates exist, revoke the older ones. --- ## Health Check Failures ### `/health` Returns Non-200 The basic health check should always return 200. If it doesn't, the service is not running. ### `/health/ready` Returns 503 **Meaning:** One or more dependencies are unhealthy. **Response body:** ```json { "status": "degraded", "cache": "unavailable" } ``` **Investigation:** - `cache: "unavailable"` — Redis is disconnected. Check ElastiCache connectivity. The service still works (NullPermissionCache fallback) but every request hits the DB. --- ## Common Error Codes | Code | Meaning | Common Causes | | -------------------- | -------------------------------- | ------------------------------------------------------------------------- | | `InvalidArgument` | Missing or invalid request field | Empty tenantId, userId, or permission | | `NotFound` | Entity doesn't exist | Wrong tenant/user ID, user not in tenant | | `FailedPrecondition` | Invalid state transition | Suspending an already-suspended tenant, reactivating a non-suspended user | | `AlreadyExists` | Duplicate entity | Duplicate tenant slug, duplicate super admin grant | | `Internal` | Unexpected server error | Check platform service logs |