# Message Feedback (Thumbs Up / Down) -- PRD ## Status Draft ## Executive Summary ows-coda is a production AI agent handling royalties questions for the Abacus platform, but the team has zero structured signal on whether its answers are actually correct. This PRD proposes adding thumbs up / thumbs down feedback buttons to every assistant message, with optional free-text comments, persisted to the database. The feature gives users a voice, gives the product team a quality metric, and gives engineering a real-time signal on where the AI fails. ## The Problem Every day, the ows-coda agent answers questions about accounts, contracts, advances, revenue, and ledger balances -- questions where being wrong has real financial consequences. A user asks "what's the outstanding balance on this advance?" and the agent returns a number. Is it right? The team has no idea. Right now, the only way to discover that the AI gave a bad answer is when a user escalates through a support channel, an account manager catches a discrepancy in a spreadsheet, or -- worst case -- the error propagates into a financial decision unchallenged. There is no in-product mechanism for a user to say "this is wrong." There is no dashboard that shows answer quality trending down after a model update. There is no way to correlate tool failures with user dissatisfaction. The team is flying blind. Product managers cannot answer "is the AI useful?" with anything other than anecdotal evidence. Engineering cannot prioritize which tools to improve because they do not know which tools produce answers that users distrust. When a new model version is deployed, there is no before/after quality signal -- only silence, or a Slack thread three weeks later. This is not a nice-to-have. For an AI product that touches financial data, the absence of a feedback loop is a liability. Every AI product that has scaled -- ChatGPT, Copilot, Perplexity -- treats user feedback as infrastructure, not a feature. Without it, quality improvement is guesswork, regression detection is reactive, and user trust erodes invisibly. ## The Opportunity Structured feedback unlocks a cascade of capabilities: - **Quality measurement.** A simple thumbs-down rate per week gives the team its first real quality metric. If 15% of responses get a thumbs-down, that is a concrete number to drive toward 10%, then 5%. - **Regression detection.** After a model update or tool change, a spike in negative feedback surfaces problems within hours instead of weeks. - **Tool-level diagnostics.** By correlating feedback with the tools invoked during a conversation turn, the team can identify which tools (e.g., ledger balance lookups, contract queries) produce the most user-reported failures. - **Prompt and model tuning.** Free-text comments on negative feedback ("the numbers don't match what I see in the portal") provide actionable signal for prompt engineering and few-shot example curation. - **User trust.** Giving users a way to flag bad answers -- and knowing someone will see it -- builds confidence in the product. Users who feel heard are more likely to keep using the tool and more forgiving of occasional errors. - **Future fine-tuning dataset.** Thumbs-up responses with comments are candidate training examples for future model customization. ## Goals & Success Criteria | Goal | Metric | Target | | ------------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------------------- | | Users engage with feedback | Feedback rate (messages with feedback / total assistant messages) | > 5% within 4 weeks of launch | | Negative feedback is actionable | % of thumbs-down with a comment | > 30% | | Quality improves over time | Weekly thumbs-down rate | Decrease by 20% within 8 weeks (from baseline established in weeks 1-2) | | Feature is reliable | Feedback submission error rate | < 0.5% | | No user friction | Median time from thumb click to submission | < 5 seconds | ## User Stories 1. **As a royalties analyst**, I want to give a thumbs-down when the AI returns an incorrect advance balance, so the team knows the tool gave me bad data. 2. **As a royalties analyst**, I want to add a comment explaining what was wrong ("balance is off by $12k -- doesn't match the portal"), so the team can diagnose the root cause without needing to contact me. 3. **As a royalties analyst**, I want to give a thumbs-up when the AI gives a particularly helpful answer, so the team knows what kinds of responses are working well. 4. **As a user reviewing a past conversation**, I want to see my previous feedback (thumbs highlighted, comment preserved) when I scroll back through a chat, so I know which answers I already evaluated. 5. **As a user who made a mistake**, I want to change my feedback from thumbs-down to thumbs-up (or vice versa), so my feedback accurately reflects my assessment. 6. **As a user who submitted feedback in error**, I want to remove my feedback entirely, so it does not pollute the quality signal. 7. **As a product manager**, I want feedback data persisted to the database with message-level granularity, so I can query it for quality dashboards and trend analysis. 8. **As an engineer**, I want feedback correlated to specific message UUIDs, so I can join feedback data with tool invocation logs and conversation context for debugging. ## Proposed Solution Add thumbs-up and thumbs-down buttons to every assistant message in the chat interface. Clicking either thumb opens a lightweight modal with an optional comment textarea (max 1,000 characters). Submitting persists the feedback to the server via a `PUT` endpoint that performs an upsert -- users can revise feedback at any time without creating duplicate records. A trash icon in the modal allows complete removal via a `DELETE` endpoint, guarded by a confirmation alert. The UI uses optimistic updates: the thumb highlights immediately on submit, and reverts silently on API failure (with a toast notification). Feedback state is stored per-message in the client and rehydrated from the server on page reload. A prerequisite architectural fix ensures the SSE `done` event includes real database message UUIDs so the client can reference persisted messages when submitting feedback. ## User Experience **Button placement.** Thumbs-up and thumbs-down icons appear in the assistant message action bar, alongside existing actions (copy, regenerate). They use the same `Action` component for visual consistency. **Visual states.** Inactive thumbs render in the default muted foreground color. An active thumbs-up renders in green (`text-green-500`). An active thumbs-down renders in red (`text-red-500`). Both states are immediately visible without hover. **Interaction flow:** 1. User clicks a thumb icon. 2. A dialog opens with the modal title ("Give positive feedback" or "Give negative feedback") and an optional comment textarea. 3. User optionally types a comment (up to 1,000 characters). 4. User clicks "Submit" -- the dialog closes, the thumb highlights, and the API call fires in the background. 5. On API error, the thumb reverts and a toast notification appears. **Changing feedback.** Clicking the opposite thumb when feedback already exists opens the modal with an overwrite notice ("Your previous feedback will be replaced.") and a cleared comment field. Clicking the same thumb re-opens the modal with the existing comment pre-filled and a trash icon visible for removal. **Removing feedback.** The trash icon (visible only when re-opening existing feedback of the same rating) triggers a confirmation alert dialog. On confirm, the feedback is deleted and both thumbs return to their inactive state. **Streaming state.** Feedback buttons only appear when the message has content (consistent with existing action bar visibility). During active streaming, no feedback buttons are shown. ## Benefits **Business benefits:** - First quantitative quality metric for the AI agent -- replaces anecdotal signal with data. - Enables data-driven prioritization of prompt improvements, tool fixes, and model upgrades. - Regression detection within hours of deployment, not weeks. - Free-text comments create a lightweight, always-on user research channel. - Foundation for future quality dashboards, SLA reporting, and stakeholder confidence metrics. **User benefits:** - Users gain agency: a bad answer is no longer a dead end -- they can flag it. - Seeing feedback persisted across sessions signals that the team listens and acts. - Revise-or-remove capability means users are never locked into accidental feedback. **Engineering benefits:** - Message-level feedback joined with tool invocation logs enables root-cause analysis of AI failures. - Feedback rate serves as a proxy health metric alongside latency and error rate. - The `done` event UUID fix (prerequisite) unblocks future features that require real message IDs on the client (per-token feedback, real-time analytics, crash recovery). ## Costs **Engineering effort:** - Backend (FeedbackService.delete, PUT/DELETE routes, GET /messages enrichment, SSE done event fix): ~3 person-days - API package (types, constants, client methods): ~0.5 person-days - Client (Message type updates, orchestrator wiring, MessageFeedbackActions component, message-item integration, i18n): ~3 person-days - Testing (unit tests for service and handlers, build verification): ~1 person-day - **Total: ~7.5 person-days** (1 engineer, ~1.5 calendar weeks) **Infrastructure costs:** - Minimal. The `message_feedback` table already exists in the database. Each feedback row is approximately 100 bytes. At 10,000 messages/month with a 10% feedback rate, that is ~100 KB/month of additional storage -- negligible. **Maintenance burden:** - Low. The feature is self-contained with no background jobs, no cron tasks, and no external service dependencies beyond the existing Prisma/MySQL stack. - The `MessageFeedbackActions` component is fully encapsulated -- no prop threading, no global state -- minimizing coupling with the rest of the UI. **Opportunity cost:** - 1.5 engineering weeks not spent on other features. Justified by the fact that every subsequent AI quality improvement depends on having a feedback signal to measure against. ## Dependencies | Dependency | Status | Risk | | ------------------------------------------------- | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | `message_feedback` DB table | Exists (schema and Prisma model already in place) | None | | `FeedbackService.upsert()` and `getForMessages()` | Exist and are fully tested | None | | SSE `done` event with real message UUIDs | Must be implemented (prerequisite fix in Task 3) | Low -- the UUIDs are already generated in `turn-assembler.ts`; they just need to be threaded to the SSE event | | `Message` model `depth` field in DB | Exists (used to correlate in-memory messages with DB rows for GET /messages) | None | | Auth middleware | Applied globally; new routes inherit it | None | | Radix UI Dialog / AlertDialog components | Already in the client component library | None | ## Risks & Mitigations | Risk | Likelihood | Impact | Mitigation | | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Low adoption** -- users do not click thumbs | Medium | High (no quality signal) | Monitor feedback rate in first 2 weeks. If below 2%, consider in-app prompts after high-confidence answers, or periodic nudges. Keep the interaction to 1-2 clicks to minimize friction. | | **Noisy feedback** -- users click thumbs-down for reasons unrelated to answer quality (e.g., the AI could not help because the data does not exist) | Medium | Medium | Free-text comments provide context to distinguish "wrong answer" from "unanswerable question." Future: add structured reason codes to thumbs-down. | | **Message ID resolution timing** -- feedback submitted before the `done` event (synthetic ID) would 404 | Low | Medium | Feedback buttons are only shown on non-streaming messages. The `done` event fires before the user can reasonably interact with thumbs. Synthetic IDs (prefixed with `user-` or `assistant-`) are never valid DB UUIDs, providing a natural guard. | | **Optimistic update flicker** -- API call fails after thumb highlights | Low | Low | Revert to previous state on error with a toast notification. The visual flicker is sub-second and non-destructive. | | **Comment abuse** -- users submit inappropriate content in free-text comments | Low | Low | Comments are internal-facing (no public display). Max length of 1,000 characters limits volume. Review as part of quality analysis workflow. | | **codaServices unavailable** -- Prisma not configured in some environments | Low | Low | Routes return 503; client shows error toast. Feedback buttons still render (graceful degradation). | ## Timeline & Milestones The implementation plan is structured as 10 sequential tasks, each independently committable and testable. | Phase | Tasks | Estimated Duration | Deliverable | | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------- | | **Phase 1: Backend foundation** | Task 1 (FeedbackService.delete), Task 2 (API package types/constants/client) | 1.5 days | New service method, shared types, API client methods | | **Phase 2: Server plumbing** | Task 3 (SSE done event UUIDs), Task 4 (PUT/DELETE feedback routes), Task 5 (GET /messages enrichment) | 2 days | Real message IDs on client, feedback CRUD endpoints, feedback in message history | | **Phase 3: Client implementation** | Task 6 (Message type + orchestrator), Task 7 (i18n keys), Task 8 (MessageFeedbackActions component), Task 9 (message-item wiring) | 3 days | Fully functional feedback UI | | **Phase 4: Verification** | Task 10 (full build + test pass, manual smoke test) | 0.5 days | Green CI, verified UX | | **Total** | | **~7 person-days** | | ## Open Questions 1. **Analytics/reporting UI.** This PRD covers data collection only. When and how should the team visualize feedback trends? A follow-up PRD for a feedback dashboard (thumbs-down rate over time, top negative-feedback tool invocations, comment word clouds) would close the quality loop. 2. **Structured reason codes.** Should thumbs-down feedback include a set of predefined reasons (e.g., "incorrect data," "incomplete answer," "too slow") in addition to free-text? This would enable automated categorization but adds UI complexity. Recommend evaluating after 4 weeks of free-text data to identify the most common complaint categories. 3. **Feedback on user messages.** Currently out of scope. Are there use cases (e.g., "I phrased this poorly") that warrant user-message feedback? 4. **Notification to the team.** Should high-severity feedback (e.g., thumbs-down with a comment mentioning "wrong balance") trigger a Slack notification or Jira ticket? This could accelerate response time but risks alert fatigue. 5. **Earlier message persistence.** The current architecture writes messages to the DB only after the full stream completes. Persisting messages at stream start would make real UUIDs available earlier and enable features like per-token feedback and crash recovery. This is tracked as a follow-up architectural improvement. 6. **Feedback rate targets by role.** Should the 5% feedback rate target be broken down by user role (analyst vs. manager vs. admin)? Different roles may have different engagement patterns.