# Feature Spec: Engagement and Usage Observability **Status:** Phases 1-3 complete **Scope:** New `kb_draft_retrievals` table, admin dashboard page, navbar widgets for line managers and experts, per-entry KB impact stat **Pre-mortem risks addressed:** §1 — "Line managers do not submit consistently"; §2 — "Experts stop reviewing drafts seriously"; §3 — "The knowledge base becomes clutter instead of compounding value" --- ## Problem Ask QIEN can fail not because it breaks, but because people stop using it. The pre-mortem identifies this as the core adoption risk: line managers submit only when reminded, experts drift back to outside channels, and the knowledge base accumulates entries that nobody uses or maintains. There are two distinct problems to solve: 1. **Invisible adoption decay.** The product team and operators have no reliable signal for whether users are building habits or quietly disengaging. Without this, intervention comes after the damage is done. 2. **No intrinsic motivation to contribute.** Line managers get no feedback that submission leads to outcomes. Experts get no signal that the KB articles they write are doing anything. The KB starts empty at every new tenant, and there is no pull to seed it. This feature addresses both by building a usage data foundation, surfacing operator-level health metrics in an admin dashboard, and showing each user a lightweight personal view of their own contribution and impact. --- ## Design intent The engagement surfaces are not a leaderboard or a scoring system. The goal is **personal progress and impact visibility** — users see their own activity and the downstream effect of their work, not a comparison against colleagues. This is intentional: the product is used in a regulated-domain quality context where competitive mechanics would feel misaligned and could create perverse incentives (e.g. submitting low-quality tickets to maintain a streak). The KB impact widget is the highest-leverage element. It shows experts that the articles they write are being used to pre-fill responses — a direct signal that their knowledge is compounding. This also creates a natural quality loop: an article with zero pre-fills is a candidate for rewording, merging, or deletion. No admin intervention needed. The article count widget solves a **cold-start problem**: when a tenant is newly onboarded, there are no pre-fills yet, so the impact counter starts at zero and cannot motivate early seeding. A simple "articles written" counter rewards contribution before the flywheel is turning. --- ## Recommended phase order | Phase | Scope | Unlocks | |---|---|---| | 1 | KB retrieval tracking | The data foundation for pre-fill impact metrics | | 2 | Admin platform health dashboard | Operator visibility into adoption, ticket flow, draft quality, KB health | | 3 | User engagement widgets | Line manager navbar widgets; expert navbar and per-entry KB impact | Each phase is independently shippable. Phase 1 is a prerequisite for the KB impact stats in Phase 3 but not for the admin dashboard metrics in Phase 2 (which are derived from existing tables). --- ## Phase 1 — KB Retrieval Tracking ### Goal When the response-draft agent retrieves KB entries to generate a draft, record which entries were retrieved for which ticket. This is the raw event that powers the "pre-fill impact" metric in Phase 3. ### Metric definition A KB entry **contributed to a pre-filled response** when: - It was retrieved by the response-draft agent during draft generation for ticket T, **and** - Ticket T subsequently received a sent expert response (ticket status reached `RESPONDED`) This is Option A: retrieval-based, not edit-distance-based. It is intentionally conservative — we credit the entry for being part of the context that produced a used draft, not for surviving expert edits. This can be upgraded to edit-distance gating in a later iteration once draft quality tracking is more mature. ### Schema: new `kb_draft_retrievals` table ```sql kb_draft_retrievals id UUID PK DEFAULT gen_random_uuid() tenant_id UUID NOT NULL FK → tenants.id ON DELETE CASCADE kb_entry_id UUID NOT NULL FK → knowledge_entries.id ON DELETE CASCADE ticket_id UUID NOT NULL FK → tickets.id ON DELETE CASCADE created_at TIMESTAMP NOT NULL DEFAULT now() ``` `tenant_id` is denormalized for scoping consistency with all other tables in the project. Both `input.tenantId` and the retrieved entry's tenant are available at insert time — use `input.tenantId`. One row per (KB entry, ticket) pair at the time the response-draft agent runs. If the agent re-runs for the same ticket (e.g. after a line manager follow-up), insert new rows — do not upsert. This preserves the retrieval history and allows future analysis of which entries survive re-runs. Index on `(tenant_id, kb_entry_id)` for the per-entry impact query. Index on `ticket_id` for ticket-level lookups. ### Capture point In `src/inngest/response-draft-agent.ts`, after KB entries are retrieved and before the draft is generated, insert one `kb_draft_retrievals` row per retrieved entry. Both `input.tenantId` and `input.ticketId` are available; `sources[n].id` is the KB entry ID. No other changes to agent behaviour. This is append-only instrumentation. The insert does not need to be in the same DB transaction as the `ai_artifacts` insert — orphaned rows from a failed draft run are harmless and count as valid retrieval evidence. ### Impact query ```sql -- Count of distinct tickets for which entry E contributed to a sent response SELECT COUNT(DISTINCT kdr.ticket_id) FROM kb_draft_retrievals kdr JOIN tickets t ON t.id = kdr.ticket_id WHERE kdr.kb_entry_id = :entry_id AND kdr.tenant_id = :tenant_id -- tenant scope; prevents cross-tenant leakage AND t.status = 'RESPONDED' ``` For the aggregate expert widget (sum across all entries by author, including soft-deleted entries): ```sql SELECT COUNT(DISTINCT kdr.ticket_id) FROM kb_draft_retrievals kdr JOIN knowledge_entries ke ON ke.id = kdr.kb_entry_id JOIN tickets t ON t.id = kdr.ticket_id WHERE ke.created_by_id = :user_id -- actual Drizzle column name is created_by_id AND kdr.tenant_id = :tenant_id AND t.status = 'RESPONDED' -- intentionally no filter on ke.is_deleted — historical contributions persist after soft-delete ``` ### Phase 1 status - [x] `kb_draft_retrievals` table exists with tenant, KB entry, ticket, and created-at columns - [x] Table has `(tenant_id, kb_entry_id)` and `ticket_id` indexes - [x] Response-draft agent inserts one retrieval row per retrieved KB entry before draft generation - [x] Response-draft reruns append new rows instead of upserting - [x] Unit and integration tests cover row construction and DB capture --- ## Phase 2 — Admin Platform Health Dashboard ### Goal Give operators and the product team a single page showing whether users are building habits, where the workflow is breaking down, and whether the KB is accumulating value. All metrics are scoped to the active tenant. ### Route `/admin/platform-health` Add to the existing admin nav as an entry in `ADMIN_EXTRA_LINKS` in `NavLinks.tsx`, alongside `/admin/ai-usage`, `/admin/users`, and `/admin/config`. The page must be gated with `requireAdminUser()`, consistent with all other admin pages. ### Active user definition A user is **active in a period** if they performed at least one of the following actions during that period: - Submitted a ticket (line manager) — query `tickets.created_at` by `submitter_id` - Sent an expert response (expert) — query `ticket_messages` for `role = 'expert'` by `author_id` - Created or approved a KB entry (expert) — query `audit_log` for actions `knowledge.created` or `knowledge.proposal_accepted` by `actor_id` Login without action does not count. This measures behavioural engagement, not presence. ### Dashboard panels #### Adoption | Metric | Description | |---|---| | Active users this week / this month | Count of distinct users meeting the active-user definition, broken down by role | | User activation rate | % of invited, enabled users who have ever performed at least one active-user action | | Tickets per active line manager per week | Rolling 8-week trend chart; the core habituation signal from the pre-mortem | | Weekly submission trend | Total tickets submitted per week for the last 8 weeks | #### Ticket flow health | Metric | Description | |---|---| | Orphaned tickets | Tickets in a non-terminal state (`SUBMITTED`, `AWAITING_EXPERT`, `EXPERT_REVIEWING`) with no `ticket_messages` row where `role = 'expert'` created in the last 48 hours — `ticket.updatedAt` is not used because AI pipeline transitions also bump it | | Median time to first expert action | Median hours from ticket submission to first expert response or status change | | Follow-up rate | % of RESPONDED tickets that received at least one line manager follow-up — query `audit_log` for `ticket.status_changed` events where payload `to = AWAITING_REPLY` | | Tickets resolved without external channels | Proxy only: % of tickets reaching `CLOSED` without ever entering `NEEDS_INFO`. This is not a true measure of whether users went outside the app (that is untrackable); it measures whether the expert needed to request clarification. The limitation should be noted in the UI. | #### Draft quality | Metric | Description | |---|---| | Draft send-through rate | % of AI-generated drafts that were accepted and sent by the expert (uses existing `expert_accepted` field) | | Draft edit rate | % of sent drafts where `expert_edited = true` | | Draft regeneration rate | % of tickets where the response-draft agent ran more than once | #### KB health | Metric | Description | |---|---| | Total active KB entries | Entries where `is_deleted = false` | | Entries used in at least one draft | Distinct `kb_entry_id` values in `kb_draft_retrievals` | | Entries never retrieved | Active entries not present in `kb_draft_retrievals` — candidates for review or deletion | | Entries contributing to sent responses | Distinct entries where at least one retrieval maps to a RESPONDED ticket | ### Scope and access - Scoped to `session.tenantId` — operators see only their own tenant. - Accessible to users with the `admin` role only. - No cross-tenant comparison view in this phase. ### Data sources All metrics in Phase 2 are derived from existing tables (`tickets`, `knowledge_entries`, `knowledge_entry_revisions`, `tenant_memberships`, `ticket_processing_runs`, `ai_call_logs`) plus the new `kb_draft_retrievals` table from Phase 1. No additional event-capture work is required. ### Phase 2 status - [x] `/admin/platform-health` page exists and is gated with `requireAdminUser()` - [x] Admin nav includes the platform health page - [x] Dashboard renders adoption, ticket flow health, draft quality, and KB health panels - [x] Metrics are scoped to the active tenant - [x] Unit, integration, and E2E tests cover the platform health dashboard --- ## Phase 3 — User Engagement Widgets ### Goal Surface a lightweight personal view to each user on their existing navbar, showing their own activity and the downstream impact of their contributions. No competition. No pressure. Just visible progress. ### 3a — Line manager navbar widgets **Placement:** Compact widget area rendered in `Nav.tsx` (the async server component), visible on all line manager pages alongside the existing nav links. Shown as small labelled counters, not a banner or modal. On narrow viewports, collapse to icons with a tooltip. **Widget 1 — Tickets this month** - Label: `Tickets this month` - Value: count of tickets submitted by this user in the current calendar month - Resets on the first of each month - Tapping/hovering shows a small popover: count for last month for reference **Widget 2 — Streak** - Label: `Week streak` - Value: number of consecutive calendar weeks (Mon–Sun) in which the user submitted at least one ticket - Any submission counts — no quality gate - A broken streak resets to 0 and begins counting again from the next submission week - The intent is gentle momentum reinforcement, not punishment for missing a week - **Clicking the streak counter navigates to `/profile`**, where a full activity calendar is shown (see Section 3d) Algorithm: 1. Fetch all distinct ISO weeks in which the user has a submitted ticket, ordered descending 2. Walk backward from the current week; count consecutive weeks without a gap 3. If the current week has no submission yet, the streak is the count of consecutive prior weeks (streak is not yet broken for the current week) ISO week boundaries (Mon 00:00 to Sun 23:59) must be calculated in the **user's stored `time_zone`** (from `users.time_zone`), not UTC. Using UTC would cause streaks to appear broken or extended for non-UTC users by up to a day. **Widget 3 — Issues resolved** - Label: `Resolved` - Value: count of tickets submitted by this user that have reached `CLOSED` status, all time - Provides a sense of cumulative outcome, not just submission volume ### 3b — Expert navbar widgets **Placement:** Compact widget area in the expert navbar, visible on all expert pages. **Widget 1 — KB articles written** - Label: `KB articles` - Value: count of active (non-deleted) KB entries created by this user in this tenant - Purpose: rewards early seeding before pre-fill data exists; gives experts a concrete contribution number - Does not include entries the expert edited but did not author - **Clicking navigates to `/profile`** **Widget 2 — Pre-fill impact** - Label: `Responses helped` - Value: cumulative count of distinct tickets for which at least one of this expert's KB entries contributed to a sent response (Phase 1 impact query, scoped to this user) - This is the primary motivation metric for ongoing KB quality - Includes retrievals from soft-deleted entries — the contribution happened and should not disappear from the expert's record - Tapping/hovering shows a small popover: "Your KB entries were used in drafts for X tickets that received a response" - **Clicking navigates to `/profile`** **Component placement:** Widgets live in `Nav.tsx` (the async server component that wraps `NavLinks`), not inside `NavLinks.tsx` itself. `Nav.tsx` already fetches from the DB for session resolution; adding widget queries there is consistent with the existing pattern. `NavLinks.tsx` remains a pure navigation component. Both widgets are computed at page load for the current user. At current user volumes, on-the-fly queries are sufficient. If query time becomes noticeable at scale, cache the values server-side with a short TTL — but ensure `Nav.tsx` opts out of Next.js page-level caching (`cache: 'no-store'` on the relevant fetch or equivalent) to prevent stale widget values being served from the page cache. ### 3c — Per-entry KB impact stat **Placement:** On the KB article detail page (`/knowledge/[id]`), alongside the existing revision history. **Display:** A single line: `Used in X pre-filled responses` - Computed from the per-entry impact query (Phase 1) - Visible to experts and admins only — the KB article detail page remains behind `requireExpertUser()` and is not opened to line managers - Entries with 0 pre-fills show `Used in 0 pre-filled responses` — this is intentional; it makes low-performing entries visible as candidates for improvement - For soft-deleted entries the stat is still shown in the revision history view if accessible to an expert **Design intent (reiterated):** An expert who can see that one of their entries has helped pre-fill 18 responses and another has helped 0 has a clear, non-judgmental signal about which article needs work. This is the core KB quality loop. No admin prompt required. ### 3d — Profile page **Route:** `/profile` The profile page is the permanent home for personal settings and personal engagement history. It consolidates what is currently at `/settings` (display name editing) and adds timezone, the activity calendar, and a summary of the user's own stats. The `/settings` route should redirect to `/profile`. #### Layout **Personal info section** (top) - Display name — editable in place, same behaviour as current `/settings` - Email address — read-only - Timezone — editable; used for calendar display and, when nudge emails are introduced, for send-time optimisation **Activity calendar section** (middle) A GitHub-style contribution heatmap showing the last 52 weeks (approximately one year), with the current week on the right. Each cell represents one calendar day. Cell colour intensity is determined by the number of key actions performed on that day. Activity definition by role: - **Line manager:** ticket submissions - **Expert:** responses sent + KB articles created or approved Colour scale: four levels — no activity (neutral/empty), low (1 action), medium (2–4 actions), high (5+ actions). Exact palette to match the product's existing colour system rather than copying GitHub green. Display details: - Month labels along the top axis - Day-of-week labels (Mon / Wed / Fri) on the left axis to keep it compact - Hovering a cell shows a tooltip: date + action count (e.g. "3 May 2026 — 2 tickets submitted") - The current week's incomplete cells are shown at reduced opacity Below the calendar, display the streak count with a brief label: `X-week streak` — this contextualises the navbar counter for users who clicked through. **Stats summary section** (bottom) A compact row of the same figures shown in the navbar, presented with a little more label context: Line manager: - Tickets this month | Current streak | Total resolved Expert: - Active KB articles | Total responses helped by your KB These are not interactive — they mirror the navbar widgets and are present here for completeness when a user is reviewing their profile. #### Access - Every authenticated user can view their own `/profile` page - No user can view another user's profile page in this phase ### Phase 3 status - [x] Line manager nav widgets show monthly tickets, week streak, and resolved count - [x] Expert nav widgets show active KB article count and responses helped - [x] Week streaks and activity calendars use the user's stored time zone - [x] `/knowledge/[id]` shows `Used in X pre-filled responses` - [x] `/profile` consolidates display name, email, time zone, activity calendar, and personal stats - [x] `/settings` redirects to `/profile` - [x] Unit, integration, and E2E tests cover the engagement widgets/profile behavior --- ## Metrics to watch after shipping These are the signals that will tell us whether this feature is doing its job: | Signal | What it means if it moves | |---|---| | KB articles per expert per tenant (first 30 days) | Whether the article-count widget is accelerating seeding | | Pre-fill impact growth over time | Whether the KB is compounding as intended | | Line manager streak distribution | Whether habit formation is improving or stalling | | Admin dashboard weekly active user trend | Whether the overall adoption curve is healthy | --- ## Explicit non-goals for this phase - **Nudge emails** — passive visibility only. Nudge infrastructure (timing, wording, frequency, A/B testing) is a future feature that builds on the data captured here. - **Leaderboards or cross-user comparison** — all widgets are scoped to the current user's own data. - **Competitive mechanics** — no points, badges, or levels. - **Cross-tenant admin views** — operator dashboard is scoped to one tenant at a time. - **Edit-distance-based draft quality** — deferred; Option A retrieval tracking is sufficient for this phase. --- ## Open questions None outstanding. The following were resolved during spec authoring: | Question | Decision | |---|---| | Streak: compact counter or calendar visual in navbar? | Compact counter in navbar; clicking opens `/profile` where a full GitHub-style activity calendar is shown | | Per-entry impact visible to line managers? | No — KB article detail page remains expert/admin-only | | Pre-fill count persist after soft-delete in expert aggregate? | Yes — the contribution happened and should not be erased from the expert's record | | Orphaned-ticket threshold? | 48 hours; adjust based on observed expert response times in early tenants | | `processing_run_id` in `kb_draft_retrievals`? | Dropped — not used by any impact query; adds agent complexity with no payoff this phase | | `kb_draft_retrievals` needs `tenant_id`? | Yes — added for scoping consistency and cross-tenant safety |