# Feature Spec: AI Observability, Fallback, and Per-Tenant API Keys

**Status:** Phase 1 complete; Phases 2-3 deferred  
**Scope:** `src/lib/ai-gateway.ts`, `src/db/schema/tenants.ts`, new `ai_call_log` table, admin UI  
**Pre-mortem risk addressed:** §8 — "Cost, latency, and reliability of AI jobs become invisible product debt"

---

## Problem

Every LLM call goes through `ai-gateway.ts`, but nothing is recorded when calls succeed or fail. The current gateway hardcodes `process.env.ANTHROPIC_API_KEY` for every tenant and every agent, making it impossible to:

- Answer "what did tenant X cost this month?"
- Detect and route around provider outages or rate limits
- Let a tenant supply their own API key and be billed directly
- Understand which agents fail most often, or why

These three features — observability, fallback, per-tenant keys — share the same insertion point (the gateway) and the same schema concern (per-call logging + per-tenant key config), so they should be designed together even if they ship in phases.

---

## Recommended Phase Order

| Phase | Scope | Unlocks |
|---|---|---|
| 1 | Call logging (observability) | Visibility into cost/latency/failures; no user-facing changes |
| 2 | Fallback provider/model | Reliability during outages; no tenant-facing changes |
| 3 | Per-tenant API keys | Customer billing isolation; enables self-service tenants |

**Each phase is independently shippable.** Phase 1 is the highest-priority because it is a prerequisite for everything else (you cannot tune fallback thresholds or charge tenants without data).

---

## Phase 1 — AI Call Logging

### Goal

Every LLM call writes one row to `ai_call_logs`. No call is invisible. Operators can query by tenant, agent, ticket, provider, model, date range, and outcome.

### Schema: new `ai_call_logs` table

```sql
ai_call_logs
  id                UUID          PK DEFAULT gen_random_uuid()
  tenant_id         UUID          NOT NULL FK → tenants.id
  ticket_id         UUID          NULLABLE FK → tickets.id
  processing_run_id UUID          NULLABLE FK → ticket_processing_runs.id
  agent_name        TEXT          NOT NULL  -- 'completeness' | 'summarization' | 'response_draft' | 'tagging' | 'sentiment' | 'kb_update_proposal'
  provider          TEXT          NOT NULL  -- 'anthropic' | 'openai' | ...
  model             TEXT          NOT NULL  -- e.g. 'claude-sonnet-4-6'
  prompt_tokens     INTEGER       NULLABLE  -- from response usage metadata
  completion_tokens INTEGER       NULLABLE
  total_tokens      INTEGER       NULLABLE
  estimated_cost_usd NUMERIC(10,6) NULLABLE -- computed from token counts × known rates
  latency_ms        INTEGER       NOT NULL  -- wall-clock from call start to result
  success           BOOLEAN       NOT NULL
  error_code        TEXT          NULLABLE  -- provider error code or 'timeout' | 'consent_blocked' | 'schema_invalid'
  error_message     TEXT          NULLABLE  -- truncated to 500 chars
  used_fallback     BOOLEAN       NOT NULL DEFAULT FALSE  -- true if primary provider was bypassed
  own_api_key       BOOLEAN       NOT NULL DEFAULT FALSE  -- true if tenant's own encrypted key was used (Phase 3)
  created_at        TIMESTAMPTZ   NOT NULL DEFAULT now()
```

**Indexes:** `(tenant_id, created_at)`, `(ticket_id)`, `(agent_name, success, created_at)`

### Gateway changes

Wrap every `generateObject` call in a timing block. Capture token counts from the AI SDK's `usage` field. Write a log row whether the call succeeds or fails. Keep the write non-blocking (fire-and-forget insert; do not let a logging failure propagate).

```typescript
// Proposed gateway internal helper
async function callWithLogging<T>(
  config: AIGatewayConfig & { tenantId: string; ticketId?: string; processingRunId?: string },
  agentName: string,
  fn: () => Promise<{ object: T; usage?: { promptTokens: number; completionTokens: number } }>,
): Promise<T> {
  const start = Date.now()
  try {
    const { object, usage } = await fn()
    void logAiCall({ ...config, agentName, success: true, usage, latencyMs: Date.now() - start })
    return object
  } catch (err) {
    void logAiCall({ ...config, agentName, success: false, error: err, latencyMs: Date.now() - start })
    throw err
  }
}
```

### Cost estimation

**Decision:** Token rates live in a hardcoded `AI_COST_RATES` constant in `src/lib/ai-cost-rates.ts`. This file must include a comment noting it requires manual updates when Anthropic changes pricing. For MVP, only Anthropic rates are needed.

Store `estimated_cost_usd` at write time using these rates; do not recompute retroactively. When a model is not present in `AI_COST_RATES`, write `estimated_cost_usd = NULL` — never silently omit the row or default to zero.

### Admin UI: Usage panel (new `/admin/ai-usage` page)

- Accessible to admins only
- Shows current calendar month by default; date range selector
- Summary row: total calls, success rate, total tokens, estimated cost
- Per-agent breakdown table: calls, P50/P95 latency, success rate, cost
- Error log: last 20 failed calls with agent, error code, error message, timestamp
- No real-time refresh required for MVP; a manual refresh button is sufficient

**Done criteria for Phase 1:**
- [x] `ai_call_logs` table exists with a Drizzle migration
- [x] Every gateway function writes a log row (success and failure)
- [x] Token counts and estimated cost are populated for Anthropic calls
- [x] `AI_COST_RATES` constant exists in `src/lib/ai-cost-rates.ts` with a comment noting manual update responsibility
- [x] Rows where the model is not in `AI_COST_RATES` are written with `estimated_cost_usd = NULL` — not omitted or defaulted to zero
- [x] `/admin/ai-usage` renders summary + per-agent table + error log (accessible to admins only)
- [x] Integration test: mock a successful and a failed LLM call; assert log rows are written with correct fields
- [x] `AIGatewayConfig` renamed to `AIGatewayContext`; all callers updated and `pnpm type-check` passes
- [x] Integration test: GET `/api/admin/ai-usage` returns aggregate data scoped to the requesting tenant

---

## Phase 2 — Fallback Provider / Model

### Goal

When a primary LLM call fails due to a provider error (rate limit, timeout, 5xx), the gateway automatically retries with a configured fallback model before propagating the error. The fallback attempt is logged separately (`used_fallback = true`).

### Tenant config additions

```sql
-- New columns on tenant_configs
ai_fallback_provider  TEXT     NULLABLE  DEFAULT NULL  -- NULL = no fallback configured
ai_fallback_model     TEXT     NULLABLE  DEFAULT NULL
```

These use the same schema shape as the existing `ai_provider` / `ai_model` columns. The fallback provider/model pair is validated on write (same allowlist as primary). Both must be set or both must be null; a partial configuration is rejected.

### Fallback logic

```
1. Try primary provider + model
2. If success → return result, write one log row (used_fallback = false)
3. If failure with retryable error (rate limit, timeout, 5xx):
   a. Write one log row for the primary failure (success = false, used_fallback = false)
   b. If fallback is configured → try fallback provider + model
      - If fallback succeeds → write a second log row (success = true, used_fallback = true), return result
      - If fallback fails → write a second log row (success = false, used_fallback = true), throw original error
   c. If no fallback configured → throw original error (only the primary failure row is written)
4. If failure with non-retryable error (consent blocked, schema invalid, 4xx) → write one log row, throw immediately (no fallback)
```

**Two-row model:** every attempt — primary and fallback — produces its own `ai_call_logs` row. This makes it unambiguous whether a fallback was tried and what its outcome was.

Retryable vs non-retryable classification uses the AI SDK's error type hierarchy (`AISDKError` subtypes, `isRetryable` flags, or HTTP status codes surfaced via `APICallError.status`). Zod schema validation failures are non-retryable.

### Configuration UI

Extend the existing `/admin/config` tenant configuration form to expose "Fallback model" fields, with the same provider/model pair selectors as the primary configuration. A tooltip explains: "If the primary model fails with a provider error, this model is tried once before the ticket moves to degraded state."

**Done criteria for Phase 2:**
- [ ] `ai_fallback_provider` and `ai_fallback_model` added to `tenant_configs`
- [ ] Gateway uses `createLLMClient` factory; per-function `createAnthropic` instantiation is removed
- [ ] Gateway retries with fallback on retryable errors when configured
- [ ] Each attempt (primary and fallback) produces its own `ai_call_logs` row
- [ ] `used_fallback = true` on fallback attempt rows (success or failure)
- [ ] Admin config form exposes fallback fields with validation
- [ ] Integration test: primary returns 429 → fallback is tried → two rows written, second has `used_fallback = true, success = true`
- [ ] Integration test: primary returns 429, fallback also fails → two rows written, error propagated
- [ ] Integration test: primary returns 400 (non-retryable) → one row written, no fallback attempt, error propagated immediately

---

## Phase 3 — Per-Tenant API Keys

### Goal

A tenant can supply their own Anthropic (or other provider) API key. When a tenant key is set, all LLM calls for that tenant use that key instead of the platform key. This enables:

1. **Usage-based billing** — the tenant is charged directly by the provider, not the platform
2. **Key isolation** — a tenant's usage does not count against the platform quota
3. **Provider preference** — a tenant can use a different provider from the platform default

### Storage

API keys must **not** be stored as plaintext in the database. Options:

**Option A — Encrypted column (recommended for MVP)**  
Encrypt with AES-256-GCM using a per-environment encryption key stored in an env var (`AI_KEY_ENCRYPTION_SECRET`). Store the ciphertext in `tenant_configs.ai_api_key_encrypted`. Decrypt at gateway call time. This adds latency of ~0.1ms and requires the encryption key to be present in the environment.

**Option B — Secret manager reference**  
Store only a reference (e.g., `arn:aws:secretsmanager:...`) and fetch at call time. More secure, higher latency, requires cloud setup. Suitable for enterprise deployments but too heavy for MVP.

Go with Option A. Document Option B as the path for enterprise/SOC2 requirements.

### Schema additions

```sql
-- New columns on tenant_configs
ai_api_key_encrypted   TEXT   NULLABLE DEFAULT NULL
ai_api_key_hint        TEXT   NULLABLE DEFAULT NULL  -- last 4 chars of the key, plaintext
```

`ai_api_key_hint` lets admins see which key is set without exposing the full key. Example: "sk-ant-...3f2a".

### Admin UI

Add a "Custom API key" field to `/admin/config`:

- Single password-type input: "Anthropic API key (optional)"
- Helper text: "If set, your tenant's AI calls use this key and are billed directly by Anthropic. Leave blank to use the platform key."
- On save: encrypt and store; display hint only ("Key set — last 4 chars: ...3f2a")
- "Remove key" action: clears the encrypted field, reverts to platform key
- **Never** expose the full key after it is saved, even to admins

### Gateway resolution order

```
1. If tenant has ai_api_key_encrypted → decrypt, use for this call
2. Else → use process.env.ANTHROPIC_API_KEY (platform key)
```

The same resolution applies to fallback calls (Phase 2): if a tenant has their own key, it is used for both primary and fallback.

### Security requirements

- The encryption key (`AI_KEY_ENCRYPTION_SECRET`) must be at least 32 bytes. Because Next.js has no single boot hook in serverless deployments, validation is enforced via a `/api/health` endpoint that checks for the presence of the env var when any tenant has an encrypted key set. The deployment pipeline must call this endpoint and gate traffic cutover on a 200 response.
- Keys are never logged, never included in error messages, never returned to the client.
- Rotation: if `AI_KEY_ENCRYPTION_SECRET` is rotated, all encrypted keys must be re-encrypted. Provide a migration script.
- The `ai_api_key_encrypted` column is excluded from `SELECT *` patterns in DB queries — use explicit column lists.

### Billing visibility

When a tenant is using their own key, the `/admin/ai-usage` page shows a banner: "This tenant uses its own API key. Costs shown are estimates and may differ from your Anthropic bill." Cost estimates are still written to `ai_call_logs` with `own_api_key = true` (the column is defined in Phase 1's schema) so they can be excluded from platform-level cost accounting.

**Done criteria for Phase 3:**
- [ ] `ai_api_key_encrypted` and `ai_api_key_hint` added to `tenant_configs`
- [ ] Encryption/decryption utility with test coverage
- [ ] Gateway uses tenant key when present
- [ ] `own_api_key = true` is set on all `ai_call_logs` rows written while using a tenant's own key
- [ ] Admin config UI: set key (password input), display hint, remove key
- [ ] Full key is never returned to client or logged
- [ ] `/api/health` returns non-200 if `AI_KEY_ENCRYPTION_SECRET` is absent and any tenant has an encrypted key; deployment pipeline gates on this check
- [ ] Integration test: tenant with own key → gateway uses decrypted key
- [ ] Integration test: tenant without key → gateway uses platform env key
- [ ] Integration test: `ai_api_key_encrypted` column is absent from any `SELECT *` or broad-column-list queries (no accidental exposure)
- [ ] Key rotation migration script exists, or is explicitly deferred to backlog with a tracking reference
- [ ] Security review: confirm key is absent from logs, error messages, API responses

---

## Cross-Cutting Design Decisions

### Provider abstraction

The current gateway hardcodes `createAnthropic(...)` in every function. Before Phase 2 ships, refactor the gateway to a single `createLLMClient(provider, apiKey)` factory that returns an AI SDK-compatible model object. This eliminates the repetition and is a prerequisite for supporting a second provider (e.g., OpenAI as a fallback).

```typescript
function createLLMClient(provider: string, model: string, apiKey: string) {
  switch (provider) {
    case 'anthropic': return createAnthropic({ apiKey })(model)
    case 'openai': return createOpenAI({ apiKey })(model)
    default: throw new Error(`Unknown provider: ${provider}`)
  }
}
```

The provider allowlist lives in `tenant_configs` validation — only values present in `createLLMClient`'s switch statement are accepted.

### Gateway function signatures

All three phases should be rolled into the gateway with backward-compatible additions. The existing `AIGatewayConfig` interface (exported from `src/lib/ai-gateway.ts`) is **renamed** to `AIGatewayContext` and extended with the new fields. All callers must be updated to use the new name. The expanded shape:

```typescript
export interface AIGatewayContext {
  tenantId: string
  ticketId?: string
  processingRunId?: string
  provider: string
  model: string
  temperature: number
  dataProcessingConsent: boolean
  // Phase 2:
  fallbackProvider?: string | null
  fallbackModel?: string | null
  // Phase 3:
  apiKeyEncrypted?: string | null
}
```

Callers (Inngest agents) already pass `config: AIGatewayConfig` sourced from `tenant_configs`. Renaming `AIGatewayConfig` → `AIGatewayContext` and extending `tenant-config-loader.ts` to populate the new fields are the only callsite changes required. Phase 1 done criteria should include confirming all callers compile after the rename.

### What is NOT in scope

- Multi-provider load balancing (use fallback only, not round-robin)
- Cost budgets or hard spend limits per tenant (track first, limit later)
- Key rotation automation
- Storing prompt/response content in `ai_call_logs` (privacy risk; keep logs lightweight)
- OpenAI or other providers in Phase 1 or 2 (Anthropic only; the abstraction enables others)

---

## Decisions (formerly Open Questions)

1. **Cost rate table:** Hardcoded `AI_COST_RATES` constant in `src/lib/ai-cost-rates.ts`. Comment in that file must note manual update responsibility. See Phase 1 cost estimation section.

2. **`ai_call_logs` retention:** 90 days for cost/audit purposes. No automatic pruning for MVP. Estimated volume: ~6 agents × peak ticket volume × 90 days; add a volume estimate to the pre-mortem checklist before launch. Future retention policy is a backlog item.

3. **`/admin/ai-usage` access level:** Admins only. Cost data is financially sensitive; experts do not need access for Phase 1 or Phase 2.

4. **Fallback on schema validation failures:** No. Zod validation failures indicate a model compatibility issue that a different provider is unlikely to fix. Fallback is attempted only on provider-level errors (network, rate limit, timeout, 5xx), not on application-layer validation failures.