# Live Validation Punch List

Phase 1d is implemented and verified offline. Work through this list to validate the
real API, real downloads, real audio transcription, and phone/voice path before starting
Phase 2.

## 0. Local setup

- [ ] Confirm you are at repo root:
  ```bash
  pwd
  ```
  Expected suffix: `MentorCore`.

- [ ] Install/sync Python workspace dependencies:
  ```bash
  uv sync
  ```

- [ ] Install web dependencies:
  ```bash
  cd web && npm install
  ```

- [ ] Export Anthropic API key in each shell that will run compiler/runtime commands:
  ```bash
  export ANTHROPIC_API_KEY="..."
  ```

- [ ] Run the offline gate:
  ```bash
  ./scripts/check.sh
  ```

## 1. Real Marcus compile baseline

- [ ] Compile Marcus with real downloads, embeddings, style extraction, and eval:
  ```bash
  uv run mentor compile marcus-aurelius
  ```

- [ ] Run standalone eval:
  ```bash
  ./scripts/eval-persona.sh marcus-aurelius
  ```

- [ ] Record the latest eval line:
  ```bash
  tail -n 1 personas/marcus-aurelius/current/eval/history.jsonl
  ```

- [ ] Confirm thresholds:
  - groundedness >= 0.7
  - style_match >= 0.6
  - misattribution = 0

## 2. Real Ayrton/audio source review

- [ ] Regenerate candidate sources if you want fresh discovery:
  ```bash
  uv run mentor compile "Ayrton Senna" --propose
  ```

- [ ] Review `sources/ayrton-senna.yaml`.

- [ ] Approve only sources you want ingested by setting:
  ```yaml
  approved: true
  ```

- [ ] Prefer one explicit audio/interview source. Valid patterns:
  ```yaml
  fetch_type: rss
  rss_url: https://...
  approved: true
  ```
  or:
  ```yaml
  fetch_type: youtube
  url: https://...
  approved: true
  ```

- [ ] If RSS chooses the wrong episode, add an explicit episode URL:
  ```yaml
  episode_url: https://...mp3
  ```

- [ ] Build approved Ayrton sources:
  ```bash
  uv run mentor compile ayrton-senna --build
  ```

- [ ] Confirm transcript/corpus/eval artifacts exist:
  ```bash
  find personas/ayrton-senna/current -maxdepth 2 -type f | sort
  tail -n 1 personas/ayrton-senna/current/eval/history.jsonl
  ```

## 3. Try text chat locally

- [ ] Start runtime:
  ```bash
  uv run uvicorn runtime.api.app:app --reload
  ```

- [ ] In another shell, start web client:
  ```bash
  cd web && npm run dev
  ```

- [ ] Open the local web app shown by Next.js and try:
  - Marcus text chat
  - Ayrton text chat

- [ ] Try terminal chat:
  ```bash
  uv run mentor chat marcus-aurelius
  uv run mentor chat ayrton-senna
  ```

## 4. Voice validation

- [ ] Run speech tests:
  ```bash
  uv run pytest tests/speech -q
  ```

- [ ] Run latency benchmark:
  ```bash
  uv run python scripts/latency-bench.py --persona marcus-aurelius --runs 10
  ```

- [ ] Check first-audio p50 target:
  - LAN target: < 2.0s perceived latency.

- [ ] Deploy local services when ready:
  ```bash
  ANTHROPIC_API_KEY=... ./scripts/deploy.sh
  ```

- [ ] Phone/PWA test:
  - Open the Tailscale-served web app on phone.
  - Install/open PWA.
  - Test push-to-talk.
  - Test Bluetooth headset mic/speaker.
  - Do one walking test with Marcus.

## 5. Red-team and live model gates

- [ ] Run live red-team tests:
  ```bash
  uv run pytest tests/live -m live
  ```

- [ ] Confirm no fabricated verbatim quotes.

- [ ] If a prompt/eval regression appears, add a failing test before changing prompt or scoring code.

## 6. Ready for Phase 2 when all are true

- [ ] `./scripts/check.sh` passes.
- [ ] Marcus real compile passes thresholds.
- [ ] Ayrton real audio compile produces transcript, corpus, index, style, and eval.
- [ ] Text chat works locally.
- [ ] Voice path works locally.
- [ ] Latency benchmark is under target.
- [ ] Phone/PWA voice test works over Tailscale.
- [ ] Live red-team misattribution gate passes.

## Phase 2 starting backlog

- [ ] Detail Phase 2 build plan before coding.
- [ ] Windows VR PC push-to-talk client, wheel-button mapping.
- [ ] Dual-persona session model: driver coach primary, crew chief secondary.
- [ ] `runtime/telemetry/` iRacing ingestion via `pyirsdk`.
- [ ] Debrief workflow correlating driver feel with telemetry.
- [ ] Setup-experiment memory.
- [ ] Personal-tier cloned voice flow, gated by `voice_policy: cloned-personal`.