# Ingestion Pipeline — Cost & Quality Optimization Patterns

**Purpose:** Reference doc for a coding agent implementing improvements to Nano's content ingestion pipeline. Covers established patterns for query-focused extraction, chapter-based chunking, hierarchical summarization, and tiered model usage.

**Current pipeline location:** `/workspace/agent/` (ingest-queue.md, scheduled overnight task)
**Wiki output:** `/workspace/extra/second-brain/wiki/`

---

## The core idea

James currently shares content with a context note (e.g., *"less about tools, more about process — agent hiring tips are the most interesting part"*). Today that note is read by Sonnet at summary time, after full transcript processing. The optimization: use the note earlier in the pipeline to:
1. **Skip irrelevant chunks** entirely (don't process sponsor segments, intros, tangential chapters)
2. **Weight summaries** toward James's stated interest without discarding other content
3. **Use cheaper models** for bulk work, Sonnet only for final synthesis

---

## Pattern 1: Query-Focused Summarization (QFS)

**What it is:** The user's context note becomes a *query* that guides every summarization step — not just final synthesis.

**How to apply:**

```python
CHAPTER_SUMMARY_PROMPT = """
You are summarizing a chapter from a podcast for a knowledge wiki.

User's focus: "{user_query}"

Chapter title: "{chapter_title}"
Transcript: "{transcript_chunk}"

Write a concise summary. Prioritize content relevant to the user's focus.
If this chapter has no relevant content, output: SKIP
"""
```

Key behaviors:
- Output `SKIP` for irrelevant chunks → they're dropped before reaching Sonnet
- Remaining chunks are shorter because only relevant content is extracted
- The final synthesis receives pre-filtered, pre-focused material

**Cost impact:** If 40% of chapters are marked SKIP, you save 40% of chunk-processing tokens.

**Reference:** [Query-Focused Summarization guide](https://www.shadecoder.com/topics/query-focused-summarization-a-comprehensive-guide-for-2025)

---

## Pattern 2: Chapter-Based Chunking (Structural Chunking)

**What it is:** Use chapter markers from YouTube/podcast metadata as chunk boundaries instead of fixed-token or semantic splits.

**Why chapters beat other approaches for this use case:**

| Approach | Cost | Reliability | Notes |
|----------|------|-------------|-------|
| **Chapter-based** | Lowest (no embeddings) | High when chapters exist | Creator intent preserved |
| **Fixed-token** | Low | Medium | Splits mid-sentence, mid-concept |
| **Semantic** | Medium (embedding calls) | High | Best when no chapters exist |

**Implementation:**

```python
def chunk_by_chapters(transcript: str, chapters: list[dict]) -> list[dict]:
    """
    chapters: [{"title": "Intro", "start_seconds": 0}, {"title": "Main Topic", "start_seconds": 120}, ...]
    Returns list of {"title": str, "text": str}
    """
    chunks = []
    for i, chapter in enumerate(chapters):
        start = chapter["start_seconds"]
        end = chapters[i+1]["start_seconds"] if i+1 < len(chapters) else None
        text = extract_transcript_segment(transcript, start, end)
        chunks.append({"title": chapter["title"], "text": text})
    return chunks
```

**Sponsor detection:** Add a pre-filter before QFS — chapters with titles containing "sponsor", "ad", "promo" or known sponsor names (Granola, Wispr Flow, etc.) are skipped outright, no LLM call needed.

```python
SPONSOR_KEYWORDS = ["sponsor", "ad read", "brought to you", "discount", "promo code"]
def is_sponsor_chapter(title: str, text: str) -> bool:
    return any(kw in (title + text).lower() for kw in SPONSOR_KEYWORDS)
```

---

## Pattern 3: Map-Reduce Summarization

**What it is:** The standard LangChain/LlamaIndex pattern for long documents. Two stages:
- **Map:** Summarize each chapter independently (parallelizable, cheap model)
- **Reduce:** Combine chapter summaries into a final wiki-ready entry (Sonnet)

**Three chain types — choose based on content:**

| Chain | When to use | Cost | Quality |
|-------|------------|------|---------|
| **Map-Reduce** | Most content; when chapters are independent | Low | Good |
| **Refine** | Narrative content where earlier context matters for later chapters | Medium | High |
| **Stuff** | Short content (<8K tokens total) | Low | Highest |

**Map-Reduce implementation (pseudocode):**

```python
import asyncio

async def map_phase(chunks: list[dict], user_query: str, cheap_model) -> list[str]:
    """Summarize all chapters in parallel using cheap model."""
    tasks = [
        summarize_chunk(chunk, user_query, cheap_model)
        for chunk in chunks
        if not is_sponsor_chapter(chunk["title"], chunk["text"])
    ]
    summaries = await asyncio.gather(*tasks)
    return [s for s in summaries if s != "SKIP"]

def reduce_phase(summaries: list[str], user_query: str, wiki_context: dict, sonnet_model) -> str:
    """Synthesize chapter summaries into a wiki entry using Sonnet."""
    combined = "\n\n".join(f"## {i+1}\n{s}" for i, s in enumerate(summaries))
    return sonnet_model.complete(WIKI_SYNTHESIS_PROMPT.format(
        summaries=combined,
        user_query=user_query,
        source_title=wiki_context["title"],
        existing_wiki_pages=wiki_context["related_pages"]
    ))
```

**LangChain reference:**
```python
from langchain.chains.summarize import load_summarize_chain
chain = load_summarize_chain(llm, chain_type="map_reduce")
```

---

## Pattern 4: Tiered Model Architecture

**The rule:** Use the cheapest model that can do the job at each stage.

**Recommended tiers for this pipeline:**

| Stage | Task | Model | Why |
|-------|------|-------|-----|
| Sponsor detection | Keyword check | No model — regex | Always deterministic |
| Chapter relevance filter | Is this chapter relevant to the query? | Haiku | Simple yes/no judgment |
| Chapter summarization | Summarize one chapter | Haiku | Short input, straightforward task |
| Section synthesis | Combine 3-5 related chapters | GPT-4o-mini | Cheaper output pricing |
| Final wiki entry | Full synthesis, cross-references | Sonnet | Only where quality matters |
| Transcript extraction (audio→text) | Transcription | gpt-4o-mini-transcribe | $0.003/min |

**Cost comparison per 1-hour podcast:**

| Pipeline | Cost |
|---------|------|
| All Sonnet (current) | ~$1.50 |
| Haiku map + Sonnet reduce | ~$0.35 |
| Haiku map + GPT-4o-mini reduce | ~$0.15 |
| With sponsor/irrelevant skipping (40% skip rate) | ~$0.09 |

**Model IDs (May 2026):**
- Haiku: `claude-haiku-4-5-20251001`
- Sonnet: `claude-sonnet-4-6`
- GPT-4o-mini: `gpt-4o-mini` (via OpenAI API or OneCLI gateway)
- Whisper: `gpt-4o-mini-transcribe`

---

## Pattern 5: Prompt Caching

**What it is:** Anthropic's feature for caching prompt prefixes. A cached 50K-token transcript costs 0.1x normal on re-reads (90% discount).

**When it helps:** When the same content is summarized multiple times — e.g., a podcast is ingested once, then James asks a follow-up question later in the day.

**How to apply:**

```python
# Mark the transcript for caching when first sent
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": full_transcript,
                "cache_control": {"type": "ephemeral"}  # 5-min TTL
            },
            {
                "type": "text",
                "text": f"Summarize with focus on: {user_query}"
            }
        ]
    }
]
```

**Payoff:**
- 5-min cache: profitable after 1 re-read (1.25x write vs 0.1x read × 10 reads)
- 1-hour cache: profitable after 2 re-reads

**Reference:** [Anthropic Prompt Caching docs](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching)

---

## Pattern 6: RAPTOR (optional, for high-query volume)

**What it is:** RAPTOR (ICLR 2024) builds a hierarchical summary tree. Unlike flat map-reduce, it clusters related chunks semantically, then summarizes clusters — so the final tree captures both local and global structure.

**When to use it:** Only if the wiki gets queried heavily (not just written to). For James's current use case (write-heavy, read-light), flat map-reduce is sufficient. RAPTOR becomes worth it when James starts querying the wiki conversationally ("what have I learned about X across all sources?").

**Implementation:** [github.com/parthsarthi03/raptor](https://github.com/parthsarthi03/raptor) — production-ready, MIT license.

---

## Recommended implementation order for coding agent

**Phase 1 (biggest bang, lowest effort):**
1. Add sponsor/irrelevant chapter detection (regex, no model call)
2. Parse YouTube chapter metadata and use as chunk boundaries
3. Add `user_query` parameter that flows into chapter summary prompts (QFS)
4. Switch chapter summaries from Sonnet → Haiku

**Phase 2 (structural improvement):**
5. Implement async parallel map phase (all chapters summarized simultaneously)
6. Add `SKIP` output handling — drop SKIP chunks before reduce phase
7. Add prompt caching for transcripts during reduce phase

**Phase 3 (optional, if cost still matters):**
8. Evaluate switching reduce phase from Sonnet → GPT-4o-mini
9. Add semantic chunking fallback for content with no chapter metadata
10. Evaluate RAPTOR if wiki query volume grows

---

## Prompts for coding agent

### Phase 1 prompt
```
Implement the following improvements to the ingestion pipeline in /workspace/agent/:

1. Extract YouTube chapter timestamps from video descriptions (parse "MM:SS Title" lines)
2. Split transcripts at chapter boundaries instead of fixed-size chunks
3. Add a `user_query` parameter to the ingestion task (comes from James's context note in ingest-queue.md)
4. Add sponsor chapter detection: skip chapters where title or first 200 chars match SPONSOR_KEYWORDS list
5. In the chapter summary prompt, include the user_query: "Summarize this chapter. Focus on: {user_query}. If not relevant, output SKIP."
6. Change chapter-level model from claude-sonnet to claude-haiku-4-5-20251001
7. Keep the final synthesis (reduce) step on claude-sonnet-4-6

Reference: /workspace/agent/research/ingestion-pipeline-patterns.md
```

### Phase 2 prompt
```
Extend the ingestion pipeline with:

1. Async parallel execution of all chapter summaries (use asyncio.gather or equivalent)
2. Filter out SKIP responses before the reduce phase
3. Add prompt caching to the reduce step: mark the concatenated chapter summaries with cache_control: ephemeral
4. Update the ingest-queue.md format to include a user_query field
5. Update the overnight scheduled task to pass user_query from ingest-queue entries to the pipeline

Reference: /workspace/agent/research/ingestion-pipeline-patterns.md
```

---

## Key references
- RAPTOR paper: https://arxiv.org/abs/2401.18059
- RAPTOR GitHub: https://github.com/parthsarthi03/raptor
- LangChain summarization: https://python.langchain.com/docs/how_to/summarize_refine/
- LlamaIndex response synthesis: https://developers.llamaindex.ai/python/examples/low_level/response_synthesis/
- Anthropic prompt caching: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
- GraphRAG / QFS: https://arxiv.org/html/2404.16130v2