# Twitter/X Ingestion

Goal: when a user sends a Twitter/X status URL, NanoClaw should derive useful
text from the post without requiring paid X API access. The ingestion result is
added to the same `linkIngestions` array used by Instagram ingestion, so the
agent sees the original message plus resolved post text, related post text, and
available transcripts.

## No-API Resolver

The module uses a best-effort resolver ladder:

1. Cache lookup by canonical status URL.
2. `yt-dlp --dump-json` for public or browser-cookie-backed Twitter/X metadata.
3. Post text from metadata fields such as description/title.
4. Quoted or reposted status URLs discovered in metadata text, resolved once and
   rendered as `relatedPosts`.
5. Existing subtitles/captions if the extractor exposes them.
6. YouTube transcript lookup for YouTube URLs in post text.
7. Optional YouTube search (`ytsearch1:` through `yt-dlp`) for likely mirrored
   videos when `TWITTER_YOUTUBE_LOOKUP=true`.
8. Embedded video download, low-bitrate audio extraction with `ffmpeg`, and
   OpenAI audio transcription when no subtitle source is available.

This does not use the X API. Reliability depends on what `yt-dlp` can access.
For protected, age-gated, region-gated, or login-walled posts, use local browser
cookies from an account that is allowed to view the content.

## Content Contract

Successful entries look like:

```json
{
  "source": "twitter",
  "url": "https://x.com/alice/status/111",
  "canonicalUrl": "https://x.com/alice/status/111",
  "status": "ready",
  "kind": "video",
  "author": "Alice",
  "caption": "Post text...",
  "transcript": "Transcript from subtitles, YouTube, or transcription...",
  "youtubeUrl": "https://www.youtube.com/watch?v=...",
  "youtubeTranscript": "Transcript from YouTube subtitles...",
  "relatedPosts": [
    {
      "url": "https://x.com/bob/status/222",
      "canonicalUrl": "https://x.com/bob/status/222",
      "author": "Bob",
      "caption": "Quoted post text..."
    }
  ],
  "cachedAt": "2026-06-10T19:00:00.000Z"
}
```

On failure, routing continues and the agent still sees a failed artifact:

```json
{
  "source": "twitter",
  "url": "https://x.com/alice/status/111",
  "canonicalUrl": "https://x.com/alice/status/111",
  "status": "failed",
  "error": "metadata_resolve_failed"
}
```

## Runtime Dependencies

- `yt-dlp` on PATH, or `TWITTER_YTDLP_BIN=/path/to/yt-dlp`
- `ffmpeg` on PATH, or `TWITTER_FFMPEG_BIN=/path/to/ffmpeg`
- `OPENAI_API_KEY` only when embedded video transcription is needed and no
  subtitle/YouTube transcript source is available

Optional settings:

- `OPENAI_TRANSCRIBE_MODEL`, default `gpt-4o-mini-transcribe`
- `TWITTER_YTDLP_COOKIES_FROM_BROWSER`, e.g. `safari`, `chrome`, or `brave`
- `TWITTER_COOKIES_FILE`, passed to `yt-dlp --cookies`
- `TWITTER_INGESTION_MAX_SECONDS`, default `180`
- `TWITTER_INGESTION_MAX_BYTES`, default `78643200`
- `TWITTER_INGESTION_TIMEOUT_MS`, default `90000`
- `TWITTER_INGESTION_FAILURE_CACHE_MS`, default `3600000`
- `TWITTER_INGESTION_RELATED_DEPTH`, default `1`, max `3`
- `TWITTER_YOUTUBE_LOOKUP=true` to search YouTube for likely mirrored videos

Install the external tools with your normal package manager. On macOS with
Homebrew:

```sh
brew install yt-dlp ffmpeg
```

Successful and failed results are cached under `data/twitter-ingestion/cache`.
Raw downloaded media is stored only in a temporary work directory and removed
after processing.