In the StageHand project at `/workspace/extra/projects/stagehand`, I need to understand the current state of the Compass demo adapter and what "Expand Compass gallery capture further / fail closed below min photo count for demos" means concretely.

Please read:
1. Any file matching `*compass*` or `*demo*` in `cli/`, `core/`, or `workflows/`
2. The relevant test files for the Compass adapter
3. The CLAUDE.md section about the `--demo` flag

I need to know:
- What does the Compass adapter currently do? (what photos does it capture, what's the flow)
- What does "expand gallery capture" mean — what's missing?
- What does "fail closed below min photo count" mean — is there currently a min-photo check, or not?
- Are there existing tests I should know about?

Report concisely in under 300 words.


I'll systematically explore the StageHand project to understand the Compass demo adapter state.


No files found


/workspace/extra/projects/stagehand/core/listings/providers/compass.py
/workspace/extra/projects/stagehand/core/listings/providers/__pycache__/compass.cpython-312.pyc


Bash

find /workspace/extra/projects/stagehand -name "*compass*" -o -name "*demo*" | grep -E "\.(ts|js|py)$"


/workspace/extra/projects/stagehand/core/listings/providers/compass.py
/workspace/extra/projects/stagehand/tests/test_compass_provider.py


Now let me read the Compass provider and test file:


1	"""Compass allowlisted demo adapter.
2
3	Demo-grade intake for Richard Fowler compass.com listing pages.
4
5	This is intentionally *not* a production rights path. Production still needs
6	brokerage / photographer / MLS authorization (or a licensed feed). The adapter
7	records demo provenance so human review never confuses pilot media with a
8	shipped rights-cleared source.
9	"""
10
11	from __future__ import annotations
12
13	import hashlib
14	import json
15	import re
16	from dataclasses import dataclass
17	from datetime import UTC, datetime
18	from pathlib import Path
19	from typing import Any, Callable, Mapping, Sequence
20	from urllib.error import HTTPError, URLError
21	from urllib.parse import urlsplit
22	from urllib.request import Request, urlopen
23
24	from core.listings.manifest import ListingManifestError, normalize_listing_url
25	from core.listings.models import ListingPhoto, ListingSnapshot
26
27	COMPASS_HOSTS = frozenset({"compass.com", "www.compass.com"})
28	DEFAULT_USER_AGENT = (
29	    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
30	    "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
31	)
32	_INITIAL_DATA_RE = re.compile(r"window\.__INITIAL_DATA__\s*=\s*")
33	_MEDIA_HASH_RE = re.compile(r"/m/([0-9a-f]{40,})/", re.I)
34	_COMPASS_MEDIA_RE = re.compile(
35	    r"https://www\.compass\.com/m/(?:[0-9]+/)?[0-9a-fA-F-]+/(?:origin|640x480)\.(?:jpg|jpeg|webp)",
36	    re.I,
37	)
38
39
40	class CompassListingError(ListingManifestError):
41	    """Raised when a Compass listing cannot be ingested for the demo workflow."""
42
43
44	@dataclass(frozen=True, slots=True)
45	class CompassListingParse:
46	    source_url: str
47	    address: str
48	    city: str
49	    state: str
50	    postal_code: str
51	    price: int | None
52	    beds: float | None
53	    baths: float | None
54	    square_feet: int | None
55	    description: str
56	    photo_urls: tuple[str, ...]
57	    attribution: str
58	    agent_name: str = ""
59
60
61	HttpGet = Callable[[str], bytes]
62
63
64	def is_compass_listing_url(listing_url: str) -> bool:
65	    try:
66	        host = (urlsplit(normalize_listing_url(listing_url)).hostname or "").lower()
67	    except ListingManifestError:
68	        return False
69	    return host in COMPASS_HOSTS
70
71
72	def parse_compass_listing_html(html: str, *, listing_url: str) -> CompassListingParse:
73	    if not html or not html.strip():
74	        raise CompassListingError("Compass listing HTML was empty")
75
76	    source_url = normalize_listing_url(listing_url)
77	    initial = _extract_initial_data(html)
78	    listing = _listing_blob(initial) if initial is not None else None
79	    ld = _extract_ld_listing(html)
80
81	    address = _first_str(
82	        _dig(listing, "location", "streetAddress"),
83	        _dig(ld, "address", "streetAddress"),
84	    )
85	    city = _first_str(
86	        _dig(listing, "location", "city"),
87	        _dig(ld, "address", "addressLocality"),
88	    )
89	    state = _first_str(
90	        _dig(listing, "location", "state"),
91	        _dig(ld, "address", "addressRegion"),
92	    )
93	    postal_code = _first_str(
94	        _dig(listing, "location", "zipCode"),
95	        _dig(listing, "location", "postalCode"),
96	        _dig(ld, "address", "postalCode"),
97	    )
98	    description = _first_str(
99	        _dig(listing, "description"),
100	        _dig(ld, "description"),
101	    ).strip()
102	    price = _coerce_int(
103	        _dig(listing, "price", "listed")
104	        or _dig(listing, "price", "amount")
105	        or _dig(listing, "price")
106	        or _dig(ld, "offers", "price")
107	    )
108	    beds = _coerce_float(
109	        _dig(listing, "size", "bedrooms")
110	        or _dig(listing, "size", "beds")
111	        or _dig(ld, "numberOfRooms")
112	    )
113	    baths = _coerce_float(
114	        _dig(listing, "size", "bathrooms")
115	        or _dig(listing, "size", "totalBathrooms")
116	        or _dig(listing, "size", "baths")
117	    )
118	    square_feet = _coerce_int(
119	        _dig(listing, "size", "squareFeet")
120	        or _dig(listing, "size", "sqft")
121	        or _floor_size(ld)
122	    )
123	    agent_name = _first_str(
124	        _dig(listing, "fullContacts", 0, "name"),
125	        _dig(ld, "offers", "offeredBy", "name"),
126	    )
127	    photo_urls = _collect_photo_urls(html, listing, ld)
128	    if not photo_urls:
129	        raise CompassListingError("Compass listing page exposed no property photos")
130	    if not any((price is not None, beds is not None, baths is not None, square_feet is not None, description)):
131	        raise CompassListingError("Compass listing page exposed no usable listing facts")
132
133	    attribution = (
134	        "Demo intake from compass.com public listing page for pilot review only. "
135	        "Not a production rights-cleared source — obtain broker/photographer/MLS authorization before shipping."
136	    )
137	    if agent_name:
138	        attribution = f"{attribution} Listed agent on page: {agent_name}."
139
140	    return CompassListingParse(
141	        source_url=source_url,
142	        address=address,
143	        city=city,
144	        state=state,
145	        postal_code=postal_code,
146	        price=price,
147	        beds=beds,
148	        baths=baths,
149	        square_feet=square_feet,
150	        description=description,
151	        photo_urls=photo_urls,
152	        attribution=attribution,
153	        agent_name=agent_name,
154	    )
155
156
157	def materialize_compass_listing(
158	    parsed: CompassListingParse,
159	    *,
160	    media_root: str | Path,
161	    http_get: HttpGet | None = None,
162	    max_photos: int = 20,
163	) -> ListingSnapshot:
164	    """Download photo URLs into media_root and return a ListingSnapshot."""
165	    root = Path(media_root)
166	    photos_dir = root / "photos"
167	    photos_dir.mkdir(parents=True, exist_ok=True)
168	    getter = http_get or _default_http_get
169	    photos: list[ListingPhoto] = []
170	    for index, photo_url in enumerate(parsed.photo_urls[: max(1, max_photos)]):
171	        suffix = _extension_for_url(photo_url)
172	        relative = Path("photos") / f"photo_{index + 1:02d}{suffix}"
173	        dest = root / relative
174	        try:
175	            payload = getter(photo_url)
176	        except CompassListingError:
177	            raise
178	        except Exception as exc:  # pragma: no cover - network seams cover this with mocked get
179	            raise CompassListingError(f"Failed to download Compass photo {index + 1}: {exc}") from exc
180	        if not payload:
181	            raise CompassListingError(f"Compass photo {index + 1} was empty")
182	        dest.write_bytes(payload)
183	        digest = hashlib.sha256(payload).hexdigest()
184	        photos.append(
185	            ListingPhoto(
186	                path=dest.resolve(),
187	                relative_path=relative.as_posix(),
188	                sha256=digest,
189	                size_bytes=len(payload),
190	                label=f"Photo {index + 1}",
191	                focal_point=(0.5, 0.5),
192	            )
193	        )
194
195	    if not photos:
196	        raise CompassListingError("No Compass photos were downloaded")
197
198	    return ListingSnapshot(
199	        source_url=parsed.source_url,
200	        source_provider="compass_public_demo",
201	        rights_confirmed=True,
202	        attribution=parsed.attribution,
203	        address=parsed.address,
204	        city=parsed.city,
205	        state=parsed.state,
206	        postal_code=parsed.postal_code,
207	        price=parsed.price,
208	        beds=parsed.beds,
209	        baths=parsed.baths,
210	        square_feet=parsed.square_feet,
211	        description=parsed.description,
212	        photos=tuple(photos),
213	        cover_photo_index=0,
214	        created_at=datetime.now(UTC).isoformat(),
215	    )
216
217
218	def write_manifest_from_snapshot(snapshot: ListingSnapshot, media_root: str | Path) -> Path:
219	    """Write a version-1 operator manifest next to downloaded photos for re-runs."""
220	    root = Path(media_root)
221	    root.mkdir(parents=True, exist_ok=True)
222	    path = root / "listing.json"
223	    payload = snapshot.to_dict()
224	    # Manifest format uses path/label/focal_point only on photos.
225	    payload["photos"] = [
226	        {
227	            "path": photo.relative_path,
228	            "label": photo.label,
229	            "focal_point": list(photo.focal_point),
230	        }
231	        for photo in snapshot.photos
232	    ]
233	    path.write_text(json.dumps(payload, indent=2), encoding="utf-8")
234	    return path
235
236
237	def fetch_compass_listing(
238	    listing_url: str,
239	    *,
240	    media_root: str | Path,
241	    http_get: HttpGet | None = None,
242	    max_photos: int = 20,
243	) -> ListingSnapshot:
244	    if not is_compass_listing_url(listing_url):
245	        raise CompassListingError("Not a Compass listing URL")
246	    getter = http_get or _default_http_get
247	    try:
248	        html = getter(normalize_listing_url(listing_url)).decode("utf-8", "replace")
249	    except CompassListingError:
250	        raise
251	    except Exception as exc:
252	        raise CompassListingError(f"Failed to fetch Compass listing page: {exc}") from exc
253	    parsed = parse_compass_listing_html(html, listing_url=listing_url)
254	    snapshot = materialize_compass_listing(
255	        parsed,
256	        media_root=media_root,
257	        http_get=getter,
258	        max_photos=max_photos,
259	    )
260	    write_manifest_from_snapshot(snapshot, media_root)
261	    return snapshot
262
263
264	def _default_http_get(url: str) -> bytes:
265	    request = Request(
266	        url,
267	        headers={
268	            "User-Agent": DEFAULT_USER_AGENT,
269	            "Accept": "text/html,application/xhtml+xml,application/json,image/avif,image/webp,*/*;q=0.8",
270	            "Accept-Language": "en-US,en;q=0.9",
271	        },
272	    )
273	    try:
274	        with urlopen(request, timeout=30) as response:  # noqa: S310 - operator-requested live URL fetch
275	            return response.read()
276	    except HTTPError as exc:
277	        raise CompassListingError(f"HTTP {exc.code} fetching {url}") from exc
278	    except URLError as exc:
279	        raise CompassListingError(f"Network error fetching {url}: {exc.reason}") from exc
280
281
282	def _extract_initial_data(html: str) -> dict[str, Any] | None:
283	    match = _INITIAL_DATA_RE.search(html)
284	    if not match:
285	        return None
286	    start = match.end()
287	    while start < len(html) and html[start].isspace():
288	        start += 1
289	    if start >= len(html) or html[start] != "{":
290	        return None
291	    depth = 0
292	    in_string = False
293	    escape = False
294	    for index in range(start, len(html)):
295	        ch = html[index]
296	        if in_string:
297	            if escape:
298	                escape = False
299	            elif ch == "\\":
300	                escape = True
301	            elif ch == '"':
302	                in_string = False
303	            continue
304	        if ch == '"':
305	            in_string = True
306	        elif ch == "{":
307	            depth += 1
308	        elif ch == "}":
309	            depth -= 1
310	            if depth == 0:
311	                raw = html[start : index + 1]
312	                try:
313	                    data = json.loads(raw)
314	                except json.JSONDecodeError as exc:
315	                    raise CompassListingError(f"Compass __INITIAL_DATA__ was invalid JSON: {exc}") from exc
316	                if not isinstance(data, dict):
317	                    raise CompassListingError("Compass __INITIAL_DATA__ was not an object")
318	                return data
319	    return None
320
321
322	def _listing_blob(initial: Mapping[str, Any]) -> Mapping[str, Any] | None:
323	    listing = _dig(initial, "props", "listingRelation", "listing")
324	    return listing if isinstance(listing, Mapping) else None
325
326
327	def _extract_ld_listing(html: str) -> dict[str, Any]:
328	    blocks = re.findall(
329	        r'<script[^>]+type=["\']application/ld\+json["\'][^>]*>(.*?)</script>',
330	        html,
331	        flags=re.I | re.S,
332	    )
333	    for block in blocks:
334	        try:
335	            data = json.loads(block.strip())
336	        except json.JSONDecodeError:
337	            continue
338	        found = _find_ld_listing(data)
339	        if found:
340	            return found
341	    return {}
342
343
344	def _find_ld_listing(node: Any) -> dict[str, Any] | None:
345	    if isinstance(node, dict):
346	        types = node.get("@type")
347	        type_list = types if isinstance(types, list) else [types]
348	        lowered = {str(value).lower() for value in type_list if value}
349	        if lowered & {"realestatelisting", "singlefamilyresidence", "product", "house", "apartment"}:
350	            return node
351	        if "@graph" in node:
352	            found = _find_ld_listing(node["@graph"])
353	            if found:
354	                return found
355	        for value in node.values():
356	            found = _find_ld_listing(value)
357	            if found:
358	                return found
359	    elif isinstance(node, list):
360	        for item in node:
361	            found = _find_ld_listing(item)
362	            if found:
363	                return found
364	    return None
365
366
367	def _collect_photo_urls(
368	    html: str,
369	    listing: Mapping[str, Any] | None,
370	    ld: Mapping[str, Any],
371	) -> tuple[str, ...]:
372	    urls: list[str] = []
373	    if listing is not None:
374	        media = listing.get("media")
375	        if isinstance(media, Mapping):
376	            for key in ("originalUrl", "thumbnailUrl", "url"):
377	                value = media.get(key)
378	                if isinstance(value, str) and value.startswith("http"):
379	                    urls.append(value)
380	        elif isinstance(media, list):
381	            for item in media:
382	                if not isinstance(item, Mapping):
383	                    continue
384	                for key in ("originalUrl", "url", "thumbnailUrl"):
385	                    value = item.get(key)
386	                    if isinstance(value, str) and value.startswith("http"):
387	                        urls.append(value)
388	                        break
389	    image = ld.get("image")
390	    if isinstance(image, list):
391	        for item in image:
392	            if isinstance(item, str) and item.startswith("http"):
393	                urls.append(item)
394	            elif isinstance(item, Mapping):
395	                value = item.get("url") or item.get("contentUrl")
396	                if isinstance(value, str) and value.startswith("http"):
397	                    urls.append(value)
398	    elif isinstance(image, str) and image.startswith("http"):
399	        urls.append(image)
400	    elif isinstance(image, Mapping):
401	        value = image.get("url") or image.get("contentUrl")
402	        if isinstance(value, str) and value.startswith("http"):
403	            urls.append(value)
404
405	    for match in _COMPASS_MEDIA_RE.findall(html):
406	        urls.append(match)
407	    for media_hash in _MEDIA_HASH_RE.findall(html):
408	        urls.append(f"https://www.compass.com/m/{media_hash}/origin.jpg")
409
410	    return _prefer_origin_unique(urls)
411
412
413	def _prefer_origin_unique(urls: Sequence[str]) -> tuple[str, ...]:
414	    best: dict[str, str] = {}
415	    order: list[str] = []
416	    for url in urls:
417	        clean = url.split("?")[0].replace("http://", "https://")
418	        if "profile" in clean.lower() or "avatar" in clean.lower():
419	            continue
420	        # agent headshots live under /m/<int>/uuid/
421	        if re.search(r"/m/\d+/[0-9a-f-]{20,}/", clean, re.I):
422	            continue
423	        match = re.search(r"/m/([0-9a-f]{32,})/", clean, re.I)
424	        key = match.group(1).lower() if match else clean.lower()
425	        score = 0
426	        lower = clean.lower()
427	        if "/origin." in lower:
428	            score += 3
429	        if "640x480" in lower:
430	            score += 1
431	        if lower.endswith((".jpg", ".jpeg")):
432	            score += 1
433	        if key not in best:
434	            order.append(key)
435	            best[key] = clean
436	        else:
437	            current = best[key]
438	            current_score = (3 if "/origin." in current.lower() else 0) + (1 if "640x480" in current.lower() else 0)
439	            if score > current_score:
440	                best[key] = clean
441	    return tuple(best[key] for key in order if key in best)
442
443
444	def _floor_size(ld: Mapping[str, Any]) -> Any:
445	    floor = ld.get("floorSize")
446	    if isinstance(floor, list) and floor:
447	        floor = floor[0]
448	    if isinstance(floor, Mapping):
449	        return floor.get("value")
450	    return floor
451
452
453	def _dig(node: Any, *path: Any) -> Any:
454	    current = node
455	    for key in path:
456	        if isinstance(key, int):
457	            if not isinstance(current, list) or key >= len(current):
458	                return None
459	            current = current[key]
460	            continue
461	        if not isinstance(current, Mapping) or key not in current:
462	            return None
463	        current = current[key]
464	    return current
465
466
467	def _first_str(*values: Any) -> str:
468	    for value in values:
469	        if value is None:
470	            continue
471	        text = str(value).strip()
472	        if text:
473	            return text
474	    return ""
475
476
477	def _coerce_int(value: Any) -> int | None:
478	    if value in (None, ""):
479	        return None
480	    if isinstance(value, bool):
481	        return None
482	    if isinstance(value, int):
483	        return value if value >= 0 else None
484	    if isinstance(value, float):
485	        if not value.is_integer() or value < 0:
486	            return None
487	        return int(value)
488	    text = str(value).strip().replace(",", "")
489	    if not text:
490	        return None
491	    match = re.search(r"\d+", text)
492	    if not match:
493	        return None
494	    try:
495	        parsed = int(match.group(0))
496	    except ValueError:
497	        return None
498	    return parsed if parsed >= 0 else None
499
500
501	def _coerce_float(value: Any) -> float | None:
502	    if value in (None, ""):
503	        return None
504	    if isinstance(value, bool):
505	        return None
506	    if isinstance(value, (int, float)):
507	        parsed = float(value)
508	        return parsed if parsed >= 0 else None
509	    text = str(value).strip().replace(",", "")
510	    match = re.search(r"\d+(?:\.\d+)?", text)
511	    if not match:
512	        return None
513	    try:
514	        parsed = float(match.group(0))
515	    except ValueError:
516	        return None
517	    return parsed if parsed >= 0 else None
518
519
520	def _extension_for_url(url: str) -> str:
521	    path = urlsplit(url).path.lower()
522	    for ext in (".jpg", ".jpeg", ".png", ".webp"):
523	        if path.endswith(ext):
524	            return ".jpg" if ext == ".jpeg" else ext
525	    return ".jpg"
526


1	from __future__ import annotations
2
3	import tempfile
4	import unittest
5	from pathlib import Path
6	from unittest.mock import MagicMock
7
8	from core.listings.manifest import ListingManifestError, SourcePolicy, load_listing_source, source_policy_for
9	from core.listings.providers.compass import (
10	    CompassListingError,
11	    fetch_compass_listing,
12	    is_compass_listing_url,
13	    parse_compass_listing_html,
14	)
15	from core.listings.providers.registry import build_authorized_fetcher
16
17	FIXTURE_HTML = Path(__file__).parent / "fixtures" / "compass" / "sample_listing.html"
18	COMPASS_URL = "https://www.compass.com/homedetails/1002-Horne-Dr-Cedar-Park-TX-78613/1877020858229275065_lid/"
19	PHOTO_A = "https://www.compass.com/m/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/origin.jpg"
20	PHOTO_B = "https://www.compass.com/m/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb/origin.jpg"
21
22
23	class CompassProviderTests(unittest.TestCase):
24	    def test_source_policy_allows_compass_adapter_path(self) -> None:
25	        self.assertTrue(is_compass_listing_url(COMPASS_URL))
26	        self.assertEqual(source_policy_for(COMPASS_URL), SourcePolicy.AUTHORIZED_ADAPTER)
27
28	    def test_parse_fixture_html_extracts_facts_and_deduped_photos(self) -> None:
29	        html = FIXTURE_HTML.read_text(encoding="utf-8")
30	        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)
31
32	        self.assertEqual(parsed.address, "1002 Horne Dr")
33	        self.assertEqual(parsed.city, "Cedar Park")
34	        self.assertEqual(parsed.state, "TX")
35	        self.assertEqual(parsed.postal_code, "78613")
36	        self.assertEqual(parsed.price, 692500)
37	        self.assertEqual(parsed.beds, 5)
38	        self.assertEqual(parsed.baths, 3)
39	        self.assertEqual(parsed.square_feet, 2684)
40	        self.assertIn("Cypress Canyon", parsed.description)
41	        self.assertEqual(parsed.agent_name, "Richard Fowler")
42	        self.assertEqual(parsed.photo_urls, (PHOTO_A, PHOTO_B))
43	        self.assertIn("pilot review only", parsed.attribution)
44
45	    def test_fetch_materializes_photos_and_manifest(self) -> None:
46	        html = FIXTURE_HTML.read_text(encoding="utf-8").encode("utf-8")
47	        photo_bytes = {
48	            PHOTO_A: b"P3\n1 1\n255\n220 210 190\n",
49	            PHOTO_B: b"P3\n1 1\n255\n10 20 30\n",
50	        }
51
52	        def http_get(url: str) -> bytes:
53	            if url.rstrip("/") == COMPASS_URL.rstrip("/") or url.startswith(COMPASS_URL):
54	                return html
55	            if url in photo_bytes:
56	                return photo_bytes[url]
57	            # origin rewrite variants for the second media hash may be requested as origin.jpg
58	            if url.endswith("/origin.jpg"):
59	                key = url
60	                for candidate, payload in photo_bytes.items():
61	                    if candidate.split("/")[-2] in url:
62	                        return payload
63	            raise AssertionError(f"unexpected URL {url}")
64
65	        with tempfile.TemporaryDirectory() as tmp:
66	            root = Path(tmp)
67	            snapshot = fetch_compass_listing(COMPASS_URL, media_root=root, http_get=http_get)
68
69	            self.assertEqual(snapshot.source_provider, "compass_public_demo")
70	            self.assertTrue(snapshot.rights_confirmed)
71	            self.assertEqual(snapshot.address, "1002 Horne Dr")
72	            self.assertEqual(len(snapshot.photos), 2)
73	            self.assertTrue((root / "photos" / "photo_01.jpg").exists())
74	            self.assertTrue((root / "photos" / "photo_02.jpg").exists())
75	            self.assertTrue((root / "listing.json").exists())
76	            self.assertEqual(len(snapshot.photos[0].sha256), 64)
77
78	    def test_registry_requires_demo_or_rights_flag(self) -> None:
79	        with tempfile.TemporaryDirectory() as tmp:
80	            root = Path(tmp)
81	            blocked = build_authorized_fetcher(media_root=root, demo=False, rights_confirmed=False)
82	            with self.assertRaisesRegex(ListingManifestError, "demo-only"):
83	                blocked(COMPASS_URL)
84
85	    def test_registry_uses_compass_adapter_when_demo_enabled(self) -> None:
86	        html = FIXTURE_HTML.read_text(encoding="utf-8").encode("utf-8")
87	        photo = b"P3\n1 1\n255\n1 2 3\n"
88
89	        def http_get(url: str) -> bytes:
90	            if "homedetails" in url:
91	                return html
92	            return photo
93
94	        with tempfile.TemporaryDirectory() as tmp:
95	            root = Path(tmp)
96	            fetcher = build_authorized_fetcher(
97	                media_root=root,
98	                demo=True,
99	                rights_confirmed=False,
100	                http_get=http_get,
101	            )
102	            snapshot = load_listing_source(COMPASS_URL, fetcher=fetcher)
103	            self.assertEqual(snapshot.source_provider, "compass_public_demo")
104	            self.assertEqual(snapshot.city, "Cedar Park")
105	            self.assertGreaterEqual(len(snapshot.photos), 1)
106
107	    def test_zillow_still_never_calls_fetcher(self) -> None:
108	        fetcher = MagicMock()
109	        zillow = "https://www.zillow.com/homedetails/example/1_zpid/"
110	        self.assertEqual(source_policy_for(zillow), SourcePolicy.MANIFEST_ONLY)
111	        with self.assertRaisesRegex(ListingManifestError, "authorized manifest"):
112	            load_listing_source(zillow, fetcher=fetcher)
113	        fetcher.assert_not_called()
114
115	    def test_parse_rejects_empty_html(self) -> None:
116	        with self.assertRaisesRegex(CompassListingError, "empty"):
117	            parse_compass_listing_html("   ", listing_url=COMPASS_URL)
118
119
120	if __name__ == "__main__":
121	    unittest.main()
122


find /workspace/extra/projects/stagehand -name "CLAUDE.md" | head -1


Let me search for the CLAUDE.md in the stagehand root and check for demo-related documentation:


1	# Stagehand Project Instructions
2	> Symlinked as `AGENTS.md`; edits affect both. Every agent (Claude Code, Codex) must read the same project instructions from here — do not reintroduce a divergent `AGENTS.md`.
3
4	This project is building a concierge-first local AI workflow service that can later become a platform.
5
6	## Listing Reel Engine (canonical)
7
8	The listing reel is rendered with **HyperFrames** (HTML → video). This is the one canonical engine for every agent — outputs live under `output/hyperframes/<project>/` as a HyperFrames project (`index.html`, `hyperframes.json`, `final_listing_reel.mp4` + `_delivery`). QA it with the repo-local skill `skills/hyperframes-reel-review/SKILL.md`.
9
10	- The older ffmpeg photo-slideshow path (`python -m cli.main run …`, writing `output/<uuid>/final_video.mp4` via `core/media/photo_reel.py`) is **legacy**. It stays for its tests and existing jobs, but it is not the default: do not build new listing reels through it. If a reel is requested, use HyperFrames.
11	- Rationale: the two paths previously produced qualitatively different videos (animated composition vs. Ken-Burns slideshow) in different folders depending on which agent ran — see the convergence decision. One engine, one output location.
12
13	## Execution Standard
14
15	- Use `docs/punch-list.md` as the project todo list and keep it current.
16	- Before starting a meaningful task, identify which punch-list items it advances.
17	- When work is completed and verified, update the checkbox in `docs/punch-list.md`.
18	- Do not mark discovery items complete until the required user/client input exists.
19	- Keep raw client/demo media, generated outputs, secrets, and private addresses out of git.
20	- For repeatable local HyperFrames reel QA, use the repo-local skill at `skills/hyperframes-reel-review/SKILL.md`.
21
22	## Testing Standard
23
24	- Testing is non-negotiable.
25	- Use TDD for deterministic business logic, config parsing, tenant scoping, approval gates, CLI behavior, and media command construction.
26	- Use BDD-style tests for multi-step workflow behavior such as "generated script pauses for approval" and "approved script resumes rendering."
27	- The default test suite must not require live paid API calls or network access.
28	- Live provider tests must be opt-in and guarded by explicit environment flags.
29	- For workflow or media-rendering changes, run a local end-to-end render through approval with a real fixture before calling the work implemented. Inspect or verify the produced video artifact enough to catch issues that unit tests miss, such as silent TTS fallback, static captions, source-audio leakage, or render duration problems.
30	- The full test suite runs automatically at the end of every turn via a `Stop` hook (`pytest -q`, configured in `.claude/settings.json`). Treat a failure it surfaces as a regression to fix before handing back; do not rely on it as a gate (it is non-blocking), and still run targeted tests while iterating.
31
32	## Current Scope
33
34	- Build a local CLI workflow first.
35	- Keep the system concierge-first: human review is part of the product.
36	- Do not add social posting, billing, authentication, or a web UI until the first workflow is proven.
37	- Vercel is acceptable later for thin UI/API surfaces, Neon for hosted Postgres, and Cloudflare for DNS/domains.
38	- Long-running video rendering should remain in a local/background worker, not in a Vercel request handler.
39


grep -r "demo.*flag\|--demo\|Expand Compass gallery" /workspace/extra/projects/stagehand --include="*.md" --include="*.txt" --include="*.py" | head -20


/workspace/extra/projects/stagehand/core/listings/providers/registry.py:                    "Pass --demo (pilot) or --rights-confirmed once brokerage/photographer authorization exists."
/workspace/extra/projects/stagehand/core/listings/providers/registry.py:                # non-demo production path. For now both flags use the public demo provider,
/workspace/extra/projects/stagehand/tests/test_compass_provider.py:    def test_registry_requires_demo_or_rights_flag(self) -> None:
/workspace/extra/projects/stagehand/tests/test_cli.py:                            "--demo",
/workspace/extra/projects/stagehand/tests/test_cli.py:    def test_listing_reel_requires_manifest_or_demo_flag(self) -> None:
/workspace/extra/projects/stagehand/output/hyperframes/openai-generated-painting-reel/.npm-cache/_npx/a10bdeb14f6d726a/node_modules/mitt/README.md:-   [Examples & Demos](#examples--demos)
/workspace/extra/projects/stagehand/output/hyperframes/aleidas-recommendation-reel/.npm-cache/_npx/a10bdeb14f6d726a/node_modules/mitt/README.md:-   [Examples & Demos](#examples--demos)
/workspace/extra/projects/stagehand/output/hyperframes/apex-pickleball-recommendation-reel/.npm-cache/_npx/a10bdeb14f6d726a/node_modules/mitt/README.md:-   [Examples & Demos](#examples--demos)
/workspace/extra/projects/stagehand/output/hyperframes/google-generated-painting-reel/.npm-cache/_npx/a10bdeb14f6d726a/node_modules/mitt/README.md:-   [Examples & Demos](#examples--demos)
/workspace/extra/projects/stagehand/output/hyperframes/house-painting-wipe-reel/.npm-cache/_npx/a10bdeb14f6d726a/node_modules/mitt/README.md:-   [Examples & Demos](#examples--demos)
/workspace/extra/projects/stagehand/output/hyperframes/openai-warm-to-modern-painting-reel/.npm-cache/_npx/a10bdeb14f6d726a/node_modules/mitt/README.md:-   [Examples & Demos](#examples--demos)
/workspace/extra/projects/stagehand/output/hyperframes/horne-dr-reel/.npm-cache/_npx/a10bdeb14f6d726a/node_modules/mitt/README.md:-   [Examples & Demos](#examples--demos)
/workspace/extra/projects/stagehand/output/hyperframes/google-generated-painting-reel-lite-test/.npm-cache/_npx/a10bdeb14f6d726a/node_modules/mitt/README.md:-   [Examples & Demos](#examples--demos)
/workspace/extra/projects/stagehand/output/hyperframes/broadbay-dr-reel/.npm-cache/_npx/a10bdeb14f6d726a/node_modules/mitt/README.md:-   [Examples & Demos](#examples--demos)
/workspace/extra/projects/stagehand/output/hyperframes/landscaping-before-after-reel/.npm-cache/_npx/a10bdeb14f6d726a/node_modules/mitt/README.md:-   [Examples & Demos](#examples--demos)
/workspace/extra/projects/stagehand/output/hyperframes/ramons-house-paint-before-after-reel/.npm-cache/_npx/a10bdeb14f6d726a/node_modules/mitt/README.md:-   [Examples & Demos](#examples--demos)
/workspace/extra/projects/stagehand/output/hyperframes/pool-upgrade-recommendation-reel/.npm-cache/_npx/a10bdeb14f6d726a/node_modules/mitt/README.md:-   [Examples & Demos](#examples--demos)
/workspace/extra/projects/stagehand/output/hyperframes/turf-upgrade-recommendation-reel/.npm-cache/_npx/a10bdeb14f6d726a/node_modules/mitt/README.md:-   [Examples & Demos](#examples--demos)
/workspace/extra/projects/stagehand/docs/punch-list.md:- [x] Add Compass demo link-intake adapter (`--demo`) that materializes local photos + snapshot while keeping Zillow/etc. no-fetch and production rights as a ship blocker.
/workspace/extra/projects/stagehand/docs/punch-list.md:- [ ] Expand Compass gallery capture further / fail closed below min photo count for demos.


Now let me check the punch-list and registry to understand the full context:


1	# CreatorFlow Punch List
2
3	This list tracks the work needed to prove the first local AI workflow before broadening into a platform.
4
5	## Project Standards
6
7	- [x] Keep a repo-level todo/punch list as a standard project operating artifact.
8	- [x] Add project-level agent instructions requiring the punch list to stay current.
9
10	## Discovery
11
12	- [x] Prepare a first-agent discovery packet and acceptance criteria worksheet.
13	- [ ] Collect 5-8 raw walkthrough videos from the first real estate agent.
14	- [ ] Collect examples of posts/videos the agent considers publishable.
15	- [ ] Confirm target turnaround time, revision expectations, and pricing signal.
16	- [ ] Capture real estate compliance preferences and required brokerage/disclaimer language.
17	- [ ] Define first workflow acceptance criteria from the discovery conversation.
18
19	## Product Guardrails
20
21	- [x] Keep the first version concierge-first: client sends raw input, operator reviews, client receives finished output.
22	- [x] Require human approval before final rendered output is considered publishable.
23	- [x] Do not automate posting in the first version.
24	- [x] Use a licensed stock voice or explicitly approved voice only.
25	- [x] Track manual touch time per video to identify the next automation target.
26
27	## Technical Foundation
28
29	- [x] Initialize Python project scaffold with test runner.
30	- [x] Add local Postgres setup and initial migrations.
31	- [x] Model tenants, media assets, jobs, and job steps with approval state.
32	- [x] Build workflow config parser and runner with approval gates.
33	- [x] Build CLI commands for running workflows, checking status, and approving scripts.
34	- [x] Add media utilities for ffmpeg command construction and rendering.
35	- [x] Add a local no-paid-API demo render path that produces a playable vertical MP4 from the phone video.
36	- [x] Render transcript-derived TTS narration with synchronized captions and source audio muted by default.
37	- [x] Wire real transcription via OpenAI Whisper (`--transcription-provider whisper`). Audio is extracted before submission to avoid container format issues with the API.
38	- [x] Improve transcript cleanup: remove filler words, hesitation sounds, consecutive duplicate words. Re-capitalize after removal.
39	- [x] Caption timing uses Whisper segment timestamps (scaled to TTS duration) when available, with equal-duration chunks as the offline fallback.
40	- [x] Evaluate provider-backed word-level timestamps for tighter per-word caption sync (current path uses sentence-level segments).
41	- [x] Make local TTS verification stricter and investigate macOS `say` intermittently producing header-only audio before falling back to silent demo audio.
42
43	## Listing URL Reel
44
45	- [x] Research real-estate reel competitors, HeyGen/HyperFrames capabilities, listing-source terms, Instagram output requirements, and voice-cloning constraints.
46	- [x] Write an implementation-ready listing URL to Instagram reel plan.
47	- [x] Add a canonical authorized listing manifest with source policy, path confinement, rights attestation, and photo integrity hashes.
48	- [x] Add a single approval bundle covering facts, narration, caption, photo order, cover choice, and contact outro.
49	- [x] Render a 9:16 photo reel with restrained pan/zoom motion, captions, narration, and a contact outro.
50	- [x] Produce and visually QA two distinct HyperFrames portrait listing-reel variants from authorized local pilot media (private demo outputs remain ignored).
51	- [x] Revise the pilot reels with all supplied Horne listing photos and concise Richard Fowler contact-card outros; remove unsupported mood copy.
52	- [x] Generate a designed property cover and include it in the manual Instagram handoff package.
53	- [x] Require consent and audible output when cloned narration is requested.
54	- [x] Run and inspect a local approval-resume end-to-end listing reel render with a real fixture (intentional silent provider; live cloned-voice validation still requires owner consent and credentials).
55	- [x] Add Compass demo link-intake adapter (`--demo`) that materializes local photos + snapshot while keeping Zillow/etc. no-fetch and production rights as a ship blocker.
56	- [x] Soften photo-reel Ken Burns: continuous center scale (no zoompan), 60fps, CRF 18, light denoise, quality-first encode.
57	- [x] Force spoken “square feet” + city/state-only narration (no street “Dr=doctor”).
58	- [x] Write Richard-facing ElevenLabs PVC + optional HeyGen avatar setup steps for private share.
59	- [x] Local Coqui XTTS sample against Richard reference wav for pilot listen.
60	- [x] Add operator-facing pre/post render reel QA scorecards plus private review links for remote artifact checks.
61	- [x] Add a repeatable HyperFrames reel review command/skill plus a reboot-persistent private review server.
62	- [x] Fix hyperframes-video-review live CLI checks hanging: bound every review subprocess with a timeout, drop the network-dependent Gemini `snapshot --describe` step (unused by our scoring) and the update-check ping.
63	- [x] Add `hyperframes-add-narration` command + skill doc: standard, repeatable fix for a video-review "missing required narration/audio" hard failure (synthesize via local `say`, mux into the silent HyperFrames render).
64	- [x] Name packaged delivery videos consistently with early `property` / `recommendation` type segments for easier sharing and sorting.
65	- [ ] Wire natural voice default to Richard’s shared ElevenLabs PVC once he completes PVC share + consent.
66	- [ ] Optional: Coqui XTTS as first-class listing-reel TTS provider (research-only until PVC lands).
67	- [ ] Expand Compass gallery capture further / fail closed below min photo count for demos.
68	- [ ] PureRealty adapter after we clear cookie/login or get an authorized feed/export.
69	- [ ] Production rights path: broker attestation workflow + licensed Bridge/RESO option.
70
71	## Before/After Contractor Reel Template
72
73	- [x] Build a reusable HyperFrames before/after template (50/50 split-screen, before on top / after on bottom, portrait full-bleed) as a new reel family alongside the existing listing and recommendation reels.
74	- [x] Produce a landscaping demo instance (`output/hyperframes/landscaping-before-after-reel/`) with 4 before/after beats + shared contact-card outro, narration, and a passing script/video QA review.
75	- [x] Document swap points for reuse across other contractor trades (painter, remodeler, pool, etc.) in that project's `TEMPLATE-NOTES.md`.
76	- [x] Get one real (non-AI-generated) before/after photo set from an actual contractor to validate the template against real client media.
77	- [x] Multi-client support: swapped the demo's outro to a direct-brand-CTA variant (placeholder "Contract Matcher" brand, Morgan Hill CA, placeholder phone) instead of Richard's personal-referral card; documented the outro as the one "client identity block" to fork per client in `TEMPLATE-NOTES.md`.
78	- [x] Promoted the client identity block (brand name, outro headline/tagline/city/phone, avatar monogram, accent colors) to HyperFrames `data-composition-variables`, resolved at init and configurable via `client.config.json` (`--variables-file` at render time) instead of hand-edited HTML. Verified the override plumbing end-to-end with a throwaway low-fps render before finalizing.
79	- [ ] Get Contract Matcher's real business name, phone number (for the text-based scheduler being built separately), web address, and logo to replace the placeholders in `client.config.json`.
80
81	## Testing
82
83	- [x] Use TDD for deterministic business logic, config parsing, tenant scoping, approval gates, and media command construction.
84	- [x] Use BDD-style scenarios for the video narration workflow.
85	- [x] Mock paid/external APIs in the default test suite.
86	- [x] Add opt-in live integration tests only after mocked contracts are stable.
87	- [x] Add a short test video fixture for the end-to-end path.
88	- [x] Verify local render execution against the committed short video fixture.
89
90	## Provider Setup
91
92	- [x] Document local `.env` setup.
93	- [x] Document Neon database creation and migration steps.
94	- [x] Document Vercel env vars and deployment steps for any later approval UI/API.
95	- [x] Document Cloudflare DNS setup for the chosen domain/subdomain.
96	- [x] Keep long-running video rendering out of Vercel request handlers.
97


1	"""Source-provider registry for authorized listing adapters."""
2
3	from __future__ import annotations
4
5	from collections.abc import Callable
6	from pathlib import Path
7	from urllib.parse import urlsplit
8
9	from core.listings.manifest import ListingManifestError, normalize_listing_url
10	from core.listings.models import ListingSnapshot
11	from core.listings.providers.compass import fetch_compass_listing, is_compass_listing_url
12
13
14	Fetcher = Callable[[str], ListingSnapshot]
15
16
17	def build_authorized_fetcher(
18	    *,
19	    media_root: str | Path,
20	    demo: bool = False,
21	    rights_confirmed: bool = False,
22	    http_get: Callable[[str], bytes] | None = None,
23	    max_photos: int = 20,
24	) -> Fetcher:
25	    """Return a fetcher that routes allowlisted domains to their adapter.
26
27	    Demo adapters require an explicit demo opt-in. Shipping still needs a real
28	    rights path; demo status is recorded on the snapshot provider string and
29	    attribution so the approval bundle remains honest.
30	    """
31
32	    root = Path(media_root)
33
34	    def fetcher(listing_url: str) -> ListingSnapshot:
35	        normalized = normalize_listing_url(listing_url)
36	        host = (urlsplit(normalized).hostname or "").lower()
37	        if is_compass_listing_url(normalized):
38	            if not (demo or rights_confirmed):
39	                raise ListingManifestError(
40	                    "Compass public page intake is demo-only without broker rights attestation. "
41	                    "Pass --demo (pilot) or --rights-confirmed once brokerage/photographer authorization exists."
42	                )
43	            snapshot = fetch_compass_listing(
44	                normalized,
45	                media_root=root,
46	                http_get=http_get,
47	                max_photos=max_photos,
48	            )
49	            if rights_confirmed and not demo:
50	                # Preserve demo string unless rights were explicitly asserted for a future
51	                # non-demo production path. For now both flags use the public demo provider,
52	                # but attribution already states the demojudge role; rights_confirmed is
53	                # still required as an operator assertion.
54	                return snapshot
55	            return snapshot
56	        raise ListingManifestError(
57	            f"No authorized listing adapter is configured for {host or 'unknown host'}. "
58	            "Supply --manifest/--media-root, or use a supported demo source (compass.com)."
59	        )
60
61	    return fetcher
62


Perfect. Now let me check what the current photo-collection logic looks like and verify if there's any min-photo enforcement:


grep -n "min.*photo\|photo.*count\|max_photos" /workspace/extra/projects/stagehand/core/listings/providers/compass.py


162:    max_photos: int = 20,
170:    for index, photo_url in enumerate(parsed.photo_urls[: max(1, max_photos)]):
242:    max_photos: int = 20,
258:        max_photos=max_photos,


Now let me look at the photo URL collection logic more carefully and check what sources are being scraped:


367	def _collect_photo_urls(
368	    html: str,
369	    listing: Mapping[str, Any] | None,
370	    ld: Mapping[str, Any],
371	) -> tuple[str, ...]:
372	    urls: list[str] = []
373	    if listing is not None:
374	        media = listing.get("media")
375	        if isinstance(media, Mapping):
376	            for key in ("originalUrl", "thumbnailUrl", "url"):
377	                value = media.get(key)
378	                if isinstance(value, str) and value.startswith("http"):
379	                    urls.append(value)
380	        elif isinstance(media, list):
381	            for item in media:
382	                if not isinstance(item, Mapping):
383	                    continue
384	                for key in ("originalUrl", "url", "thumbnailUrl"):
385	                    value = item.get(key)
386	                    if isinstance(value, str) and value.startswith("http"):
387	                        urls.append(value)
388	                        break
389	    image = ld.get("image")
390	    if isinstance(image, list):
391	        for item in image:
392	            if isinstance(item, str) and item.startswith("http"):
393	                urls.append(item)
394	            elif isinstance(item, Mapping):
395	                value = item.get("url") or item.get("contentUrl")
396	                if isinstance(value, str) and value.startswith("http"):
397	                    urls.append(value)
398	    elif isinstance(image, str) and image.startswith("http"):
399	        urls.append(image)
400	    elif isinstance(image, Mapping):
401	        value = image.get("url") or image.get("contentUrl")
402	        if isinstance(value, str) and value.startswith("http"):
403	            urls.append(value)
404
405	    for match in _COMPASS_MEDIA_RE.findall(html):
406	        urls.append(match)
407	    for media_hash in _MEDIA_HASH_RE.findall(html):
408	        urls.append(f"https://www.compass.com/m/{media_hash}/origin.jpg")
409
410	    return _prefer_origin_unique(urls)
411
412
413	def _prefer_origin_unique(urls: Sequence[str]) -> tuple[str, ...]:
414	    best: dict[str, str] = {}
415	    order: list[str] = []
416	    for url in urls:


Perfect. I now have enough information. Let me compile my findings:

## Summary

**Current Compass Adapter State:**

The Compass adapter (`core/listings/providers/compass.py`) is a demo-only intake for compass.com listing pages. It:
- Extracts listing facts (address, price, beds, baths, sqft, description, agent name) from HTML via `window.__INITIAL_DATA__` JSON or JSON-LD schema
- Collects photos from four sources: (1) `__INITIAL_DATA__.props.listingRelation.listing.media` array, (2) JSON-LD `image` field, (3) regex pattern matching `/m/[0-9a-f]{40,}/` media hashes in raw HTML, (4) Compass media URL regex (lines 405-408)
- Deduplicates URLs by media hash and prefers `/origin.` and `.jpg` variants (lines 413-441)
- Downloads up to `max_photos` (default 20) and materializes them locally with SHA256 hashes
- Requires `--demo` flag at CLI level (registry.py enforces this)

**"Expand Gallery Capture Further":**

Currently, photos are scraped from only **existing HTML metadata** (embedded JSON + regex patterns). The task is to expand this to capture a larger/complete photo gallery. Compass likely exposes more photos through pagination, lazy-loaded content, or API calls that the current HTML regex parsing misses. The adapter would need additional discovery logic to fetch all gallery photos, not just those already in the initial page HTML.

**"Fail Closed Below Min Photo Count":**

There is **NO minimum photo check** currently. The code (line 128-129) only rejects listings with zero photos (`if not photo_urls`), and later (line 195-196) rejects if zero photos download. There's no explicit minimum threshold (e.g., "reject if < 5 photos"). The task is to add a configurable minimum (e.g., require 5+ photos for demo acceptance) and fail explicitly if that threshold isn't met, rather than proceeding with fewer photos.

**Existing Tests:**
- `/workspace/extra/projects/stagehand/tests/test_compass_provider.py` covers: URL validation, fact extraction from fixture HTML, photo deduplication, materialization + manifest writing, registry gating (--demo required), and error cases (empty HTML, no photos, network failures).