Looking at all the eligible projects:
1. StageHand (P1): Blocked on James's 5 toolchain questions - SKIP
2. nanoclaw (P1): Blocked on Studio deploy - can't deploy without James; also needs metric confirm - SKIP
3. AgenticLever (P2): "Confirm goal" - SKIP
4. RoadSense (P2): "Answer 5 launch questions" - SKIP
5. qip-qien (P2): review-required, SEV-1 fix already committed (ee84254), needs merge sign-off

Actually, everything here is blocked in some way. The instructions are clear: skip anything waiting on James or external input, and for review-required items, produce the change and summary for James to approve. But qip-qien's change is already done and sitting in the decision queue—there's nothing new for me to produce there. Let me check if there's anything else in nanoclaw I could tackle autonomously, since the next action listed is just James reviewing the push.

Looking at the nanoclaw metric, it's also blocked on James to name it. And for StageHand, while it's waiting on James's answers about the toolchain questions, I notice there's an autonomous punch-list item I could work on: expanding the Compass gallery capture to handle lazy-loaded and paginated photos beyond the initial HTML metadata. The gallery expansion work is technically autonomous and doesn't depend on James's answers, but it's a medium-complexity feature that would require understanding Compass's lazy-loading behavior and implementing a more sophisticated scraper—possibly with a headless browser or paginated API calls using a TDD approach.

With only 21% burnable capacity at the medium tier, this feels like it could balloon into a full high-tier task, especially without direct access to a live Compass page for testing. I should look for something more bounded instead. Let me reconsider the other options—maybe I can help move the qip-qien review forward by sending James a clearer summary of the SEV-1 fix to help him make the merge decision. tier constraints, I should probably focus on what's genuinely actionable. The StageHand Compass expansion is a clear option but might exceed medium tier scope, while the qip-qien code review is more contained—I could do a thorough analysis and present it to James for approval. Let me also check if there are any smaller improvements or issues in the nanoclaw repo that don't depend on James's decisions.


1	"""Compass allowlisted demo adapter.
2
3	Demo-grade intake for Richard Fowler compass.com listing pages.
4
5	This is intentionally *not* a production rights path. Production still needs
6	brokerage / photographer / MLS authorization (or a licensed feed). The adapter
7	records demo provenance so human review never confuses pilot media with a
8	shipped rights-cleared source.
9	"""
10
11	from __future__ import annotations
12
13	import hashlib
14	import json
15	import re
16	from dataclasses import dataclass
17	from datetime import UTC, datetime
18	from pathlib import Path
19	from typing import Any, Callable, Mapping, Sequence
20	from urllib.error import HTTPError, URLError
21	from urllib.parse import urlsplit
22	from urllib.request import Request, urlopen
23
24	from core.listings.manifest import ListingManifestError, normalize_listing_url
25	from core.listings.models import ListingPhoto, ListingSnapshot
26
27	COMPASS_HOSTS = frozenset({"compass.com", "www.compass.com"})
28	DEFAULT_USER_AGENT = (
29	    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
30	    "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
31	)
32	_INITIAL_DATA_RE = re.compile(r"window\.__INITIAL_DATA__\s*=\s*")
33	_MEDIA_HASH_RE = re.compile(r"/m/([0-9a-f]{40,})/", re.I)
34	_COMPASS_MEDIA_RE = re.compile(
35	    r"https://www\.compass\.com/m/(?:[0-9]+/)?[0-9a-fA-F-]+/(?:origin|640x480)\.(?:jpg|jpeg|webp)",
36	    re.I,
37	)
38
39
40	class CompassListingError(ListingManifestError):
41	    """Raised when a Compass listing cannot be ingested for the demo workflow."""
42
43
44	@dataclass(frozen=True, slots=True)
45	class CompassListingParse:
46	    source_url: str
47	    address: str
48	    city: str
49	    state: str
50	    postal_code: str
51	    price: int | None
52	    beds: float | None
53	    baths: float | None
54	    square_feet: int | None
55	    description: str
56	    photo_urls: tuple[str, ...]
57	    attribution: str
58	    agent_name: str = ""
59
60
61	HttpGet = Callable[[str], bytes]
62
63
64	def is_compass_listing_url(listing_url: str) -> bool:
65	    try:
66	        host = (urlsplit(normalize_listing_url(listing_url)).hostname or "").lower()
67	    except ListingManifestError:
68	        return False
69	    return host in COMPASS_HOSTS
70
71
72	def parse_compass_listing_html(html: str, *, listing_url: str) -> CompassListingParse:
73	    if not html or not html.strip():
74	        raise CompassListingError("Compass listing HTML was empty")
75
76	    source_url = normalize_listing_url(listing_url)
77	    initial = _extract_initial_data(html)
78	    listing = _listing_blob(initial) if initial is not None else None
79	    ld = _extract_ld_listing(html)
80

"""Compass allowlisted demo adapter.

Demo-grade intake for Richard Fowler compass.com listing pages.

This is intentionally *not* a production rights path. Production still needs
brokerage / photographer / MLS authorization (or a licensed feed). The adapter
records demo provenance so human review never confuses pilot media with a
shipped rights-cleared source.
"""

from __future__ import annotations

import hashlib
import json
import re
from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
from typing import Any, Callable, Mapping, Sequence
from urllib.error import HTTPError, URLError
from urllib.parse import urlsplit
from urllib.request import Request, urlopen

from core.listings.manifest import ListingManifestError, normalize_listing_url
from core.listings.models import ListingPhoto, ListingSnapshot

COMPASS_HOSTS = frozenset({"compass.com", "www.compass.com"})
DEFAULT_USER_AGENT = (
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
    "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
)
_INITIAL_DATA_RE = re.compile(r"window\.__INITIAL_DATA__\s*=\s*")
_MEDIA_HASH_RE = re.compile(r"/m/([0-9a-f]{40,})/", re.I)
_COMPASS_MEDIA_RE = re.compile(
    r"https://www\.compass\.com/m/(?:[0-9]+/)?[0-9a-fA-F-]+/(?:origin|640x480)\.(?:jpg|jpeg|webp)",
    re.I,
)

class CompassListingError(ListingManifestError):
    """Raised when a Compass listing cannot be ingested for the demo workflow."""

@dataclass(frozen=True, slots=True)
class CompassListingParse:
    source_url: str
    address: str
    city: str
    state: str
    postal_code: str
    price: int | None
    beds: float | None
    baths: float | None
    square_feet: int | None
    description: str
    photo_urls: tuple[str, ...]
    attribution: str
    agent_name: str = ""

HttpGet = Callable[[str], bytes]

def is_compass_listing_url(listing_url: str) -> bool:
    try:
        host = (urlsplit(normalize_listing_url(listing_url)).hostname or "").lower()
    except ListingManifestError:
        return False
    return host in COMPASS_HOSTS

def parse_compass_listing_html(html: str, *, listing_url: str) -> CompassListingParse:
    if not html or not html.strip():
        raise CompassListingError("Compass listing HTML was empty")

    source_url = normalize_listing_url(listing_url)
    initial = _extract_initial_data(html)
    listing = _listing_blob(initial) if initial is not None else None
    ld = _extract_ld_listing(html)


Ingest Claude blog post on marketing operations automation


80
81	    address = _first_str(
82	        _dig(listing, "location", "streetAddress"),
83	        _dig(ld, "address", "streetAddress"),
84	    )
85	    city = _first_str(
86	        _dig(listing, "location", "city"),
87	        _dig(ld, "address", "addressLocality"),
88	    )
89	    state = _first_str(
90	        _dig(listing, "location", "state"),
91	        _dig(ld, "address", "addressRegion"),
92	    )
93	    postal_code = _first_str(
94	        _dig(listing, "location", "zipCode"),
95	        _dig(listing, "location", "postalCode"),
96	        _dig(ld, "address", "postalCode"),
97	    )
98	    description = _first_str(
99	        _dig(listing, "description"),
100	        _dig(ld, "description"),
101	    ).strip()
102	    price = _coerce_int(
103	        _dig(listing, "price", "listed")
104	        or _dig(listing, "price", "amount")
105	        or _dig(listing, "price")
106	        or _dig(ld, "offers", "price")
107	    )
108	    beds = _coerce_float(
109	        _dig(listing, "size", "bedrooms")
110	        or _dig(listing, "size", "beds")
111	        or _dig(ld, "numberOfRooms")
112	    )
113	    baths = _coerce_float(
114	        _dig(listing, "size", "bathrooms")
115	        or _dig(listing, "size", "totalBathrooms")
116	        or _dig(listing, "size", "baths")
117	    )
118	    square_feet = _coerce_int(
119	        _dig(listing, "size", "squareFeet")
120	        or _dig(listing, "size", "sqft")
121	        or _floor_size(ld)
122	    )
123	    agent_name = _first_str(
124	        _dig(listing, "fullContacts", 0, "name"),
125	        _dig(ld, "offers", "offeredBy", "name"),
126	    )
127	    photo_urls = _collect_photo_urls(html, listing, ld)
128	    if not photo_urls:
129	        raise CompassListingError("Compass listing page exposed no property photos")
130	    if not any((price is not None, beds is not None, baths is not None, square_feet is not None, description)):
131	        raise CompassListingError("Compass listing page exposed no usable listing facts")
132
133	    attribution = (
134	        "Demo intake from compass.com public listing page for pilot review only. "
135	        "Not a production rights-cleared source — obtain broker/photographer/MLS authorization before shipping."
136	    )
137	    if agent_name:
138	        attribution = f"{attribution} Listed agent on page: {agent_name}."
139
140	    return CompassListingParse(
141	        source_url=source_url,
142	        address=address,
143	        city=city,
144	        state=state,
145	        postal_code=postal_code,
146	        price=price,
147	        beds=beds,
148	        baths=baths,
149	        square_feet=square_feet,
150	        description=description,
151	        photo_urls=photo_urls,
152	        attribution=attribution,
153	        agent_name=agent_name,
154	    )
155
156
157	def materialize_compass_listing(
158	    parsed: CompassListingParse,
159	    *,
160	    media_root: str | Path,
161	    http_get: HttpGet | None = None,
162	    max_photos: int = 20,
163	    min_photos: int = 5,
164	) -> ListingSnapshot:
165	    """Download photo URLs into media_root and return a ListingSnapshot."""
166	    root = Path(media_root)
167	    photos_dir = root / "photos"
168	    photos_dir.mkdir(parents=True, exist_ok=True)
169	    getter = http_get or _default_http_get
170	    photos: list[ListingPhoto] = []
171	    for index, photo_url in enumerate(parsed.photo_urls[: max(1, max_photos)]):
172	        suffix = _extension_for_url(photo_url)
173	        relative = Path("photos") / f"photo_{index + 1:02d}{suffix}"
174	        dest = root / relative
175	        try:
176	            payload = getter(photo_url)
177	        except CompassListingError:
178	            raise
179	        except Exception as exc:  # pragma: no cover - network seams cover this with mocked get
180	            raise CompassListingError(f"Failed to download Compass photo {index + 1}: {exc}") from exc
181	        if not payload:
182	            raise CompassListingError(f"Compass photo {index + 1} was empty")
183	        dest.write_bytes(payload)
184	        digest = hashlib.sha256(payload).hexdigest()
185	        photos.append(
186	            ListingPhoto(
187	                path=dest.resolve(),
188	                relative_path=relative.as_posix(),
189	                sha256=digest,
190	                size_bytes=len(payload),
191	                label=f"Photo {index + 1}",
192	                focal_point=(0.5, 0.5),
193	            )
194	        )
195
196	    if not photos:
197	        raise CompassListingError("No Compass photos were downloaded")
198	    if len(photos) < min_photos:
199	        raise CompassListingError(
200	            f"Compass demo requires at least {min_photos} photos for a usable reel; "
201	            f"only {len(photos)} downloaded. Expand the gallery or lower min_photos."
202	        )
203
204	    return ListingSnapshot(
205	        source_url=parsed.source_url,
206	        source_provider="compass_public_demo",
207	        rights_confirmed=True,
208	        attribution=parsed.attribution,
209	        address=parsed.address,
210	        city=parsed.city,
211	        state=parsed.state,
212	        postal_code=parsed.postal_code,
213	        price=parsed.price,
214	        beds=parsed.beds,
215	        baths=parsed.baths,
216	        square_feet=parsed.square_feet,
217	        description=parsed.description,
218	        photos=tuple(photos),
219	        cover_photo_index=0,

    address = _first_str(
        _dig(listing, "location", "streetAddress"),
        _dig(ld, "address", "streetAddress"),
    )
    city = _first_str(
        _dig(listing, "location", "city"),
        _dig(ld, "address", "addressLocality"),
    )
    state = _first_str(
        _dig(listing, "location", "state"),
        _dig(ld, "address", "addressRegion"),
    )
    postal_code = _first_str(
        _dig(listing, "location", "zipCode"),
        _dig(listing, "location", "postalCode"),
        _dig(ld, "address", "postalCode"),
    )
    description = _first_str(
        _dig(listing, "description"),
        _dig(ld, "description"),
    ).strip()
    price = _coerce_int(
        _dig(listing, "price", "listed")
        or _dig(listing, "price", "amount")
        or _dig(listing, "price")
        or _dig(ld, "offers", "price")
    )
    beds = _coerce_float(
        _dig(listing, "size", "bedrooms")
        or _dig(listing, "size", "beds")
        or _dig(ld, "numberOfRooms")
    )
    baths = _coerce_float(
        _dig(listing, "size", "bathrooms")
        or _dig(listing, "size", "totalBathrooms")
        or _dig(listing, "size", "baths")
    )
    square_feet = _coerce_int(
        _dig(listing, "size", "squareFeet")
        or _dig(listing, "size", "sqft")
        or _floor_size(ld)
    )
    agent_name = _first_str(
        _dig(listing, "fullContacts", 0, "name"),
        _dig(ld, "offers", "offeredBy", "name"),
    )
    photo_urls = _collect_photo_urls(html, listing, ld)
    if not photo_urls:
        raise CompassListingError("Compass listing page exposed no property photos")
    if not any((price is not None, beds is not None, baths is not None, square_feet is not None, description)):
        raise CompassListingError("Compass listing page exposed no usable listing facts")

    attribution = (
        "Demo intake from compass.com public listing page for pilot review only. "
        "Not a production rights-cleared source — obtain broker/photographer/MLS authorization before shipping."
    )
    if agent_name:
        attribution = f"{attribution} Listed agent on page: {agent_name}."

    return CompassListingParse(
        source_url=source_url,
        address=address,
        city=city,
        state=state,
        postal_code=postal_code,
        price=price,
        beds=beds,
        baths=baths,
        square_feet=square_feet,
        description=description,
        photo_urls=photo_urls,
        attribution=attribution,
        agent_name=agent_name,
    )

def materialize_compass_listing(
    parsed: CompassListingParse,
    *,
    media_root: str | Path,
    http_get: HttpGet | None = None,
    max_photos: int = 20,
    min_photos: int = 5,
) -> ListingSnapshot:
    """Download photo URLs into media_root and return a ListingSnapshot."""
    root = Path(media_root)
    photos_dir = root / "photos"
    photos_dir.mkdir(parents=True, exist_ok=True)
    getter = http_get or _default_http_get
    photos: list[ListingPhoto] = []
    for index, photo_url in enumerate(parsed.photo_urls[: max(1, max_photos)]):
        suffix = _extension_for_url(photo_url)
        relative = Path("photos") / f"photo_{index + 1:02d}{suffix}"
        dest = root / relative
        try:
            payload = getter(photo_url)
        except CompassListingError:
            raise
        except Exception as exc:  # pragma: no cover - network seams cover this with mocked get
            raise CompassListingError(f"Failed to download Compass photo {index + 1}: {exc}") from exc
        if not payload:
            raise CompassListingError(f"Compass photo {index + 1} was empty")
        dest.write_bytes(payload)
        digest = hashlib.sha256(payload).hexdigest()
        photos.append(
            ListingPhoto(
                path=dest.resolve(),
                relative_path=relative.as_posix(),
                sha256=digest,
                size_bytes=len(payload),
                label=f"Photo {index + 1}",
                focal_point=(0.5, 0.5),
            )
        )

    if not photos:
        raise CompassListingError("No Compass photos were downloaded")
    if len(photos) < min_photos:
        raise CompassListingError(
            f"Compass demo requires at least {min_photos} photos for a usable reel; "
            f"only {len(photos)} downloaded. Expand the gallery or lower min_photos."
        )

    return ListingSnapshot(
        source_url=parsed.source_url,
        source_provider="compass_public_demo",
        rights_confirmed=True,
        attribution=parsed.attribution,
        address=parsed.address,
        city=parsed.city,
        state=parsed.state,
        postal_code=parsed.postal_code,
        price=parsed.price,
        beds=parsed.beds,
        baths=parsed.baths,
        square_feet=parsed.square_feet,
        description=parsed.description,
        photos=tuple(photos),
        cover_photo_index=0,


220	        created_at=datetime.now(UTC).isoformat(),
221	    )
222
223
224	def write_manifest_from_snapshot(snapshot: ListingSnapshot, media_root: str | Path) -> Path:
225	    """Write a version-1 operator manifest next to downloaded photos for re-runs."""
226	    root = Path(media_root)
227	    root.mkdir(parents=True, exist_ok=True)
228	    path = root / "listing.json"
229	    payload = snapshot.to_dict()
230	    # Manifest format uses path/label/focal_point only on photos.
231	    payload["photos"] = [
232	        {
233	            "path": photo.relative_path,
234	            "label": photo.label,
235	            "focal_point": list(photo.focal_point),
236	        }
237	        for photo in snapshot.photos
238	    ]
239	    path.write_text(json.dumps(payload, indent=2), encoding="utf-8")
240	    return path
241
242
243	def fetch_compass_listing(
244	    listing_url: str,
245	    *,
246	    media_root: str | Path,
247	    http_get: HttpGet | None = None,
248	    max_photos: int = 20,
249	    min_photos: int = 5,
250	) -> ListingSnapshot:
251	    if not is_compass_listing_url(listing_url):
252	        raise CompassListingError("Not a Compass listing URL")
253	    getter = http_get or _default_http_get
254	    try:
255	        html = getter(normalize_listing_url(listing_url)).decode("utf-8", "replace")
256	    except CompassListingError:
257	        raise
258	    except Exception as exc:
259	        raise CompassListingError(f"Failed to fetch Compass listing page: {exc}") from exc
260	    parsed = parse_compass_listing_html(html, listing_url=listing_url)
261	    snapshot = materialize_compass_listing(
262	        parsed,
263	        media_root=media_root,
264	        http_get=getter,
265	        max_photos=max_photos,
266	        min_photos=min_photos,
267	    )
268	    write_manifest_from_snapshot(snapshot, media_root)
269	    return snapshot
270
271
272	def _default_http_get(url: str) -> bytes:
273	    request = Request(
274	        url,
275	        headers={
276	            "User-Agent": DEFAULT_USER_AGENT,
277	            "Accept": "text/html,application/xhtml+xml,application/json,image/avif,image/webp,*/*;q=0.8",
278	            "Accept-Language": "en-US,en;q=0.9",
279	        },
280	    )
281	    try:
282	        with urlopen(request, timeout=30) as response:  # noqa: S310 - operator-requested live URL fetch
283	            return response.read()
284	    except HTTPError as exc:
285	        raise CompassListingError(f"HTTP {exc.code} fetching {url}") from exc
286	    except URLError as exc:
287	        raise CompassListingError(f"Network error fetching {url}: {exc.reason}") from exc
288
289
290	def _extract_initial_data(html: str) -> dict[str, Any] | None:
291	    match = _INITIAL_DATA_RE.search(html)
292	    if not match:
293	        return None
294	    start = match.end()
295	    while start < len(html) and html[start].isspace():
296	        start += 1
297	    if start >= len(html) or html[start] != "{":
298	        return None
299	    depth = 0
300	    in_string = False
301	    escape = False
302	    for index in range(start, len(html)):
303	        ch = html[index]
304	        if in_string:
305	            if escape:
306	                escape = False
307	            elif ch == "\\":
308	                escape = True
309	            elif ch == '"':
310	                in_string = False
311	            continue
312	        if ch == '"':
313	            in_string = True
314	        elif ch == "{":
315	            depth += 1
316	        elif ch == "}":
317	            depth -= 1
318	            if depth == 0:
319	                raw = html[start : index + 1]

        created_at=datetime.now(UTC).isoformat(),
    )

def write_manifest_from_snapshot(snapshot: ListingSnapshot, media_root: str | Path) -> Path:
    """Write a version-1 operator manifest next to downloaded photos for re-runs."""
    root = Path(media_root)
    root.mkdir(parents=True, exist_ok=True)
    path = root / "listing.json"
    payload = snapshot.to_dict()
    # Manifest format uses path/label/focal_point only on photos.
    payload["photos"] = [
        {
            "path": photo.relative_path,
            "label": photo.label,
            "focal_point": list(photo.focal_point),
        }
        for photo in snapshot.photos
    ]
    path.write_text(json.dumps(payload, indent=2), encoding="utf-8")
    return path

def fetch_compass_listing(
    listing_url: str,
    *,
    media_root: str | Path,
    http_get: HttpGet | None = None,
    max_photos: int = 20,
    min_photos: int = 5,
) -> ListingSnapshot:
    if not is_compass_listing_url(listing_url):
        raise CompassListingError("Not a Compass listing URL")
    getter = http_get or _default_http_get
    try:
        html = getter(normalize_listing_url(listing_url)).decode("utf-8", "replace")
    except CompassListingError:
        raise
    except Exception as exc:
        raise CompassListingError(f"Failed to fetch Compass listing page: {exc}") from exc
    parsed = parse_compass_listing_html(html, listing_url=listing_url)
    snapshot = materialize_compass_listing(
        parsed,
        media_root=media_root,
        http_get=getter,
        max_photos=max_photos,
        min_photos=min_photos,
    )
    write_manifest_from_snapshot(snapshot, media_root)
    return snapshot

def _default_http_get(url: str) -> bytes:
    request = Request(
        url,
        headers={
            "User-Agent": DEFAULT_USER_AGENT,
            "Accept": "text/html,application/xhtml+xml,application/json,image/avif,image/webp,*/*;q=0.8",
            "Accept-Language": "en-US,en;q=0.9",
        },
    )
    try:
        with urlopen(request, timeout=30) as response:  # noqa: S310 - operator-requested live URL fetch
            return response.read()
    except HTTPError as exc:
        raise CompassListingError(f"HTTP {exc.code} fetching {url}") from exc
    except URLError as exc:
        raise CompassListingError(f"Network error fetching {url}: {exc.reason}") from exc

def _extract_initial_data(html: str) -> dict[str, Any] | None:
    match = _INITIAL_DATA_RE.search(html)
    if not match:
        return None
    start = match.end()
    while start < len(html) and html[start].isspace():
        start += 1
    if start >= len(html) or html[start] != "{":
        return None
    depth = 0
    in_string = False
    escape = False
    for index in range(start, len(html)):
        ch = html[index]
        if in_string:
            if escape:
                escape = False
            elif ch == "\\":
                escape = True
            elif ch == '"':
                in_string = False
            continue
        if ch == '"':
            in_string = True
        elif ch == "{":
            depth += 1
        elif ch == "}":
            depth -= 1
            if depth == 0:
                raw = html[start : index + 1]


319	                raw = html[start : index + 1]
320	                try:
321	                    data = json.loads(raw)
322	                except json.JSONDecodeError as exc:
323	                    raise CompassListingError(f"Compass __INITIAL_DATA__ was invalid JSON: {exc}") from exc
324	                if not isinstance(data, dict):
325	                    raise CompassListingError("Compass __INITIAL_DATA__ was not an object")
326	                return data
327	    return None
328
329
330	def _listing_blob(initial: Mapping[str, Any]) -> Mapping[str, Any] | None:
331	    listing = _dig(initial, "props", "listingRelation", "listing")
332	    return listing if isinstance(listing, Mapping) else None
333
334
335	def _extract_ld_listing(html: str) -> dict[str, Any]:
336	    blocks = re.findall(
337	        r'<script[^>]+type=["\']application/ld\+json["\'][^>]*>(.*?)</script>',
338	        html,
339	        flags=re.I | re.S,
340	    )
341	    for block in blocks:
342	        try:
343	            data = json.loads(block.strip())
344	        except json.JSONDecodeError:
345	            continue
346	        found = _find_ld_listing(data)
347	        if found:
348	            return found
349	    return {}
350
351
352	def _find_ld_listing(node: Any) -> dict[str, Any] | None:
353	    if isinstance(node, dict):
354	        types = node.get("@type")
355	        type_list = types if isinstance(types, list) else [types]
356	        lowered = {str(value).lower() for value in type_list if value}
357	        if lowered & {"realestatelisting", "singlefamilyresidence", "product", "house", "apartment"}:
358	            return node
359	        if "@graph" in node:
360	            found = _find_ld_listing(node["@graph"])
361	            if found:
362	                return found
363	        for value in node.values():
364	            found = _find_ld_listing(value)
365	            if found:
366	                return found
367	    elif isinstance(node, list):
368	        for item in node:
369	            found = _find_ld_listing(item)
370	            if found:
371	                return found
372	    return None
373
374
375	def _collect_photo_urls(
376	    html: str,
377	    listing: Mapping[str, Any] | None,
378	    ld: Mapping[str, Any],
379	) -> tuple[str, ...]:
380	    urls: list[str] = []
381	    if listing is not None:
382	        media = listing.get("media")
383	        if isinstance(media, Mapping):
384	            for key in ("originalUrl", "thumbnailUrl", "url"):
385	                value = media.get(key)
386	                if isinstance(value, str) and value.startswith("http"):
387	                    urls.append(value)
388	        elif isinstance(media, list):
389	            for item in media:
390	                if not isinstance(item, Mapping):
391	                    continue
392	                for key in ("originalUrl", "url", "thumbnailUrl"):
393	                    value = item.get(key)
394	                    if isinstance(value, str) and value.startswith("http"):
395	                        urls.append(value)
396	                        break
397	    image = ld.get("image")
398	    if isinstance(image, list):
399	        for item in image:
400	            if isinstance(item, str) and item.startswith("http"):
401	                urls.append(item)
402	            elif isinstance(item, Mapping):
403	                value = item.get("url") or item.get("contentUrl")
404	                if isinstance(value, str) and value.startswith("http"):
405	                    urls.append(value)
406	    elif isinstance(image, str) and image.startswith("http"):
407	        urls.append(image)
408	    elif isinstance(image, Mapping):
409	        value = image.get("url") or image.get("contentUrl")
410	        if isinstance(value, str) and value.startswith("http"):
411	            urls.append(value)
412
413	    for match in _COMPASS_MEDIA_RE.findall(html):
414	        urls.append(match)
415	    for media_hash in _MEDIA_HASH_RE.findall(html):
416	        urls.append(f"https://www.compass.com/m/{media_hash}/origin.jpg")
417
418	    return _prefer_origin_unique(urls)

                raw = html[start : index + 1]
                try:
                    data = json.loads(raw)
                except json.JSONDecodeError as exc:
                    raise CompassListingError(f"Compass __INITIAL_DATA__ was invalid JSON: {exc}") from exc
                if not isinstance(data, dict):
                    raise CompassListingError("Compass __INITIAL_DATA__ was not an object")
                return data
    return None

def _listing_blob(initial: Mapping[str, Any]) -> Mapping[str, Any] | None:
    listing = _dig(initial, "props", "listingRelation", "listing")
    return listing if isinstance(listing, Mapping) else None

def _extract_ld_listing(html: str) -> dict[str, Any]:
    blocks = re.findall(
        r'<script[^>]+type=["\']application/ld\+json["\'][^>]*>(.*?)</script>',
        html,
        flags=re.I | re.S,
    )
    for block in blocks:
        try:
            data = json.loads(block.strip())
        except json.JSONDecodeError:
            continue
        found = _find_ld_listing(data)
        if found:
            return found
    return {}

def _find_ld_listing(node: Any) -> dict[str, Any] | None:
    if isinstance(node, dict):
        types = node.get("@type")
        type_list = types if isinstance(types, list) else [types]
        lowered = {str(value).lower() for value in type_list if value}
        if lowered & {"realestatelisting", "singlefamilyresidence", "product", "house", "apartment"}:
            return node
        if "@graph" in node:
            found = _find_ld_listing(node["@graph"])
            if found:
                return found
        for value in node.values():
            found = _find_ld_listing(value)
            if found:
                return found
    elif isinstance(node, list):
        for item in node:
            found = _find_ld_listing(item)
            if found:
                return found
    return None

def _collect_photo_urls(
    html: str,
    listing: Mapping[str, Any] | None,
    ld: Mapping[str, Any],
) -> tuple[str, ...]:
    urls: list[str] = []
    if listing is not None:
        media = listing.get("media")
        if isinstance(media, Mapping):
            for key in ("originalUrl", "thumbnailUrl", "url"):
                value = media.get(key)
                if isinstance(value, str) and value.startswith("http"):
                    urls.append(value)
        elif isinstance(media, list):
            for item in media:
                if not isinstance(item, Mapping):
                    continue
                for key in ("originalUrl", "url", "thumbnailUrl"):
                    value = item.get(key)
                    if isinstance(value, str) and value.startswith("http"):
                        urls.append(value)
                        break
    image = ld.get("image")
    if isinstance(image, list):
        for item in image:
            if isinstance(item, str) and item.startswith("http"):
                urls.append(item)
            elif isinstance(item, Mapping):
                value = item.get("url") or item.get("contentUrl")
                if isinstance(value, str) and value.startswith("http"):
                    urls.append(value)
    elif isinstance(image, str) and image.startswith("http"):
        urls.append(image)
    elif isinstance(image, Mapping):
        value = image.get("url") or image.get("contentUrl")
        if isinstance(value, str) and value.startswith("http"):
            urls.append(value)

    for match in _COMPASS_MEDIA_RE.findall(html):
        urls.append(match)
    for media_hash in _MEDIA_HASH_RE.findall(html):
        urls.append(f"https://www.compass.com/m/{media_hash}/origin.jpg")

    return _prefer_origin_unique(urls)


418	    return _prefer_origin_unique(urls)
419
420
421	def _prefer_origin_unique(urls: Sequence[str]) -> tuple[str, ...]:
422	    best: dict[str, str] = {}
423	    order: list[str] = []
424	    for url in urls:
425	        clean = url.split("?")[0].replace("http://", "https://")
426	        if "profile" in clean.lower() or "avatar" in clean.lower():
427	            continue
428	        # agent headshots live under /m/<int>/uuid/
429	        if re.search(r"/m/\d+/[0-9a-f-]{20,}/", clean, re.I):
430	            continue
431	        match = re.search(r"/m/([0-9a-f]{32,})/", clean, re.I)
432	        key = match.group(1).lower() if match else clean.lower()
433	        score = 0
434	        lower = clean.lower()
435	        if "/origin." in lower:
436	            score += 3
437	        if "640x480" in lower:
438	            score += 1
439	        if lower.endswith((".jpg", ".jpeg")):
440	            score += 1
441	        if key not in best:
442	            order.append(key)
443	            best[key] = clean
444	        else:
445	            current = best[key]
446	            current_score = (3 if "/origin." in current.lower() else 0) + (1 if "640x480" in current.lower() else 0)
447	            if score > current_score:
448	                best[key] = clean
449	    return tuple(best[key] for key in order if key in best)
450
451
452	def _floor_size(ld: Mapping[str, Any]) -> Any:
453	    floor = ld.get("floorSize")
454	    if isinstance(floor, list) and floor:
455	        floor = floor[0]
456	    if isinstance(floor, Mapping):
457	        return floor.get("value")
458	    return floor
459
460
461	def _dig(node: Any, *path: Any) -> Any:
462	    current = node
463	    for key in path:
464	        if isinstance(key, int):
465	            if not isinstance(current, list) or key >= len(current):
466	                return None
467	            current = current[key]
468	            continue
469	        if not isinstance(current, Mapping) or key not in current:
470	            return None
471	        current = current[key]
472	    return current
473
474
475	def _first_str(*values: Any) -> str:
476	    for value in values:
477	        if value is None:

    return _prefer_origin_unique(urls)

def _prefer_origin_unique(urls: Sequence[str]) -> tuple[str, ...]:
    best: dict[str, str] = {}
    order: list[str] = []
    for url in urls:
        clean = url.split("?")[0].replace("http://", "https://")
        if "profile" in clean.lower() or "avatar" in clean.lower():
            continue
        # agent headshots live under /m/<int>/uuid/
        if re.search(r"/m/\d+/[0-9a-f-]{20,}/", clean, re.I):
            continue
        match = re.search(r"/m/([0-9a-f]{32,})/", clean, re.I)
        key = match.group(1).lower() if match else clean.lower()
        score = 0
        lower = clean.lower()
        if "/origin." in lower:
            score += 3
        if "640x480" in lower:
            score += 1
        if lower.endswith((".jpg", ".jpeg")):
            score += 1
        if key not in best:
            order.append(key)
            best[key] = clean
        else:
            current = best[key]
            current_score = (3 if "/origin." in current.lower() else 0) + (1 if "640x480" in current.lower() else 0)
            if score > current_score:
                best[key] = clean
    return tuple(best[key] for key in order if key in best)

def _floor_size(ld: Mapping[str, Any]) -> Any:
    floor = ld.get("floorSize")
    if isinstance(floor, list) and floor:
        floor = floor[0]
    if isinstance(floor, Mapping):
        return floor.get("value")
    return floor

def _dig(node: Any, *path: Any) -> Any:
    current = node
    for key in path:
        if isinstance(key, int):
            if not isinstance(current, list) or key >= len(current):
                return None
            current = current[key]
            continue
        if not isinstance(current, Mapping) or key not in current:
            return None
        current = current[key]
    return current

def _first_str(*values: Any) -> str:
    for value in values:
        if value is None:


Now I have a clear picture of the codebase. Let me look at the HTML fixture to understand what photo data structure is currently captured:


1	<!DOCTYPE html>
2
3
4	  <title>1002 Horne Dr, Cedar Park, TX 78613</title>
5
43
44
45

46
81
82
83
84
85

<!DOCTYPE html>

  <title>1002 Horne Dr, Cedar Park, TX 78613</title>


Now I have a clear picture of the codebase. Let me think about whether the "Expand Compass gallery capture further — lazy-loaded/paginated photos beyond initial HTML metadata" task is feasible for a medium tier cycle.

The current `_collect_photo_urls` function extracts photos from:
1. `window.__INITIAL_DATA__` blob - which has the listing's `media` field (but this only has the first photo/thumbnail in a single object)
2. JSON-LD script blocks (`application/ld+json`)
3. Media hash regex patterns in the raw HTML (`_MEDIA_HASH_RE`)
4. Compass media URL regex patterns

The core challenge is that Compass lazy-loads the full photo gallery through JavaScript, so the initial HTML only contains the first few photos. Getting the complete gallery would require either executing JavaScript or hitting separate API endpoints. Looking at the fixture data, the `listing.media` field appears to be a single object with just thumbnail and original URLs, though real Compass pages might structure this as a list with many photo objects, and there could be a dedicated media API endpoint we could leverage.

The current `_collect_photo_urls` function already handles both cases—treating `media` as either a single object or a list—but since we don't have access to a real Compass page to verify the actual structure, I need to think defensively about how the data might be organized in `__INITIAL_DATA__` or whether it's fetched through a separate API call entirely. I can expand the HTML scanning to capture lazy-loaded image attributes like `data-src` and `srcset` that contain Compass media URLs, and also check for photo arrays in other `__INITIAL_DATA__` paths like `listing.listingImages` or `listing.mediaCollection`. This approach is safer since it's purely additive and testable with updated fixtures, so I'll scope it as a medium-tier task. use) for the media collection, handle lazy-loaded images with `data-src` attributes, and check for `<picture>` tags with `<source>` elements. But I'm realizing the real issue might be deeper—if Compass's photo gallery requires JavaScript execution or separate API calls to load the full set, parsing the DOM alone won't capture those photos. I should look for API endpoint patterns in the listing URL structure that might give me direct access to the photo data.

For now, I'll take a practical middle-ground approach: expand the HTML scanning to catch lazy-loaded attributes like `data-src`, `data-original`, and `data-lazy` that contain Compass media URLs, and also scan for photo URLs embedded in `__INITIAL_DATA__` at various possible paths. I'll document the remaining gap—that JavaScript-rendered photos need a headless browser—so it's clear what's still missing. This way I'm making testable progress without needing live API access. I'll take a TDD approach here—write the test first to verify that lazy-loaded images with `data-src` attributes get captured, then implement the regex scanning to extract those URLs from the HTML. I need to check what the existing regex patterns already handle and extend them to cover lazy-load attributes like `data-src`, `data-lazy-src`, and `data-original`. The regex pattern already captures those full Compass URLs in `data-src` attributes, so lazy-loaded images with complete URLs wouldn't be missed. The real problem is when the gallery data loads dynamically via JavaScript after the page renders—those URLs never appear in the initial HTML. Beyond that, I might be missing photos with different URL patterns entirely, like bare hashes without the `/origin.jpg` suffix, or photos served from a different CDN, plus photos nested under different keys in the `__INITIAL_DATA__` object beyond just `listing.media`. could be nested under different paths like `initial.props.initialState.listing.media` or stored in a separate `mediaCollection` field. For a practical bounded approach, I should focus on the most likely variations rather than trying to handle every possible structure. Adding more photo paths...

Now I'm thinking about how to test this properly—I could either update the fixture HTML or write a unit test directly against `_collect_photo_urls`, but testing through `parse_compass_listing_html` with a custom HTML string is cleaner since it exercises the full flow. I'm also considering whether to test the private function directly or keep it integration-focused.

Writing a test that constructs HTML with photos in the listing.photos array... Now I'm laying out the test workflow: write the failing test first, confirm it fails, then implement the fix in the provider code, verify it passes, and finally update the tracking documents. I'll start by adding the new test cases after the existing photo tests.


All P1/P2 projects are either blocked on James (StageHand, nanoclaw deploy, AgenticLever goal confirm, RoadSense launch questions) or already queued for merge sign-off (qip-qien). The one actionable autonomous item in the punch list is the StageHand gallery expansion — looking at compass.py, the existing scan misses photo arrays stored under alternative keys in `__INITIAL_DATA__` (`photos`, `mediaCollection`, etc.). I can add that coverage with TDD for medium tier.


120	    def test_parse_rejects_empty_html(self) -> None:
121	        with self.assertRaisesRegex(CompassListingError, "empty"):
122	            parse_compass_listing_html("   ", listing_url=COMPASS_URL)
123
124	    # --- min_photos guard ---
125
126	    def _make_parsed(self, photo_urls: tuple[str, ...]) -> CompassListingParse:
127	        return CompassListingParse(
128	            source_url=COMPASS_URL,
129	            address="1002 Horne Dr",
130	            city="Cedar Park",
131	            state="TX",
132	            postal_code="78613",
133	            price=692500,
134	            beds=5,
135	            baths=3,
136	            square_feet=2684,
137	            description="Demo listing.",
138	            photo_urls=photo_urls,
139	            attribution="Demo intake.",
140	        )
141
142	    def test_materialize_rejects_below_min_photos(self) -> None:
143	        """Fail closed: fewer than min_photos downloaded → CompassListingError."""
144	        photos = tuple(
145	            f"https://www.compass.com/m/{'a' * 64}/{i}/origin.jpg" for i in range(2)
146	        )
147	        parsed = self._make_parsed(photos)
148	        dummy = b"P3\n1 1\n255\n1 2 3\n"
149
150	        with tempfile.TemporaryDirectory() as tmp:
151	            with self.assertRaisesRegex(CompassListingError, "at least 5 photos"):
152	                materialize_compass_listing(
153	                    parsed,
154	                    media_root=tmp,
155	                    http_get=lambda url: dummy,
156	                    min_photos=5,
157	                )
158
159	    def test_materialize_accepts_at_min_photo_threshold(self) -> None:
160	        """Exactly min_photos downloaded → succeeds."""
161	        photos = tuple(
162	            f"https://www.compass.com/m/{'a' * 62}{i:02d}/origin.jpg" for i in range(5)
163	        )
164	        parsed = self._make_parsed(photos)
165	        dummy = b"P3\n1 1\n255\n1 2 3\n"
166
167	        with tempfile.TemporaryDirectory() as tmp:
168	            snapshot = materialize_compass_listing(
169	                parsed,
170	                media_root=tmp,
171	                http_get=lambda url: dummy,
172	                min_photos=5,
173	            )
174	        self.assertEqual(len(snapshot.photos), 5)
175
176	    def test_materialize_default_min_photos_is_five(self) -> None:
177	        """Default min_photos=5; 2 photos raises without explicit kwarg."""
178	        photos = tuple(
179	            f"https://www.compass.com/m/{'b' * 64}/{i}/origin.jpg" for i in range(2)
180	        )
181	        parsed = self._make_parsed(photos)
182	        dummy = b"P3\n1 1\n255\n5 6 7\n"
183
184	        with tempfile.TemporaryDirectory() as tmp:
185	            with self.assertRaisesRegex(CompassListingError, "at least 5 photos"):
186	                materialize_compass_listing(
187	                    parsed,
188	                    media_root=tmp,
189	                    http_get=lambda url: dummy,
190	                )
191
192
193	if __name__ == "__main__":
194	    unittest.main()

    def test_parse_rejects_empty_html(self) -> None:
        with self.assertRaisesRegex(CompassListingError, "empty"):
            parse_compass_listing_html("   ", listing_url=COMPASS_URL)

    # --- min_photos guard ---

    def _make_parsed(self, photo_urls: tuple[str, ...]) -> CompassListingParse:
        return CompassListingParse(
            source_url=COMPASS_URL,
            address="1002 Horne Dr",
            city="Cedar Park",
            state="TX",
            postal_code="78613",
            price=692500,
            beds=5,
            baths=3,
            square_feet=2684,
            description="Demo listing.",
            photo_urls=photo_urls,
            attribution="Demo intake.",
        )

    def test_materialize_rejects_below_min_photos(self) -> None:
        """Fail closed: fewer than min_photos downloaded → CompassListingError."""
        photos = tuple(
            f"https://www.compass.com/m/{'a' * 64}/{i}/origin.jpg" for i in range(2)
        )
        parsed = self._make_parsed(photos)
        dummy = b"P3\n1 1\n255\n1 2 3\n"

        with tempfile.TemporaryDirectory() as tmp:
            with self.assertRaisesRegex(CompassListingError, "at least 5 photos"):
                materialize_compass_listing(
                    parsed,
                    media_root=tmp,
                    http_get=lambda url: dummy,
                    min_photos=5,
                )

    def test_materialize_accepts_at_min_photo_threshold(self) -> None:
        """Exactly min_photos downloaded → succeeds."""
        photos = tuple(
            f"https://www.compass.com/m/{'a' * 62}{i:02d}/origin.jpg" for i in range(5)
        )
        parsed = self._make_parsed(photos)
        dummy = b"P3\n1 1\n255\n1 2 3\n"

        with tempfile.TemporaryDirectory() as tmp:
            snapshot = materialize_compass_listing(
                parsed,
                media_root=tmp,
                http_get=lambda url: dummy,
                min_photos=5,
            )
        self.assertEqual(len(snapshot.photos), 5)

    def test_materialize_default_min_photos_is_five(self) -> None:
        """Default min_photos=5; 2 photos raises without explicit kwarg."""
        photos = tuple(
            f"https://www.compass.com/m/{'b' * 64}/{i}/origin.jpg" for i in range(2)
        )
        parsed = self._make_parsed(photos)
        dummy = b"P3\n1 1\n255\n5 6 7\n"

        with tempfile.TemporaryDirectory() as tmp:
            with self.assertRaisesRegex(CompassListingError, "at least 5 photos"):
                materialize_compass_listing(
                    parsed,
                    media_root=tmp,
                    http_get=lambda url: dummy,
                )

if __name__ == "__main__":
    unittest.main()


Now I'll add the RED tests for photo array expansion. I'll add them after the `test_materialize_default_min_photos_is_five` test and before the `if __name__ == "__main__":` line.

The tests will be:
1. Test that `listing.photos` array in `__INITIAL_DATA__` is extracted
2. Test that `listing.mediaCollection` array in `__INITIAL_DATA__` is extracted (alternative key)
3. Test that `listing.gallery` array is extracted (another alternative)

Let me write a utility function to build minimal listing HTML to avoid repetition.
Actually, I'll keep it focused with just two tests: one for `listing.photos` and another for `listing.mediaCollection`. These target the actual gap where `_collect_photo_urls` only checks `listing.media` but misses these alternative array keys.

Writing the tests now...


Now I'll add the RED tests for alternative photo array keys, then implement the fix.


Edit

if __name__ == "__main__":
    unittest.main()

# --- alternative photo array keys in __INITIAL_DATA__ ---

    _MINIMAL_LISTING_TMPL = """\
<!DOCTYPE html>
"""

    def _html_with(self, extra_json: str) -> str:
        return self._MINIMAL_LISTING_TMPL.format(extra=extra_json)

    def test_parse_captures_photos_from_listing_photos_key(self) -> None:
        """listing.photos array in __INITIAL_DATA__ is extracted as photo URLs."""
        PHOTO_C = "https://www.compass.com/m/" + "c" * 64 + "/origin.jpg"
        PHOTO_D = "https://www.compass.com/m/" + "d" * 64 + "/origin.jpg"
        html = self._html_with(
            f'"photos":[{{"originalUrl":"{PHOTO_C}"}},{{"originalUrl":"{PHOTO_D}"}}]'
        )
        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)
        self.assertIn(PHOTO_C, parsed.photo_urls)
        self.assertIn(PHOTO_D, parsed.photo_urls)

    def test_parse_captures_photos_from_media_collection_key(self) -> None:
        """listing.mediaCollection array in __INITIAL_DATA__ is extracted as photo URLs."""
        PHOTO_E = "https://www.compass.com/m/" + "e" * 64 + "/origin.jpg"
        PHOTO_F = "https://www.compass.com/m/" + "f" * 64 + "/origin.jpg"
        html = self._html_with(
            f'"mediaCollection":[{{"url":"{PHOTO_E}"}},{{"originalUrl":"{PHOTO_F}"}}]'
        )
        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)
        self.assertIn(PHOTO_E, parsed.photo_urls)
        self.assertIn(PHOTO_F, parsed.photo_urls)

if __name__ == "__main__":
    unittest.main()


The file /workspace/extra/projects/stagehand/tests/test_compass_provider.py has been updated successfully. (file state is current in your context — no need to Read it back)

# --- alternative photo array keys in __INITIAL_DATA__ ---

    _MINIMAL_LISTING_TMPL = """\
<!DOCTYPE html>
"""

    def _html_with(self, extra_json: str) -> str:
        return self._MINIMAL_LISTING_TMPL.format(extra=extra_json)

    def test_parse_captures_photos_from_listing_photos_key(self) -> None:
        """listing.photos array in __INITIAL_DATA__ is extracted as photo URLs."""
        PHOTO_C = "https://www.compass.com/m/" + "c" * 64 + "/origin.jpg"
        PHOTO_D = "https://www.compass.com/m/" + "d" * 64 + "/origin.jpg"
        html = self._html_with(
            f'"photos":[{{"originalUrl":"{PHOTO_C}"}},{{"originalUrl":"{PHOTO_D}"}}]'
        )
        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)
        self.assertIn(PHOTO_C, parsed.photo_urls)
        self.assertIn(PHOTO_D, parsed.photo_urls)

    def test_parse_captures_photos_from_media_collection_key(self) -> None:
        """listing.mediaCollection array in __INITIAL_DATA__ is extracted as photo URLs."""
        PHOTO_E = "https://www.compass.com/m/" + "e" * 64 + "/origin.jpg"
        PHOTO_F = "https://www.compass.com/m/" + "f" * 64 + "/origin.jpg"
        html = self._html_with(
            f'"mediaCollection":[{{"url":"{PHOTO_E}"}},{{"originalUrl":"{PHOTO_F}"}}]'
        )
        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)
        self.assertIn(PHOTO_E, parsed.photo_urls)
        self.assertIn(PHOTO_F, parsed.photo_urls)

if __name__ == "__main__":
    unittest.main()

from __future__ import annotations

import tempfile
import unittest
from pathlib import Path
from unittest.mock import MagicMock

from core.listings.manifest import ListingManifestError, SourcePolicy, load_listing_source, source_policy_for
from core.listings.providers.compass import (
    CompassListingError,
    CompassListingParse,
    fetch_compass_listing,
    is_compass_listing_url,
    materialize_compass_listing,
    parse_compass_listing_html,
)
from core.listings.providers.registry import build_authorized_fetcher

FIXTURE_HTML = Path(__file__).parent / "fixtures" / "compass" / "sample_listing.html"
COMPASS_URL = "https://www.compass.com/homedetails/1002-Horne-Dr-Cedar-Park-TX-78613/1877020858229275065_lid/"
PHOTO_A = "https://www.compass.com/m/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/origin.jpg"
PHOTO_B = "https://www.compass.com/m/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb/origin.jpg"

class CompassProviderTests(unittest.TestCase):
    def test_source_policy_allows_compass_adapter_path(self) -> None:
        self.assertTrue(is_compass_listing_url(COMPASS_URL))
        self.assertEqual(source_policy_for(COMPASS_URL), SourcePolicy.AUTHORIZED_ADAPTER)

    def test_parse_fixture_html_extracts_facts_and_deduped_photos(self) -> None:
        html = FIXTURE_HTML.read_text(encoding="utf-8")
        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)

        self.assertEqual(parsed.address, "1002 Horne Dr")
        self.assertEqual(parsed.city, "Cedar Park")
        self.assertEqual(parsed.state, "TX")
        self.assertEqual(parsed.postal_code, "78613")
        self.assertEqual(parsed.price, 692500)
        self.assertEqual(parsed.beds, 5)
        self.assertEqual(parsed.baths, 3)
        self.assertEqual(parsed.square_feet, 2684)
        self.assertIn("Cypress Canyon", parsed.description)
        self.assertEqual(parsed.agent_name, "Richard Fowler")
        self.assertEqual(parsed.photo_urls, (PHOTO_A, PHOTO_B))
        self.assertIn("pilot review only", parsed.attribution)

    def test_fetch_materializes_photos_and_manifest(self) -> None:
        html = FIXTURE_HTML.read_text(encoding="utf-8").encode("utf-8")
        photo_bytes = {
            PHOTO_A: b"P3\n1 1\n255\n220 210 190\n",
            PHOTO_B: b"P3\n1 1\n255\n10 20 30\n",
        }

        def http_get(url: str) -> bytes:
            if url.rstrip("/") == COMPASS_URL.rstrip("/") or url.startswith(COMPASS_URL):
                return html
            if url in photo_bytes:
                return photo_bytes[url]
            # origin rewrite variants for the second media hash may be requested as origin.jpg
            if url.endswith("/origin.jpg"):
                key = url
                for candidate, payload in photo_bytes.items():
                    if candidate.split("/")[-2] in url:
                        return payload
            raise AssertionError(f"unexpected URL {url}")

        with tempfile.TemporaryDirectory() as tmp:
            root = Path(tmp)
            # Fixture has only 2 photos; min_photos=2 bypasses the guard (tested separately).
            snapshot = fetch_compass_listing(COMPASS_URL, media_root=root, http_get=http_get, min_photos=2)

            self.assertEqual(snapshot.source_provider, "compass_public_demo")
            self.assertTrue(snapshot.rights_confirmed)
            self.assertEqual(snapshot.address, "1002 Horne Dr")
            self.assertEqual(len(snapshot.photos), 2)
            self.assertTrue((root / "photos" / "photo_01.jpg").exists())
            self.assertTrue((root / "photos" / "photo_02.jpg").exists())
            self.assertTrue((root / "listing.json").exists())
            self.assertEqual(len(snapshot.photos[0].sha256), 64)

    def test_registry_requires_demo_or_rights_flag(self) -> None:
        with tempfile.TemporaryDirectory() as tmp:
            root = Path(tmp)
            blocked = build_authorized_fetcher(media_root=root, demo=False, rights_confirmed=False)
            with self.assertRaisesRegex(ListingManifestError, "demo-only"):
                blocked(COMPASS_URL)

    def test_registry_uses_compass_adapter_when_demo_enabled(self) -> None:
        html = FIXTURE_HTML.read_text(encoding="utf-8").encode("utf-8")
        photo = b"P3\n1 1\n255\n1 2 3\n"

        def http_get(url: str) -> bytes:
            if "homedetails" in url:
                return html
            return photo

        with tempfile.TemporaryDirectory() as tmp:
            root = Path(tmp)
            # Fixture has only 2 photos; min_photos=2 bypasses the guard (tested separately).
            fetcher = build_authorized_fetcher(
                media_root=root,
                demo=True,
                rights_confirmed=False,
                http_get=http_get,
                min_photos=2,
            )
            snapshot = load_listing_source(COMPASS_URL, fetcher=fetcher)
            self.assertEqual(snapshot.source_provider, "compass_public_demo")
            self.assertEqual(snapshot.city, "Cedar Park")
            self.assertGreaterEqual(len(snapshot.photos), 1)

    def test_zillow_still_never_calls_fetcher(self) -> None:
        fetcher = MagicMock()
        zillow = "https://www.zillow.com/homedetails/example/1_zpid/"
        self.assertEqual(source_policy_for(zillow), SourcePolicy.MANIFEST_ONLY)
        with self.assertRaisesRegex(ListingManifestError, "authorized manifest"):
            load_listing_source(zillow, fetcher=fetcher)
        fetcher.assert_not_called()

    def test_parse_rejects_empty_html(self) -> None:
        with self.assertRaisesRegex(CompassListingError, "empty"):
            parse_compass_listing_html("   ", listing_url=COMPASS_URL)

    # --- min_photos guard ---

    def _make_parsed(self, photo_urls: tuple[str, ...]) -> CompassListingParse:
        return CompassListingParse(
            source_url=COMPASS_URL,
            address="1002 Horne Dr",
            city="Cedar Park",
            state="TX",
            postal_code="78613",
            price=692500,
            beds=5,
            baths=3,
            square_feet=2684,
            description="Demo listing.",
            photo_urls=photo_urls,
            attribution="Demo intake.",
        )

    def test_materialize_rejects_below_min_photos(self) -> None:
        """Fail closed: fewer than min_photos downloaded → CompassListingError."""
        photos = tuple(
            f"https://www.compass.com/m/{'a' * 64}/{i}/origin.jpg" for i in range(2)
        )
        parsed = self._make_parsed(photos)
        dummy = b"P3\n1 1\n255\n1 2 3\n"

        with tempfile.TemporaryDirectory() as tmp:
            with self.assertRaisesRegex(CompassListingError, "at least 5 photos"):
                materialize_compass_listing(
                    parsed,
                    media_root=tmp,
                    http_get=lambda url: dummy,
                    min_photos=5,
                )

    def test_materialize_accepts_at_min_photo_threshold(self) -> None:
        """Exactly min_photos downloaded → succeeds."""
        photos = tuple(
            f"https://www.compass.com/m/{'a' * 62}{i:02d}/origin.jpg" for i in range(5)
        )
        parsed = self._make_parsed(photos)
        dummy = b"P3\n1 1\n255\n1 2 3\n"

        with tempfile.TemporaryDirectory() as tmp:
            snapshot = materialize_compass_listing(
                parsed,
                media_root=tmp,
                http_get=lambda url: dummy,
                min_photos=5,
            )
        self.assertEqual(len(snapshot.photos), 5)

    def test_materialize_default_min_photos_is_five(self) -> None:
        """Default min_photos=5; 2 photos raises without explicit kwarg."""
        photos = tuple(
            f"https://www.compass.com/m/{'b' * 64}/{i}/origin.jpg" for i in range(2)
        )
        parsed = self._make_parsed(photos)
        dummy = b"P3\n1 1\n255\n5 6 7\n"

        with tempfile.TemporaryDirectory() as tmp:
            with self.assertRaisesRegex(CompassListingError, "at least 5 photos"):
                materialize_compass_listing(
                    parsed,
                    media_root=tmp,
                    http_get=lambda url: dummy,
                )

if __name__ == "__main__":
    unittest.main()

+    # --- alternative photo array keys in __INITIAL_DATA__ ---

+        "location": {{"streetAddress":"1 Test St","city":"Austin","state":"TX","zipCode":"78701"}},

+        "size": {{"bedrooms":3,"bathrooms":2,"squareFeet":1500}},

+    def _html_with(self, extra_json: str) -> str:

+        return self._MINIMAL_LISTING_TMPL.format(extra=extra_json)

+    def test_parse_captures_photos_from_listing_photos_key(self) -> None:

+        """listing.photos array in __INITIAL_DATA__ is extracted as photo URLs."""

+        PHOTO_C = "https://www.compass.com/m/" + "c" * 64 + "/origin.jpg"

+        PHOTO_D = "https://www.compass.com/m/" + "d" * 64 + "/origin.jpg"

+            f'"photos":[{{"originalUrl":"{PHOTO_C}"}},{{"originalUrl":"{PHOTO_D}"}}]'

+        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)

+        self.assertIn(PHOTO_C, parsed.photo_urls)

+        self.assertIn(PHOTO_D, parsed.photo_urls)

+    def test_parse_captures_photos_from_media_collection_key(self) -> None:

+        """listing.mediaCollection array in __INITIAL_DATA__ is extracted as photo URLs."""

+        PHOTO_E = "https://www.compass.com/m/" + "e" * 64 + "/origin.jpg"

+        PHOTO_F = "https://www.compass.com/m/" + "f" * 64 + "/origin.jpg"

+            f'"mediaCollection":[{{"url":"{PHOTO_E}"}},{{"originalUrl":"{PHOTO_F}"}}]'

+        self.assertIn(PHOTO_E, parsed.photo_urls)

+        self.assertIn(PHOTO_F, parsed.photo_urls)


Bash

cd /workspace/extra/projects/stagehand && PYTHONPATH=. python3 -m unittest tests.test_compass_provider.CompassProviderTests.test_parse_captures_photos_from_listing_photos_key tests.test_compass_provider.CompassProviderTests.test_parse_captures_photos_from_media_collection_key -v 2>&1

Confirm new tests are RED


test_parse_captures_photos_from_listing_photos_key (tests.test_compass_provider.CompassProviderTests.test_parse_captures_photos_from_listing_photos_key)
listing.photos array in __INITIAL_DATA__ is extracted as photo URLs. ... ok
test_parse_captures_photos_from_media_collection_key (tests.test_compass_provider.CompassProviderTests.test_parse_captures_photos_from_media_collection_key)
listing.mediaCollection array in __INITIAL_DATA__ is extracted as photo URLs. ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK


Hmm, the tests are already GREEN! This means the current implementation already captures these URLs. That's because the `_COMPASS_MEDIA_RE` or `_MEDIA_HASH_RE` regexes scan the entire raw HTML string and find all Compass media URLs - including URLs inside the JSON blob.

Let me verify: the `_collect_photo_urls` function does:
```python
for match in _COMPASS_MEDIA_RE.findall(html):
    urls.append(match)
for media_hash in _MEDIA_HASH_RE.findall(html):
    urls.append(f"https://www.compass.com/m/{media_hash}/origin.jpg")
```

The regex pattern matches the full Compass media URL format with the domain, media hash in hex characters, and the image format suffix, so it successfully catches everything in the HTML including those embedded in JSON.

But the real limitation is different—Compass loads additional photos through JavaScript API calls after the page initially loads, and those URLs never appear in the static HTML. The current approach handles everything in the initial page source, but can't capture photos that require JavaScript execution to fetch.

To actually improve this, I'd need to either find a Compass API endpoint that returns the full gallery based on patterns in the initial HTML, or make additional HTTP requests to fetch more photos beyond what's statically available. I'm checking the regex pattern for Compass photo URLs to see what variations it might miss—like different thumbnail sizes such as 1024x768 instead of just the 640x480 and origin formats it currently handles.

The hash-matching regex requires at least 40 characters, which works for the test fixture's 64-character hashes, but I'm wondering if Compass might use shorter hashes or UUID-style hashes with dashes. There's also a separate pattern that looks for 32+ character hashes and applies an agent headshot filter, so I need to understand how these different patterns interact. The regex pattern for Compass media URLs actually does capture UUIDs with the hex and dash characters, so property photos with UUID-format hashes would be matched. I'm going in circles here—the regexes seem solid for extracting in-HTML URLs, so I should focus on what else might be missing.

The real gap is photos that aren't embedded in the HTML at all—those would require either a secondary API call to a Compass photos endpoint or headless browser rendering. Without knowing Compass's actual API format, I can't implement option one correctly, and option two would require Playwright or Puppeteer, which is a significant architectural change. Since the pagination issue stems from JavaScript-rendered content that I can't test without a live page or realistic fixture data, maybe I should try a different approach: make an educated guess about Compass's gallery API based on the listing ID structure and attempt a secondary fetch.

Looking at common real estate platform patterns, they typically expose endpoints like `/api/v1/listing/{listingId}/media` or GraphQL queries for listing photos. I could add a stub function for fetching from the gallery API with clear documentation about what needs investigation, wire it into the main listing fetch with proper error handling, or alternatively focus on improving the thumbnail-to-origin URL upgrade logic to catch non-standard image sizes that might otherwise be missed.

The regex pattern currently only matches `origin` or `640x480` variants, so photos served at other dimensions like `1200x900` would slip through. However, the hash extraction regex is more flexible—it pulls out any `/m/HASH/` pattern and reconstructs an `/origin.jpg` URL, so even if the HTML only contains a non-standard size, the hash gets captured and upgraded to the origin version anyway.

But there's a catch: the hash extraction requires at least 40 characters, so if Compass uses shorter hashes like 32-character UUIDs, those wouldn't be caught by the main hash regex. The dedup function uses a 32+ character threshold, which means hashes between 32 and 39 characters would fall into a gap where neither pattern fully captures them.

The fix is straightforward: lower the hash regex threshold from 40 to 32 characters to handle UUID-length hashes, which would let us reconstruct `/origin.jpg` for more URL formats. Let me trace through a concrete example to verify this works—if a Compass URL with a UUID-format hash ends in something like `thumbnail.jpg` instead of `origin` or `640x480`, the current regex wouldn't match it as a photo, but lowering the threshold would catch it.

Writing test case... I see the real issue now—neither regex would catch a 32-character hash, so this is genuinely a gap in coverage. I could either lower the `_MEDIA_HASH_RE` threshold to 32 characters, or create a new broader regex that matches any `/m/HASH/` pattern in image sources and constructs the origin URL from it. The second approach seems cleaner since it handles the pattern more explicitly. Updating the regex pattern to catch 32-character hashes would expand coverage to UUID-format hashes, and the existing origin.jpg construction and headshot filtering would handle these new matches appropriately. So the `thumbnailUrl` value gets added to the urls list through the Mapping branch, and then `_prefer_origin_unique` searches for the hash pattern with a 32+ character threshold, which would match the 32-char hash and extract it for constructing the origin URL.

The scoring logic checks for "/origin." (adding 3 points), "640x480" (adding 1 point), and file extensions like ".jpg" (adding 1 point). For the `https://www.compass.com/m/{hash32}/1200x900.jpg` URL, none of the origin or resolution bonuses apply, so it gets a score of 1 from the file extension alone. This means the 1200x900 version gets stored instead of the preferred origin version.

The real issue is that `_MEDIA_HASH_RE` requires at least 40 characters to match a hash, so it can't construct an `/origin.jpg` candidate for 32-character hashes found in the HTML. If I lower that threshold to 32+ characters, the regex would generate origin URL candidates for all hashes, and then the scoring system would correctly prefer those over the thumbnail sizes. The URL does get added through `thumbnailUrl` with a score of 1, so the 1200x900 variant wins instead of the origin. This is actually a bug—I need to lower the hash pattern threshold from 40 characters to 32. But I'm checking whether this could cause false positives with agent headshot URLs, which use a different pattern with numeric IDs and UUIDs. that the UUID format without dashes (32 hex characters) would actually match the regex pattern, but this would only be a problem if agent photos use that exact format without a numeric prefix. Since the agent headshot filter relies on detecting a numeric prefix pattern, it might not catch these cases anyway, so I need to think through whether agent profile photos even appear in the initial HTML or if they're loaded separately.

The existing code already filters out profile and avatar keywords, which handles most agent photos. Lowering the regex threshold to 32 characters would capture more property photos from non-standard URL formats with minimal risk of false positives. I should write a test for this and implement the change.

Actually, looking more carefully at how the media hash regex works—it searches the entire HTML for 32+ character hex strings and constructs `/origin.jpg` URLs from them. Agent headshots use UUIDs with dashes, which wouldn't match a pure hex pattern anyway, so lowering the threshold is safe.

Now I'm writing a test that puts a 32-character hash into the HTML as a non-standard image size variant (not the origin or standard 640x480 dimensions) so the normal media parsing doesn't already capture it with an `/origin.jpg` URL.

Writing test case... the regex pattern needs to match 32-character hashes instead of requiring 40 or more, which would allow the code to properly construct the origin image URL from the thumbnail hash and pass the test. The test would fail before the fix and pass after, since the regex change allows the 32-character hash to be matched and `origin.jpg` to be constructed and preferred over `1200x900.jpg`. The `thumbnailUrl` from `listing.media` gets captured in the URLs list, and the regex scan of the HTML will now pick up both the thumbnail and the constructed origin URL. the hash matches the regex pattern since 'a' is a valid hex digit and we have exactly 32 characters, so the origin URL with its higher score would be preferred over the 1200x900 version. The existing tests already cover the 64-character hash case, so I just need to add the two new test cases for this scenario. Let me reconsider what I'm testing here. The 32-character hash upgrade is the real gap I need to fix. I'm also considering whether there are other gaps in the photo collection logic—like photos that only appear in JavaScript API responses or alternative `__INITIAL_DATA__` keys—but those are either out of scope or already handled by the raw HTML scan. I'll focus on lowering the `_MEDIA_HASH_RE` threshold to capture 32-character hashes, which should unlock more photo URLs from the HTML.

Now I'm ready to write the actual test case for this scenario.


The tests passed immediately — the raw HTML regex scan already catches URLs in JSON blobs. Those tests confirmed existing behavior but found no gap. Let me find the actual gap: `_MEDIA_HASH_RE` requires 40+ chars, so 32-char UUID-style hashes in non-standard thumbnail sizes won't be upgraded to `/origin.jpg`.


Edit

    def test_parse_captures_photos_from_listing_photos_key(self) -> None:
        """listing.photos array in __INITIAL_DATA__ is extracted as photo URLs."""
        PHOTO_C = "https://www.compass.com/m/" + "c" * 64 + "/origin.jpg"
        PHOTO_D = "https://www.compass.com/m/" + "d" * 64 + "/origin.jpg"
        html = self._html_with(
            f'"photos":[{{"originalUrl":"{PHOTO_C}"}},{{"originalUrl":"{PHOTO_D}"}}]'
        )
        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)
        self.assertIn(PHOTO_C, parsed.photo_urls)
        self.assertIn(PHOTO_D, parsed.photo_urls)

    def test_parse_captures_photos_from_media_collection_key(self) -> None:
        """listing.mediaCollection array in __INITIAL_DATA__ is extracted as photo URLs."""
        PHOTO_E = "https://www.compass.com/m/" + "e" * 64 + "/origin.jpg"
        PHOTO_F = "https://www.compass.com/m/" + "f" * 64 + "/origin.jpg"
        html = self._html_with(
            f'"mediaCollection":[{{"url":"{PHOTO_E}"}},{{"originalUrl":"{PHOTO_F}"}}]'
        )
        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)
        self.assertIn(PHOTO_E, parsed.photo_urls)
        self.assertIn(PHOTO_F, parsed.photo_urls)

    def test_parse_captures_photos_from_listing_photos_key(self) -> None:
        """listing.photos array in __INITIAL_DATA__ is extracted as photo URLs (via raw HTML scan)."""
        PHOTO_C = "https://www.compass.com/m/" + "c" * 64 + "/origin.jpg"
        PHOTO_D = "https://www.compass.com/m/" + "d" * 64 + "/origin.jpg"
        html = self._html_with(
            f'"photos":[{{"originalUrl":"{PHOTO_C}"}},{{"originalUrl":"{PHOTO_D}"}}]'
        )
        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)
        self.assertIn(PHOTO_C, parsed.photo_urls)
        self.assertIn(PHOTO_D, parsed.photo_urls)

    def test_parse_captures_photos_from_media_collection_key(self) -> None:
        """listing.mediaCollection array in __INITIAL_DATA__ is extracted as photo URLs (via raw HTML scan)."""
        PHOTO_E = "https://www.compass.com/m/" + "e" * 64 + "/origin.jpg"
        PHOTO_F = "https://www.compass.com/m/" + "f" * 64 + "/origin.jpg"
        html = self._html_with(
            f'"mediaCollection":[{{"url":"{PHOTO_E}"}},{{"originalUrl":"{PHOTO_F}"}}]'
        )
        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)
        self.assertIn(PHOTO_E, parsed.photo_urls)
        self.assertIn(PHOTO_F, parsed.photo_urls)

    def test_parse_upgrades_32char_hash_thumbnail_to_origin(self) -> None:
        """32-char hex hashes (UUID without dashes) in non-standard sizes are upgraded to /origin.jpg.

        _MEDIA_HASH_RE used to require 40+ chars, which silently kept thumbnails at non-origin
        quality when Compass uses shorter-format hashes.  The fix lowers the threshold to 32.
        """
        hash32 = "b" * 32  # exactly 32 hex chars — shorter than former 40+ threshold
        thumbnail_url = f"https://www.compass.com/m/{hash32}/1200x900.jpg"
        expected_origin = f"https://www.compass.com/m/{hash32}/origin.jpg"
        html = self._html_with(f'"media":{{"thumbnailUrl":"{thumbnail_url}"}}')
        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)
        self.assertIn(expected_origin, parsed.photo_urls)
        self.assertNotIn(thumbnail_url, parsed.photo_urls)


    def test_parse_captures_photos_from_listing_photos_key(self) -> None:
        """listing.photos array in __INITIAL_DATA__ is extracted as photo URLs."""
        PHOTO_C = "https://www.compass.com/m/" + "c" * 64 + "/origin.jpg"
        PHOTO_D = "https://www.compass.com/m/" + "d" * 64 + "/origin.jpg"
        html = self._html_with(
            f'"photos":[{{"originalUrl":"{PHOTO_C}"}},{{"originalUrl":"{PHOTO_D}"}}]'
        )
        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)
        self.assertIn(PHOTO_C, parsed.photo_urls)
        self.assertIn(PHOTO_D, parsed.photo_urls)

    def test_parse_captures_photos_from_media_collection_key(self) -> None:
        """listing.mediaCollection array in __INITIAL_DATA__ is extracted as photo URLs."""
        PHOTO_E = "https://www.compass.com/m/" + "e" * 64 + "/origin.jpg"
        PHOTO_F = "https://www.compass.com/m/" + "f" * 64 + "/origin.jpg"
        html = self._html_with(
            f'"mediaCollection":[{{"url":"{PHOTO_E}"}},{{"originalUrl":"{PHOTO_F}"}}]'
        )
        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)
        self.assertIn(PHOTO_E, parsed.photo_urls)
        self.assertIn(PHOTO_F, parsed.photo_urls)

    def test_parse_captures_photos_from_listing_photos_key(self) -> None:
        """listing.photos array in __INITIAL_DATA__ is extracted as photo URLs (via raw HTML scan)."""
        PHOTO_C = "https://www.compass.com/m/" + "c" * 64 + "/origin.jpg"
        PHOTO_D = "https://www.compass.com/m/" + "d" * 64 + "/origin.jpg"
        html = self._html_with(
            f'"photos":[{{"originalUrl":"{PHOTO_C}"}},{{"originalUrl":"{PHOTO_D}"}}]'
        )
        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)
        self.assertIn(PHOTO_C, parsed.photo_urls)
        self.assertIn(PHOTO_D, parsed.photo_urls)

    def test_parse_captures_photos_from_media_collection_key(self) -> None:
        """listing.mediaCollection array in __INITIAL_DATA__ is extracted as photo URLs (via raw HTML scan)."""
        PHOTO_E = "https://www.compass.com/m/" + "e" * 64 + "/origin.jpg"
        PHOTO_F = "https://www.compass.com/m/" + "f" * 64 + "/origin.jpg"
        html = self._html_with(
            f'"mediaCollection":[{{"url":"{PHOTO_E}"}},{{"originalUrl":"{PHOTO_F}"}}]'
        )
        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)
        self.assertIn(PHOTO_E, parsed.photo_urls)
        self.assertIn(PHOTO_F, parsed.photo_urls)

    def test_parse_upgrades_32char_hash_thumbnail_to_origin(self) -> None:
        """32-char hex hashes (UUID without dashes) in non-standard sizes are upgraded to /origin.jpg.

        _MEDIA_HASH_RE used to require 40+ chars, which silently kept thumbnails at non-origin
        quality when Compass uses shorter-format hashes.  The fix lowers the threshold to 32.
        """
        hash32 = "b" * 32  # exactly 32 hex chars — shorter than former 40+ threshold
        thumbnail_url = f"https://www.compass.com/m/{hash32}/1200x900.jpg"
        expected_origin = f"https://www.compass.com/m/{hash32}/origin.jpg"
        html = self._html_with(f'"media":{{"thumbnailUrl":"{thumbnail_url}"}}')
        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)
        self.assertIn(expected_origin, parsed.photo_urls)
        self.assertNotIn(thumbnail_url, parsed.photo_urls)

-        """listing.photos array in __INITIAL_DATA__ is extracted as photo URLs."""

+        """listing.photos array in __INITIAL_DATA__ is extracted as photo URLs (via raw HTML scan)."""

-        """listing.mediaCollection array in __INITIAL_DATA__ is extracted as photo URLs."""

+        """listing.mediaCollection array in __INITIAL_DATA__ is extracted as photo URLs (via raw HTML scan)."""

+    def test_parse_upgrades_32char_hash_thumbnail_to_origin(self) -> None:

+        """32-char hex hashes (UUID without dashes) in non-standard sizes are upgraded to /origin.jpg.

+        _MEDIA_HASH_RE used to require 40+ chars, which silently kept thumbnails at non-origin

+        quality when Compass uses shorter-format hashes.  The fix lowers the threshold to 32.

+        hash32 = "b" * 32  # exactly 32 hex chars — shorter than former 40+ threshold

+        thumbnail_url = f"https://www.compass.com/m/{hash32}/1200x900.jpg"

+        expected_origin = f"https://www.compass.com/m/{hash32}/origin.jpg"

+        html = self._html_with(f'"media":{{"thumbnailUrl":"{thumbnail_url}"}}')

+        parsed = parse_compass_listing_html(html, listing_url=COMPASS_URL)

+        self.assertIn(expected_origin, parsed.photo_urls)

+        self.assertNotIn(thumbnail_url, parsed.photo_urls)


Bash

PYTHONPATH=. python3 -m unittest tests.test_compass_provider.CompassProviderTests.test_parse_upgrades_32char_hash_thumbnail_to_origin -v 2>&1

Confirm new 32-char hash test is RED


Error: Exit code 1
test_parse_upgrades_32char_hash_thumbnail_to_origin (tests.test_compass_provider.CompassProviderTests.test_parse_upgrades_32char_hash_thumbnail_to_origin)
32-char hex hashes (UUID without dashes) in non-standard sizes are upgraded to /origin.jpg. ... FAIL

======================================================================
FAIL: test_parse_upgrades_32char_hash_thumbnail_to_origin (tests.test_compass_provider.CompassProviderTests.test_parse_upgrades_32char_hash_thumbnail_to_origin)
32-char hex hashes (UUID without dashes) in non-standard sizes are upgraded to /origin.jpg.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/workspace/extra/projects/stagehand/tests/test_compass_provider.py", line 250, in test_parse_upgrades_32char_hash_thumbnail_to_origin
    self.assertIn(expected_origin, parsed.photo_urls)
AssertionError: 'https://www.compass.com/m/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb/origin.jpg' not found in ('https://www.compass.com/m/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb/1200x900.jpg',)

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)


RED confirmed. Now fix `_MEDIA_HASH_RE` in compass.py:


Edit

_MEDIA_HASH_RE = re.compile(r"/m/([0-9a-f]{40,})/", re.I)

_MEDIA_HASH_RE = re.compile(r"/m/([0-9a-f]{32,})/", re.I)


The file /workspace/extra/projects/stagehand/core/listings/providers/compass.py has been updated successfully. (file state is current in your context — no need to Read it back)

_MEDIA_HASH_RE = re.compile(r"/m/([0-9a-f]{40,})/", re.I)

_MEDIA_HASH_RE = re.compile(r"/m/([0-9a-f]{32,})/", re.I)

"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"


Bash

PYTHONPATH=. python3 -m unittest tests.test_compass_provider -v 2>&1

Run full test suite to confirm all GREEN


test_fetch_materializes_photos_and_manifest (tests.test_compass_provider.CompassProviderTests.test_fetch_materializes_photos_and_manifest) ... ok
test_materialize_accepts_at_min_photo_threshold (tests.test_compass_provider.CompassProviderTests.test_materialize_accepts_at_min_photo_threshold)
Exactly min_photos downloaded → succeeds. ... ok
test_materialize_default_min_photos_is_five (tests.test_compass_provider.CompassProviderTests.test_materialize_default_min_photos_is_five)
Default min_photos=5; 2 photos raises without explicit kwarg. ... ok
test_materialize_rejects_below_min_photos (tests.test_compass_provider.CompassProviderTests.test_materialize_rejects_below_min_photos)
Fail closed: fewer than min_photos downloaded → CompassListingError. ... ok
test_parse_captures_photos_from_listing_photos_key (tests.test_compass_provider.CompassProviderTests.test_parse_captures_photos_from_listing_photos_key)
listing.photos array in __INITIAL_DATA__ is extracted as photo URLs (via raw HTML scan). ... ok
test_parse_captures_photos_from_media_collection_key (tests.test_compass_provider.CompassProviderTests.test_parse_captures_photos_from_media_collection_key)
listing.mediaCollection array in __INITIAL_DATA__ is extracted as photo URLs (via raw HTML scan). ... ok
test_parse_fixture_html_extracts_facts_and_deduped_photos (tests.test_compass_provider.CompassProviderTests.test_parse_fixture_html_extracts_facts_and_deduped_photos) ... ok
test_parse_rejects_empty_html (tests.test_compass_provider.CompassProviderTests.test_parse_rejects_empty_html) ... ok
test_parse_upgrades_32char_hash_thumbnail_to_origin (tests.test_compass_provider.CompassProviderTests.test_parse_upgrades_32char_hash_thumbnail_to_origin)
32-char hex hashes (UUID without dashes) in non-standard sizes are upgraded to /origin.jpg. ... ok
test_registry_requires_demo_or_rights_flag (tests.test_compass_provider.CompassProviderTests.test_registry_requires_demo_or_rights_flag) ... ok
test_registry_uses_compass_adapter_when_demo_enabled (tests.test_compass_provider.CompassProviderTests.test_registry_uses_compass_adapter_when_demo_enabled) ... ok
test_source_policy_allows_compass_adapter_path (tests.test_compass_provider.CompassProviderTests.test_source_policy_allows_compass_adapter_path) ... ok
test_zillow_still_never_calls_fetcher (tests.test_compass_provider.CompassProviderTests.test_zillow_still_never_calls_fetcher) ... ok

----------------------------------------------------------------------
Ran 13 tests in 0.003s

OK


13/13 GREEN. Now commit and update the cards.


git add core/listings/providers/compass.py tests/test_compass_provider.py && git commit -m "$(cat <<'EOF'
fix(compass): lower _MEDIA_HASH_RE threshold 40→32 to capture UUID-format hashes

Compass can serve photos with 32-char hex hashes (UUID without dashes).
The prior {40,} threshold silently kept these at non-origin resolution when
a non-standard size (e.g. 1200x900.jpg) appeared in the initial HTML without
a matching origin.jpg URL.  Lowering to {32,} means _media_hash_re constructs
an /origin.jpg candidate for any hash ≥ 32 hex chars; _prefer_origin_unique then
picks it over the thumbnail via its higher quality score.

Also adds 3 tests:
- regression for listing.photos / mediaCollection array keys (already worked via raw HTML scan)
- RED→GREEN for 32-char hash upgrade

1 RED→GREEN; 13/13 pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
EOF
)"

Commit compass.py fix and tests


[feature/listing-reel-workflow 05cfd87] fix(compass): lower _MEDIA_HASH_RE threshold 40→32 to capture UUID-format hashes
 2 files changed, 62 insertions(+), 1 deletion(-)


Now update the punch list and project card:


67	- [x] Fail closed below min photo count for demos — `min_photos=5` default added to `materialize_compass_listing` + threaded through `fetch_compass_listing` / `build_authorized_fetcher`; 3 RED→GREEN tests (e8bcf28 on feature/listing-reel-workflow).
68	- [ ] Expand Compass gallery capture further — lazy-loaded/paginated photos beyond initial HTML metadata.
69	- [ ] PureRealty adapter after we clear cookie/login or get an authorized feed/export.
70	- [ ] Production rights path: broker attestation workflow + licensed Bridge/RESO option.
71
72	## Before/After Contractor Reel Template

- [x] Fail closed below min photo count for demos — `min_photos=5` default added to `materialize_compass_listing` + threaded through `fetch_compass_listing` / `build_authorized_fetcher`; 3 RED→GREEN tests (e8bcf28 on feature/listing-reel-workflow).
- [ ] Expand Compass gallery capture further — lazy-loaded/paginated photos beyond initial HTML metadata.
- [ ] PureRealty adapter after we clear cookie/login or get an authorized feed/export.
- [ ] Production rights path: broker attestation workflow + licensed Bridge/RESO option.

## Before/After Contractor Reel Template


Edit

- [ ] Expand Compass gallery capture further — lazy-loaded/paginated photos beyond initial HTML metadata.

- [x] Expand hash extraction range — `_MEDIA_HASH_RE` threshold lowered 40→32 chars; 32-char (UUID-format) hashes now upgraded to `/origin.jpg` instead of staying at thumbnail quality (05cfd87).
- [ ] Expand Compass gallery capture further — lazy-loaded photos loaded via JS API calls after page render (requires headless browser or Compass API reverse-engineering; not capturable from initial HTML).


The file /workspace/extra/projects/stagehand/docs/punch-list.md has been updated successfully. (file state is current in your context — no need to Read it back)

- [x] Expand hash extraction range — `_MEDIA_HASH_RE` threshold lowered 40→32 chars; 32-char (UUID-format) hashes now upgraded to `/origin.jpg` instead of staying at thumbnail quality (05cfd87).
- [ ] Expand Compass gallery capture further — lazy-loaded photos loaded via JS API calls after page render (requires headless browser or Compass API reverse-engineering; not capturable from initial HTML).

# CreatorFlow Punch List

This list tracks the work needed to prove the first local AI workflow before broadening into a platform.

## Project Standards

- [x] Keep a repo-level todo/punch list as a standard project operating artifact.
- [x] Add project-level agent instructions requiring the punch list to stay current.

## Discovery

- [x] Prepare a first-agent discovery packet and acceptance criteria worksheet.
- [ ] Collect 5-8 raw walkthrough videos from the first real estate agent.
- [ ] Collect examples of posts/videos the agent considers publishable.
- [ ] Confirm target turnaround time, revision expectations, and pricing signal.
- [ ] Capture real estate compliance preferences and required brokerage/disclaimer language.
- [ ] Define first workflow acceptance criteria from the discovery conversation.

## Product Guardrails

- [x] Keep the first version concierge-first: client sends raw input, operator reviews, client receives finished output.
- [x] Require human approval before final rendered output is considered publishable.
- [x] Do not automate posting in the first version.
- [x] Use a licensed stock voice or explicitly approved voice only.
- [x] Track manual touch time per video to identify the next automation target.

## Technical Foundation

- [x] Initialize Python project scaffold with test runner.
- [x] Add local Postgres setup and initial migrations.
- [x] Model tenants, media assets, jobs, and job steps with approval state.
- [x] Build workflow config parser and runner with approval gates.
- [x] Build CLI commands for running workflows, checking status, and approving scripts.
- [x] Add media utilities for ffmpeg command construction and rendering.
- [x] Add a local no-paid-API demo render path that produces a playable vertical MP4 from the phone video.
- [x] Render transcript-derived TTS narration with synchronized captions and source audio muted by default.
- [x] Wire real transcription via OpenAI Whisper (`--transcription-provider whisper`). Audio is extracted before submission to avoid container format issues with the API.
- [x] Improve transcript cleanup: remove filler words, hesitation sounds, consecutive duplicate words. Re-capitalize after removal.
- [x] Caption timing uses Whisper segment timestamps (scaled to TTS duration) when available, with equal-duration chunks as the offline fallback.
- [x] Evaluate provider-backed word-level timestamps for tighter per-word caption sync (current path uses sentence-level segments).
- [x] Make local TTS verification stricter and investigate macOS `say` intermittently producing header-only audio before falling back to silent demo audio.

## Listing URL Reel

- [x] Research real-estate reel competitors, HeyGen/HyperFrames capabilities, listing-source terms, Instagram output requirements, and voice-cloning constraints.
- [x] Write an implementation-ready listing URL to Instagram reel plan.
- [x] Add a canonical authorized listing manifest with source policy, path confinement, rights attestation, and photo integrity hashes.
- [x] Add a single approval bundle covering facts, narration, caption, photo order, cover choice, and contact outro.
- [x] Render a 9:16 photo reel with restrained pan/zoom motion, captions, narration, and a contact outro.
- [x] Produce and visually QA two distinct HyperFrames portrait listing-reel variants from authorized local pilot media (private demo outputs remain ignored).
- [x] Revise the pilot reels with all supplied Horne listing photos and concise Richard Fowler contact-card outros; remove unsupported mood copy.
- [x] Generate a designed property cover and include it in the manual Instagram handoff package.
- [x] Require consent and audible output when cloned narration is requested.
- [x] Run and inspect a local approval-resume end-to-end listing reel render with a real fixture (intentional silent provider; live cloned-voice validation still requires owner consent and credentials).
- [x] Add Compass demo link-intake adapter (`--demo`) that materializes local photos + snapshot while keeping Zillow/etc. no-fetch and production rights as a ship blocker.
- [x] Soften photo-reel Ken Burns: continuous center scale (no zoompan), 60fps, CRF 18, light denoise, quality-first encode.
- [x] Force spoken “square feet” + city/state-only narration (no street “Dr=doctor”).
- [x] Write Richard-facing ElevenLabs PVC + optional HeyGen avatar setup steps for private share.
- [x] Local Coqui XTTS sample against Richard reference wav for pilot listen.
- [x] Add operator-facing pre/post render reel QA scorecards plus private review links for remote artifact checks.
- [x] Add a repeatable HyperFrames reel review command/skill plus a reboot-persistent private review server.
- [x] Fix hyperframes-video-review live CLI checks hanging: bound every review subprocess with a timeout, drop the network-dependent Gemini `snapshot --describe` step (unused by our scoring) and the update-check ping.
- [x] Add `hyperframes-add-narration` command + skill doc: standard, repeatable fix for a video-review "missing required narration/audio" hard failure (synthesize via local `say`, mux into the silent HyperFrames render).
- [x] Name packaged delivery videos consistently with early `property` / `recommendation` type segments for easier sharing and sorting.
- [ ] Wire natural voice default to Richard’s shared ElevenLabs PVC once he completes PVC share + consent.
- [ ] Optional: Coqui XTTS as first-class listing-reel TTS provider (research-only until PVC lands).
- [x] Fail closed below min photo count for demos — `min_photos=5` default added to `materialize_compass_listing` + threaded through `fetch_compass_listing` / `build_authorized_fetcher`; 3 RED→GREEN tests (e8bcf28 on feature/listing-reel-workflow).
- [ ] Expand Compass gallery capture further — lazy-loaded/paginated photos beyond initial HTML metadata.
- [ ] PureRealty adapter after we clear cookie/login or get an authorized feed/export.
- [ ] Production rights path: broker attestation workflow + licensed Bridge/RESO option.

## Before/After Contractor Reel Template

- [x] Build a reusable HyperFrames before/after template (50/50 split-screen, before on top / after on bottom, portrait full-bleed) as a new reel family alongside the existing listing and recommendation reels.
- [x] Produce a landscaping demo instance (`output/hyperframes/landscaping-before-after-reel/`) with 4 before/after beats + shared contact-card outro, narration, and a passing script/video QA review.
- [x] Document swap points for reuse across other contractor trades (painter, remodeler, pool, etc.) in that project's `TEMPLATE-NOTES.md`.
- [x] Get one real (non-AI-generated) before/after photo set from an actual contractor to validate the template against real client media.
- [x] Multi-client support: swapped the demo's outro to a direct-brand-CTA variant (placeholder "Contract Matcher" brand, Morgan Hill CA, placeholder phone) instead of Richard's personal-referral card; documented the outro as the one "client identity block" to fork per client in `TEMPLATE-NOTES.md`.
- [x] Promoted the client identity block (brand name, outro headline/tagline/city/phone, avatar monogram, accent colors) to HyperFrames `data-composition-variables`, resolved at init and configurable via `client.config.json` (`--variables-file` at render time) instead of hand-edited HTML. Verified the override plumbing end-to-end with a throwaway low-fps render before finalizing.
- [ ] Get Contract Matcher's real business name, phone number (for the text-based scheduler being built separately), web address, and logo to replace the placeholders in `client.config.json`.

## Testing

- [x] Use TDD for deterministic business logic, config parsing, tenant scoping, approval gates, and media command construction.
- [x] Use BDD-style scenarios for the video narration workflow.
- [x] Mock paid/external APIs in the default test suite.
- [x] Add opt-in live integration tests only after mocked contracts are stable.
- [x] Add a short test video fixture for the end-to-end path.
- [x] Verify local render execution against the committed short video fixture.

## Provider Setup

- [x] Document local `.env` setup.
- [x] Document Neon database creation and migration steps.
- [x] Document Vercel env vars and deployment steps for any later approval UI/API.
- [x] Document Cloudflare DNS setup for the chosen domain/subdomain.
- [x] Keep long-running video rendering out of Vercel request handlers.

-- [ ] Expand Compass gallery capture further — lazy-loaded/paginated photos beyond initial HTML metadata.

+- [x] Expand hash extraction range — `_MEDIA_HASH_RE` threshold lowered 40→32 chars; 32-char (UUID-format) hashes now upgraded to `/origin.jpg` instead of staying at thumbnail quality (05cfd87).

+- [ ] Expand Compass gallery capture further — lazy-loaded photos loaded via JS API calls after page render (requires headless browser or Compass API reverse-engineering; not capturable from initial HTML).


Bash

git add docs/punch-list.md && git commit -m "$(cat <<'EOF'
chore: update punch list — hash-extraction expansion done, JS-lazy gap clarified

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
EOF
)"

Commit punch list update


[feature/listing-reel-workflow e723264] chore: update punch list — hash-extraction expansion done, JS-lazy gap clarified
 1 file changed, 2 insertions(+), 1 deletion(-)


Edit

- 2026-07-20 — Autopilot cycle (high tier): fixed Compass adapter — added `min_photos=5` guard (fail closed when gallery too thin for a usable reel). Threaded through `fetch_compass_listing` + `build_authorized_fetcher`. 3 RED→GREEN tests; 10/10 pass. Committed e8bcf28 on feature/listing-reel-workflow. Punch list updated: "fail closed" item closed, "expand gallery capture" split out as separate open item.

- 2026-07-20 — Autopilot cycle (high tier): fixed Compass adapter — added `min_photos=5` guard (fail closed when gallery too thin for a usable reel). Threaded through `fetch_compass_listing` + `build_authorized_fetcher`. 3 RED→GREEN tests; 10/10 pass. Committed e8bcf28 on feature/listing-reel-workflow. Punch list updated: "fail closed" item closed, "expand gallery capture" split out as separate open item.
- 2026-07-20 — Autopilot cycle (medium tier): fixed `_MEDIA_HASH_RE` threshold 40→32 chars — 32-char UUID-format hashes in non-standard thumbnail sizes now upgraded to `/origin.jpg`. 1 RED→GREEN test; 13/13 pass. Committed 05cfd87 on feature/listing-reel-workflow. Remaining gallery gap clarified: JS-API-loaded photos require headless browser (can't extract from initial HTML).