From da8b535678869317bc0dc9093c7ee0a06f3099e2 Mon Sep 17 00:00:00 2001 From: Alexander Yue Date: Tue, 28 Jul 2026 20:53:59 -0700 Subject: [PATCH] docs(browser): document the screenshot auto-attach escape hatch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `browser_execute` attaches every `Page.captureScreenshot` result to the next assistant turn, which is the right default — it saves a `read` call almost every time. But there was no way to capture pixels the agent does not need to look at: bulk capture, or an artifact that only gets written to disk. A bulk task attaches every frame to context with no opt-out. The tap subscribes to `onCallResult`, which fires only from `_call`'s resolve path — the command-response channel. CDP events route through a separate listener list, so pixels delivered as `Page.screencastFrame` are structurally invisible to it. That makes the escape hatch real CDP rather than an invented parameter, so what the agent learns transfers to any client. Verified against Chrome 150.0.7871.187 (--headless=new): - static, already-loaded, idle page: startScreencast delivers a frame in 7ms with no navigation, scroll, or re-layout nudge - screencast frames produce zero attachments; captureScreenshot still attaches - setDeviceMetricsOverride({ width: 1200, height: 630 }) yields a PNG whose header reads exactly 1200x630 Two protocol details found while testing that are deliberately left out of the skill text, recorded here instead: - ack is not needed for navigation-paced capture (3 navigations produced 3 frames unacked). It is required for continuous capture: on an animating page, 3s yielded 3 frames unacked vs 178 acked, so Chrome stalls after a couple of unacked frames. - `sessionId` on the frame event is not a frame counter despite the protocol describing it as "Frame number" — it is constant within a cast and increments per startScreencast. --- .../bcode-browser/skills/browser-execute/SKILL.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/packages/bcode-browser/skills/browser-execute/SKILL.md b/packages/bcode-browser/skills/browser-execute/SKILL.md index b71c9d92f..217f877ed 100644 --- a/packages/bcode-browser/skills/browser-execute/SKILL.md +++ b/packages/bcode-browser/skills/browser-execute/SKILL.md @@ -149,6 +149,21 @@ await session.Page.captureScreenshot({ format: "png" }) `Page.navigate` can return a non-empty `errorText` instead of throwing. Treat it as a failed navigation. If `ERR_TUNNEL_CONNECTION_FAILED` persists, reloading, reattaching, or reconnecting to the same endpoint cannot change its proxy route; use another source or replace the cloud browser instead of retrying it. +Not auto-attaching a screenshot: attachment is keyed on the `Page.captureScreenshot` response; pixels that arrive as an event are never attached. + +```js +await session.Page.enable() +// Optional: screencast has no `clip`, so override the viewport for exact frame size. +await session.Emulation.setDeviceMetricsOverride({ width: 1200, height: 630, deviceScaleFactor: 1, mobile: false }) +const frame = session.waitFor("Page.screencastFrame", { timeoutMs: 10_000 }) // register first +await session.Page.startScreencast({ format: "png" }) +try { + const f = await frame // f.data is base64, same as captureScreenshot +} finally { + await session.Page.stopScreencast() // otherwise the cast stays open +} +``` + ## Reusing code The agent-workspace is per-project: `./.bcode/agent-workspace/`. Use this to write memory files, scripts, and helper functions.