docs(browser): document the screenshot auto-attach escape hatch - #124
Merged
Conversation
`browser_execute` attaches every `Page.captureScreenshot` result to the next
assistant turn, which is the right default — it saves a `read` call almost
every time. But there was no way to capture pixels the agent does not need to
look at: bulk capture, or an artifact that only gets written to disk. A bulk
task attaches every frame to context with no opt-out.
The tap subscribes to `onCallResult`, which fires only from `_call`'s resolve
path — the command-response channel. CDP events route through a separate
listener list, so pixels delivered as `Page.screencastFrame` are structurally
invisible to it. That makes the escape hatch real CDP rather than an invented
parameter, so what the agent learns transfers to any client.
Verified against Chrome 150.0.7871.187 (--headless=new):
- static, already-loaded, idle page: startScreencast delivers a frame in 7ms
with no navigation, scroll, or re-layout nudge
- screencast frames produce zero attachments; captureScreenshot still attaches
- setDeviceMetricsOverride({ width: 1200, height: 630 }) yields a PNG whose
header reads exactly 1200x630
Two protocol details found while testing that are deliberately left out of the
skill text, recorded here instead:
- ack is not needed for navigation-paced capture (3 navigations produced 3
frames unacked). It is required for continuous capture: on an animating
page, 3s yielded 3 frames unacked vs 178 acked, so Chrome stalls after a
couple of unacked frames.
- `sessionId` on the frame event is not a frame counter despite the protocol
describing it as "Frame number" — it is constant within a cast and
increments per startScreencast.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
browser_executeattaches everyPage.captureScreenshotresult to the next assistant turn. That default is right — it saves areadcall almost every time the agent screenshots something it wants to look at, and we don't want to change it.But there was no way to capture pixels the agent does not need to see. Two real cases:
Both are rare, which is why this is documentation rather than tool surface. Alternatives considered and rejected in #122: a synthetic
attachToContextparam onPage.captureScreenshot(teaches the agent a CDP field Chrome has never heard of, and carried the opt-out viaWeakSetobject identity, which fails silently in the leak direction), and a fourthbrowser_executetool parameter (a permanent, eagerly-loaded schema field on every call for a rare need).Why a documented route works
The screenshot tap subscribes via
session.onCallResult(...), which fires only from_call's resolve path (session.ts:252-256) — the command/response channel. CDP events route through a separate listener list (session.ts:273-277) and never reach it.So pixels delivered as the
Page.screencastFrameevent are structurally invisible to the tap. No code change, and the agent learns a genuine protocol capability that transfers to any CDP client instead of a BrowserCode-only fiction.Change
15 added lines in
skills/browser-execute/SKILL.md, next to the auto-attach explanation it qualifies. The lead sentence states the rule (response attaches, event does not) so the agent can reason about cases beyond the example.Validation
Verified against Chrome 150.0.7871.187 with
--headless=new— the harder case for compositing — viabrowser_executeagainst a throwaway debug-port profile:startScreencastdelivers a frame in 7ms with no navigation, scroll, or re-layout nudge. This was the open question; the documented snippet needs no nudge and no polling loop.Page.captureScreenshotstill attaches (control case rendered inline as expected).setDeviceMetricsOverride({ width: 1200, height: 630, deviceScaleFactor: 1 })yields a PNG whose header reads exactly 1200x630, confirming the viewport is what sets frame size.bun test test/skills.test.ts— 2 pass (skill still materializes with{{SKILLS_DIR}}substituted).Emulation.setDeviceMetricsOverrideis markedOptional:in the snippet on purpose: it reflows the page, so it is wrong as an unconditional step for bulk capture where you want the page as-is.Protocol details found while testing, deliberately left out of the skill text
Recorded here rather than spending tokens on them in a lazily-loaded doc:
screencastFrameAckis not needed for navigation-paced capture — 3 navigations produced 3 frames with zero acks. It is required for continuous capture: on an animating page a 3s window yielded 3 frames unacked vs 178 acked, so Chrome stalls after a couple of unacked frames. The bulk-capture case that motivated this PR is navigation-paced, so the skill omits ack entirely.sessionIdon the frame event is not a frame counter, despite the protocol describing it as "Frame number". It is constant within a cast and increments perstartScreencastcall. Anything acking per-frame by treating it as a frame identity will behave confusingly.Page.screencastFrameis flagged"experimental": truein the protocol. It is the mechanism cloud's own live preview uses, so it is stable in practice.Zone
Green —
packages/bcode-browser/only, docs-only, no upstream diff, noEXCEPTIONS.mdentry needed.Closes #122.
Summary by cubic
Documented how to opt out of screenshot auto-attach in
browser_execute. UsePage.screencastFrameinstead ofPage.captureScreenshotto capture pixels without attaching (responses attach; events do not), useful for bulk capture and disk-only artifacts.Written for commit da8b535. Summary will update on new commits.