Add Extra Playwright Servers (per-watch selectable self-hosted CDP browser servers) - #4201
Add Extra Playwright Servers (per-watch selectable self-hosted CDP browser servers)#4201geekho-me wants to merge 5 commits into
Conversation
Allow configuring extra self-hosted Playwright/CDP browser servers in Settings (Captcha/Proxies). Each configured server is selectable per-watch as a "Fetch Method" and is force-routed through the Playwright fetcher. Includes the data model default, the form + ws://-URL validation, the settings UI section, the datastore property that exposes configured servers, and the processor routing in difference_detection_processor.
- Resolve extra_playwright_server_* / extra_browser_* fetch backends when reporting fetcher capabilities, so screenshots (and browser steps / xpath) are correctly detected. Previously these prefixed backends matched no fetcher class and capabilities fell back to all-False, so the preview "Current screenshot" tab wrongly claimed the fetcher could not take screenshots. Fixed in get_fetcher_capabilities() and Watch.fetcher_supports_screenshots (also fixes the existing extra_browser_*). - Style the "Extra Playwright Servers" settings section to match Extra Browsers (bordered box, inline table rows, no bullets/labels). - Add docker-compose.test-playwright.yml: a test stack with browserless and sockpuppetbrowser CDP servers (documents why microsoft/playwright is not compatible with connect_over_cdp). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- tests/unit/test_fetcher_capabilities.py: regression tests covering get_fetcher_capabilities() and Watch.fetcher_supports_screenshots for the extra_browser_* / extra_playwright_server_* fetch-backend prefixes (the screenshot-capability bug), plus the plain-requests and unknown-backend cases. No browser required. - docs/extra-playwright-servers.md: feature documentation - configuration, the ws:// CDP requirement, browserless/sockpuppet compatibility (and why the microsoft/playwright image is not compatible), and the test compose stack. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Flask-client test (no browser required) covering the new functionality:
- saving an extra Playwright server in Settings persists it
- datastore.extra_playwright_servers exposes complete servers as fetch-backend
choice tuples and filters out entries missing a name or URL
- the server is offered as a "Fetch Method" on the watch edit page
- the ws:// / wss:// URL validator rejects other schemes
Satisfies CONTRIBUTING.md ("all new functionality has a matching test").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Regenerate messages.pot and the .po/.mo catalogs (python setup.py
extract_messages && update_catalog && compile_catalog) to include the new
translatable strings added by the feature ("Extra Playwright Servers", the
form labels and the ws://-URL validation message). Fixes the lint-translations
CI check ("Check translation catalog is up-to-date").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AmirF194
left a comment
There was a problem hiding this comment.
This is a nicely scoped feature that reuses the existing custom-connection path rather than adding a parallel one. A selected extra_playwright_server_<name> resolves to the same custom_browser_connection_url that extra_browsers already feeds into playwright.py connect_over_cdp(...), and the only real delta is forcing the Playwright fetcher via the new use_playwright_fetcher flag, which mirrors the existing browser-steps override. Backward compatibility is handled the proven way: the setting defaults to [] in model/App.py and every reader uses .get('extra_playwright_servers', []), matching exactly how extra_browsers was introduced, so existing watches need no migration. The capability fix is a real catch and correctly factored, and it also fixes a pre-existing extra_browser_* false-negative on the screenshot tab.
No blocking concerns from me. The one edge behavior worth knowing: if a server is deleted or renamed after a watch selects it, the filter returns empty and it silently falls back to html_requests (processors/base.py:172-174). That is the same silent downgrade extra_browsers already has, so it is not a regression, just a candidate for a future UX note.
On CI: the red run is environmental, not this feature. basic-tests failed on two different unrelated tests across the interpreters (3.11 on test_favicon_inline_data_uri, 3.14 on test_obfuscations with a transient 500), and this PR changes no favicon, obfuscation, or price code (those strings only appear in regenerated .po catalogs). Different failure per runner plus a transient 500 is the flaky signature, and the run is from late May against what is now a stale base, so a rebase and re-run should go green. Every fetcher-relevant suite (unit including the new ones, playwright, custom-browser, selenium, proxy, upgrade-path) passed.
Optional: the actual call_browser routing (that selecting a server sets the connection URL and forces Playwright) is the one untested surface. Everything around it is covered, but in a 29-file PR that is the behavior most likely to regress silently. A small unit test that mocks the fetcher and asserts use_playwright_fetcher plus the connection URL would close it, which the test-file comment already acknowledges.

What
Adds an Extra Playwright Servers feature: configure multiple self-hosted Playwright/CDP browser servers in Settings → Captcha/Proxies, and select which one to use per-watch via the existing "Fetch Method" dropdown. It mirrors the existing "Extra Browsers" feature but is dedicated to self-hosted browser servers (e.g. browserless / sockpuppetbrowser).
Why / use case
Run multiple browser servers in different geolocations or configurations (for example, one behind a VPN for a specific country) and pick the right one for an individual watch — useful for geo-specific content, price/restock checks on region-locked stores, or spreading fetch load across servers.
How it works
settings.requests.extra_playwright_servers(model/App.py).SingleExtraPlaywrightServerform withws:///wss://URL validation, added as aFieldListto the global settings form (forms.py).settings.html, plus the SCSS/compiled CSS).ChangeDetectionStore.extra_playwright_serversexposes configured servers as("extra_playwright_server_<name>", "<name>")choices, filtering out incomplete entries (store/__init__.py).blueprint/ui/edit.py).processors/base.py, a watch whosefetch_backendstarts withextra_playwright_server_resolves the configured URL and forces the Playwright fetcher (connect_over_cdp) with that connection URL.Fix: screenshot/capability detection for prefixed fetch backends
While building this I found that
get_fetcher_capabilities()(pluggy_interface.py) andWatch.fetcher_supports_screenshots(model/Watch.py) matched the watch'sfetch_backendagainst the fetcher registry verbatim. For the prefixed backends (extra_playwright_server_*and the existingextra_browser_*) nothing matched, so capabilities fell back to all-False. The visible symptom: the preview/history "Current screenshot" tab showed "Screenshot requires a Content Fetcher … that supports screenshots" even though the screenshot had been captured.Both functions now resolve the prefixes the same way
processors/base.pydoes (extra_playwright_server_*→ Playwright fetcher;extra_browser_*→html_webdriver). This also fixes the pre-existingextra_browser_*case.Tests
tests/test_extra_playwright_servers.py— Flask-client feature test: saving a server in Settings persists it, the datastore exposes/filters it, it appears as a "Fetch Method" on the edit page, and thews:///wss://validator rejects other schemes.tests/unit/test_fetcher_capabilities.py— browser-free unit tests for capability resolution ofextra_playwright_server_*,extra_browser_*, plainhtml_requests, and unknown backends, for bothget_fetcher_capabilities()andWatch.fetcher_supports_screenshots.Docs
docs/extra-playwright-servers.md— configuration, thews:///wss://CDP requirement, compatible servers, and a test compose stack.Compatibility note
changedetection connects with Playwright's
connect_over_cdp()(Chrome DevTools Protocol) at the root path, so a compatible server is a CDP gateway that accepts a connection at/— e.g. browserless or sockpuppetbrowser. The upstreammicrosoft/playwrightrun-server(Playwright wire protocol) is not compatible withconnect_over_cdpand is documented as such.Notes for reviewers
docker-compose.test-playwright.ymlhelper stack (browserless + sockpuppetbrowser) is included to exercise the feature; happy to drop it if you'd prefer it out of the repo.static/styles/styles.csschange is the build output of the SCSS change (it comma-extends the existing extra-browsers rules to the new IDs).messages.pot/.po) was regenerated withpython setup.py extract_messages && update_catalog && compile_catalog.🤖 Generated with Claude Code