Skip to content

Add Extra Playwright Servers (per-watch selectable self-hosted CDP browser servers) - #4201

Open
geekho-me wants to merge 5 commits into
dgtlmoon:masterfrom
geekho-me:feature/extra-playwright-servers
Open

Add Extra Playwright Servers (per-watch selectable self-hosted CDP browser servers)#4201
geekho-me wants to merge 5 commits into
dgtlmoon:masterfrom
geekho-me:feature/extra-playwright-servers

Conversation

@geekho-me

Copy link
Copy Markdown

What

Adds an Extra Playwright Servers feature: configure multiple self-hosted Playwright/CDP browser servers in Settings → Captcha/Proxies, and select which one to use per-watch via the existing "Fetch Method" dropdown. It mirrors the existing "Extra Browsers" feature but is dedicated to self-hosted browser servers (e.g. browserless / sockpuppetbrowser).

Why / use case

Run multiple browser servers in different geolocations or configurations (for example, one behind a VPN for a specific country) and pick the right one for an individual watch — useful for geo-specific content, price/restock checks on region-locked stores, or spreading fetch load across servers.

How it works

  • New global setting settings.requests.extra_playwright_servers (model/App.py).
  • New SingleExtraPlaywrightServer form with ws:///wss:// URL validation, added as a FieldList to the global settings form (forms.py).
  • New settings UI section in the Captcha/Proxies tab, styled to match the existing "Extra Browsers" section (settings.html, plus the SCSS/compiled CSS).
  • ChangeDetectionStore.extra_playwright_servers exposes configured servers as ("extra_playwright_server_<name>", "<name>") choices, filtering out incomplete entries (store/__init__.py).
  • The watch edit form offers each server as a Fetch Method (blueprint/ui/edit.py).
  • In processors/base.py, a watch whose fetch_backend starts with extra_playwright_server_ resolves the configured URL and forces the Playwright fetcher (connect_over_cdp) with that connection URL.

Fix: screenshot/capability detection for prefixed fetch backends

While building this I found that get_fetcher_capabilities() (pluggy_interface.py) and Watch.fetcher_supports_screenshots (model/Watch.py) matched the watch's fetch_backend against the fetcher registry verbatim. For the prefixed backends (extra_playwright_server_* and the existing extra_browser_*) nothing matched, so capabilities fell back to all-False. The visible symptom: the preview/history "Current screenshot" tab showed "Screenshot requires a Content Fetcher … that supports screenshots" even though the screenshot had been captured.

Both functions now resolve the prefixes the same way processors/base.py does (extra_playwright_server_* → Playwright fetcher; extra_browser_*html_webdriver). This also fixes the pre-existing extra_browser_* case.

Tests

  • tests/test_extra_playwright_servers.py — Flask-client feature test: saving a server in Settings persists it, the datastore exposes/filters it, it appears as a "Fetch Method" on the edit page, and the ws:///wss:// validator rejects other schemes.
  • tests/unit/test_fetcher_capabilities.py — browser-free unit tests for capability resolution of extra_playwright_server_*, extra_browser_*, plain html_requests, and unknown backends, for both get_fetcher_capabilities() and Watch.fetcher_supports_screenshots.

Docs

  • docs/extra-playwright-servers.md — configuration, the ws:///wss:// CDP requirement, compatible servers, and a test compose stack.

Compatibility note

changedetection connects with Playwright's connect_over_cdp() (Chrome DevTools Protocol) at the root path, so a compatible server is a CDP gateway that accepts a connection at / — e.g. browserless or sockpuppetbrowser. The upstream microsoft/playwright run-server (Playwright wire protocol) is not compatible with connect_over_cdp and is documented as such.

Notes for reviewers

  • A docker-compose.test-playwright.yml helper stack (browserless + sockpuppetbrowser) is included to exercise the feature; happy to drop it if you'd prefer it out of the repo.
  • The compiled static/styles/styles.css change is the build output of the SCSS change (it comma-extends the existing extra-browsers rules to the new IDs).
  • The translation catalog (messages.pot / .po) was regenerated with python setup.py extract_messages && update_catalog && compile_catalog.

🤖 Generated with Claude Code

geekho-me and others added 5 commits May 30, 2026 15:59
Allow configuring extra self-hosted Playwright/CDP browser servers in
Settings (Captcha/Proxies). Each configured server is selectable per-watch
as a "Fetch Method" and is force-routed through the Playwright fetcher.

Includes the data model default, the form + ws://-URL validation, the
settings UI section, the datastore property that exposes configured servers,
and the processor routing in difference_detection_processor.
- Resolve extra_playwright_server_* / extra_browser_* fetch backends when
  reporting fetcher capabilities, so screenshots (and browser steps / xpath)
  are correctly detected. Previously these prefixed backends matched no
  fetcher class and capabilities fell back to all-False, so the preview
  "Current screenshot" tab wrongly claimed the fetcher could not take
  screenshots. Fixed in get_fetcher_capabilities() and
  Watch.fetcher_supports_screenshots (also fixes the existing extra_browser_*).
- Style the "Extra Playwright Servers" settings section to match Extra Browsers
  (bordered box, inline table rows, no bullets/labels).
- Add docker-compose.test-playwright.yml: a test stack with browserless and
  sockpuppetbrowser CDP servers (documents why microsoft/playwright is not
  compatible with connect_over_cdp).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- tests/unit/test_fetcher_capabilities.py: regression tests covering
  get_fetcher_capabilities() and Watch.fetcher_supports_screenshots for the
  extra_browser_* / extra_playwright_server_* fetch-backend prefixes (the
  screenshot-capability bug), plus the plain-requests and unknown-backend
  cases. No browser required.
- docs/extra-playwright-servers.md: feature documentation - configuration,
  the ws:// CDP requirement, browserless/sockpuppet compatibility (and why the
  microsoft/playwright image is not compatible), and the test compose stack.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Flask-client test (no browser required) covering the new functionality:
- saving an extra Playwright server in Settings persists it
- datastore.extra_playwright_servers exposes complete servers as fetch-backend
  choice tuples and filters out entries missing a name or URL
- the server is offered as a "Fetch Method" on the watch edit page
- the ws:// / wss:// URL validator rejects other schemes

Satisfies CONTRIBUTING.md ("all new functionality has a matching test").

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Regenerate messages.pot and the .po/.mo catalogs (python setup.py
extract_messages && update_catalog && compile_catalog) to include the new
translatable strings added by the feature ("Extra Playwright Servers", the
form labels and the ws://-URL validation message). Fixes the lint-translations
CI check ("Check translation catalog is up-to-date").

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dgtlmoon

dgtlmoon commented Jun 3, 2026

Copy link
Copy Markdown
Owner

this already exists?

image

@AmirF194 AmirF194 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a nicely scoped feature that reuses the existing custom-connection path rather than adding a parallel one. A selected extra_playwright_server_<name> resolves to the same custom_browser_connection_url that extra_browsers already feeds into playwright.py connect_over_cdp(...), and the only real delta is forcing the Playwright fetcher via the new use_playwright_fetcher flag, which mirrors the existing browser-steps override. Backward compatibility is handled the proven way: the setting defaults to [] in model/App.py and every reader uses .get('extra_playwright_servers', []), matching exactly how extra_browsers was introduced, so existing watches need no migration. The capability fix is a real catch and correctly factored, and it also fixes a pre-existing extra_browser_* false-negative on the screenshot tab.

No blocking concerns from me. The one edge behavior worth knowing: if a server is deleted or renamed after a watch selects it, the filter returns empty and it silently falls back to html_requests (processors/base.py:172-174). That is the same silent downgrade extra_browsers already has, so it is not a regression, just a candidate for a future UX note.

On CI: the red run is environmental, not this feature. basic-tests failed on two different unrelated tests across the interpreters (3.11 on test_favicon_inline_data_uri, 3.14 on test_obfuscations with a transient 500), and this PR changes no favicon, obfuscation, or price code (those strings only appear in regenerated .po catalogs). Different failure per runner plus a transient 500 is the flaky signature, and the run is from late May against what is now a stale base, so a rebase and re-run should go green. Every fetcher-relevant suite (unit including the new ones, playwright, custom-browser, selenium, proxy, upgrade-path) passed.

Optional: the actual call_browser routing (that selecting a server sets the connection URL and forces Playwright) is the one untested surface. Everything around it is covered, but in a 29-file PR that is the behavior most likely to regress silently. A small unit test that mocks the fetcher and asserts use_playwright_fetcher plus the connection URL would close it, which the test-file comment already acknowledges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants