Point it at a page you want to scrape; get back a working extractor.
html-to-schema reads a sample of HTML (a file, stdin, or a URL), infers an
extraction schema — a mapping of field -> CSS selector with a detected data
type and an example value — and then generates ready-to-run scraper code in
your choice of BeautifulSoup, Parsel, or Scrapy, plus a pytest validation test.
It turns "here's a page I want to scrape" into a correct extractor skeleton in seconds, so you spend your time on the interesting parts instead of clicking through DevTools writing selectors by hand.
Everything is pure-Python and deterministic — no LLM, no network calls (except
the optional --url fetch), no guessing at runtime.
- Detects repeating records — product cards, search results, list items, table rows. It finds the sibling group with the richest, most-repeated structure, picks the repeating container, and infers per-record fields with stable relative selectors.
- Handles detail pages — for single-record pages it infers top-level fields
(
h1title, canonical URL, meta description, breadcrumb, price…). - Reads structured data — if the page ships schema.org JSON-LD, those
fields are surfaced too (e.g.
offers.price,brand.name,sku). - Prefers stable selectors —
itemprop/data-*attributes, shared classes and semantic tags first, falling back to a structuralnth-of-typepath. - Names, types, and examples — each field gets a
snake_casename, a CSS selector, an extraction kind (text/attribute:href/attribute:src/html/jsonld), a best-effort type (str/number/url/date), and an example value pulled straight from your sample. - Generates code + a test — clean, dependency-light extractors for three popular stacks, and a pytest test that proves the code recovers the example values.
html-to-schema is not published to PyPI — install it from source, straight
from GitHub:
pip install git+https://github.com/python-web-scraping-com/html-to-schema.gitSince it ships a command-line tool, pipx is a good way
to install it in its own isolated environment:
pipx install git+https://github.com/python-web-scraping-com/html-to-schema.gitRequires Python 3.10+.
html-to-schema infer product_page.html # rich table
html-to-schema infer product_page.html --json # machine-readable JSON
cat product_page.html | html-to-schema infer - # read from stdin
html-to-schema infer --url https://example.com/products # fetch first╭─ html-to-schema ─────────────────────────────────────────────╮
│ listing records: div.product-card (×4) │
╰──────────────────────────────────────────────────────────────╯
┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┓
┃ field ┃ selector ┃ extract ┃ type ┃ example ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━┩
│ title │ h2.product-title │ text │ str │ Widget Pro│
│ url │ a.card-link │ attribute:href │ url │ /products/1│
│ image │ img.thumb │ attribute:src │ url │ /img/1.jpg│
│ price │ span.price │ text │ number │ $19.99 │
│ brand │ span.brand │ text │ str │ Acme │
│ rating │ span.rating │ text │ number │ 4.5 │
└────────┴───────────────────┴────────────────┴────────┴───────────┘
html-to-schema gen product_page.html --flavor beautifulsoup # default
html-to-schema gen product_page.html --flavor parsel
html-to-schema gen product_page.html --flavor scrapyWrite a runnable scraper and its validation test to a directory:
html-to-schema gen product_page.html --with-test -o out/
# writes out/scraper.py, out/test_scraper.py, out/sample.htmlA generated BeautifulSoup extractor for the listing above looks like this:
"""Auto-generated scraper (BeautifulSoup).
Target: listing.
Generated by html-to-schema — https://python-web-scraping.com
"""
from __future__ import annotations
import re
from bs4 import BeautifulSoup
def _text(el):
if el is None:
return None
return re.sub(r"\s+", " ", el.get_text(" ", strip=True)).strip() or None
def _attr(el, name):
if el is None:
return None
return el.get(name)
def extract(html: str) -> list[dict]:
"""Extract every record from the page."""
soup = BeautifulSoup(html, "lxml")
records = []
for rec in soup.select('div.product-card'):
records.append({
'title': _text(rec.select_one('h2.product-title')), # str
'url': _attr(rec.select_one('a.card-link'), 'href'), # url
'image': _attr(rec.select_one('img.thumb'), 'src'), # url
'price': _text(rec.select_one('span.price')), # number
'brand': _text(rec.select_one('span.brand')), # str
'rating': _text(rec.select_one('span.rating')), # number
})
return recordsfrom html_to_schema import infer, generate_code, generate_test
schema = infer(open("product_page.html").read())
print(schema.kind) # "listing"
print(schema.record_selector) # "div.product-card"
for field in schema.fields:
print(field.name, field.selector, field.type, field.example)
print(schema.to_json()) # serialize the schema
print(generate_code(schema, flavor="parsel")) # a Parsel extractor
print(generate_test(schema)) # a pytest validation testinfer() returns a Schema dataclass (schema.to_dict() / schema.to_json()).
The core is UI-agnostic, so it's easy to wrap in a web app or another tool later.
- Repeating-structure detection. Every element's direct children are
grouped by a
(tag, classes)signature. Each group is scored by how many times it repeats and how much internal structure each member has. The highest-scoring group becomes the record set, and its shared class (or a parent + child-combinator) becomes therecord_selector. Trivial repeats (nav links,<br>, bare list items) are filtered out. - Per-record field inference. Within the first record the engine looks for
a heading (title), the first link (
href), the first image (src), a price-shaped value,itemprop-marked nodes, and any remaining classed text. Each candidate selector is validated against all records so it generalizes. - Tables are special-cased: columns are named from
<th>headers and addressed withtd:nth-of-type(n); a link inside a cell becomes its own*_urlfield. - Detail pages (no strong repeating group) fall back to page-level fields and schema.org JSON-LD, which is flattened into dotted paths.
- It infers structure from one sample. Optional fields that don't appear in your sample won't be inferred; feed it a representative page.
- Selectors favor classes and semantic hooks. On sites with hashed/utility CSS classes (e.g. some Tailwind builds) it falls back to positional selectors, which are more brittle — review the output.
- Type detection (
number/url/date) is a best-effort heuristic. - It parses the HTML you give it; it does not execute JavaScript. For client-rendered pages, capture the rendered HTML first.
The python-web-scraping.com guides pick up where the generated skeleton leaves off:
- Parsing HTML with BeautifulSoup — the selectors this tool emits, explained end to end.
- Extracting JSON-LD & structured data — how the JSON-LD fields are found and parsed.
- Scraping schema.org product data — turning
offers.priceand friends into clean records. - Step-by-step guide to extracting tables from HTML — the table-detection path in depth.
- Web scraping with Scrapy — dropping the generated spider into a real Scrapy project.
git clone https://github.com/python-web-scraping-com/html-to-schema.git
cd html-to-schema
python -m venv .venv && . .venv/bin/activate
pip install -e ".[dev]"
pytestThe test suite is fully offline and deterministic: it runs against local HTML fixtures and even execs the generated extractors to confirm they recover the inferred example values (a real round-trip). No network access is required.
MIT — see LICENSE.