core: speed up the analysis catalog with prepared statements - #4534
Merged
Conversation
The core catalog backs every lookup with a query against its in-memory
SQLite database, and handed the raw *sql.DB to the generated queries. Go's
database/sql prepares and discards a statement per call, so SQLite re-parsed
and re-planned the same dozen queries thousands of times per run: profiling
analyzer.Prepare put 65% of samples inside sqlite3Parser and 85% under
database/sql dispatch, with the analyzer itself barely on the profile.
- Prepare each catalog statement once and rebind it, for both the seed path
and the analyzer's lookups.
- Install dialect seeds in a single transaction, and drop journalling for a
database that is rebuilt from scratch on every run.
- Pin the pool to one connection. Each ":memory:" connection is its own
empty database, so a second one would have seen a catalog with no tables
in it — a latent bug under any concurrent use.
On the analyzer side, resolving a column no longer allocates a candidate
slice per reference, the scope holds the catalog's column list instead of
copying it into a scope-local type, a target's field list is flattened once
instead of three times, and parameters are tracked by value.
Analyzing 500 GoogleSQL queries drops from 181ms to 82ms end to end, and 100
queries from 58ms to 34ms, with byte-identical output on GoogleSQL and
ClickHouse. Microbenchmarks (internal/core/analyzer/bench_test.go):
CatalogNew 6001532 ns/op (catalog + dialect seed)
CatalogNew 2863319 ns/op
Prepare 1579238 ns/op (5 statements, warm catalog)
Prepare 398200 ns/op
Lookup volume is unchanged at 13 catalog queries per statement, and analysis
still allocates ~400 times per statement — nearly all of it database/sql
scanning a row per lookup. Memoizing lookups removes both, but needs cache
invalidation and a read-only contract on the slices the catalog returns;
that is not worth it for an in-memory database, so it is left out.
Behaviour is covered by the existing end-to-end tests: analyze_basic,
analyze_select and analyze_dml exercise both dialects through TestReplay.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fp7UGoiFnUwK9ZkDBoGy7u
kyleconroy
force-pushed
the
claude/core-analyzer-performance-2ek5s7
branch
from
July 31, 2026 01:57
d4d0503 to
1cd9419
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A performance pass on the new core analyzer (
internal/core), the one backing ClickHouse and GoogleSQL.What was slow
The core catalog backs every lookup with a query against its in-memory SQLite database, and handed the raw
*sql.DBto the generatedcatalogdbqueries.database/sqlprepares and discards a statement per call, so SQLite re-parsed and re-planned the same dozen queries thousands of times per run.Profiling
analyzer.Prepareput 65% of samples insidesqlite3Parserand 85% underdatabase/sqldispatch — the analyzer itself was barely on the profile. Every column reference, type name, operator resolution and attribute decoration was a round trip that recompiled its SQL from text.Changes
":memory:"connection is its own empty database, so a second one would have seen a catalog with no tables in it — a latent bug under any concurrent use.Results
Analyzing a generated project of 20 tables and 5 statement shapes per table, best of 9 interleaved runs:
sqlc analyze, 100 queriessqlc analyze, 500 queriesOutput is byte-identical to
mainon both GoogleSQL and ClickHouse.What this deliberately does not do
Lookup volume is unchanged at 13 catalog queries per statement, and analysis still allocates ~400 times per statement — nearly all of it
database/sqlscanning a row per lookup, which prepared statements don't touch.Memoizing lookups removes both (0 queries, 48 allocations per 5 statements, another 2× end to end). It was prototyped and dropped: it needs cache invalidation on every mutation plus an unwritten contract that callers must not mutate the slices
ClassColumns,FindOperatorsandFindProcshand back. That didn't seem worth it for lookups against an in-memory database. Easy to revisit if the core path's allocation profile ever matters.For context, per query excluding parsing, the core analyzer now sits at ~122 µs against ~80 µs for the original PostgreSQL path and ~52 µs for MySQL — still the slower path, down from ~234 µs before this change.
Tests
No new tests. Behaviour is covered by the existing end-to-end tests —
analyze_basic,analyze_selectandanalyze_dmlexercise both dialects throughTestReplay, which passes, as does the full suite.internal/core/analyzer/bench_test.goadds the benchmarks quoted above. It contains no tests, only benchmarks, so it does not run undergo test.