From 09d70a6c761a14e2a3ccaa6b64784867bd5e2221 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 30 Jul 2026 12:52:05 +1000 Subject: [PATCH 1/4] fix(stack-drizzle): preserve per-column types in extractEncryptionSchema MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `extractEncryptionSchema` returned the widened `AnyV3Table`, discarding every column's concrete EQL v3 domain. `InferPlaintext` over an extracted schema therefore collapsed to an index signature, so along the documented flow — `extractEncryptionSchema(table)` -> `Encryption({ schemas })` -> `bulkEncryptModels` — a `string` written to an `IntegerOrd` column type-checked, and plain helper columns (`id`, a plain `text()`) were typed as encrypted. Type-safety/DX only: the runtime already recovered each builder correctly via `getEqlV3Column`. Three linked pieces, all in `@cipherstash/stack-drizzle` — the v3 core needed no change: - `makeEqlV3Column` carries its builder `C` on a phantom brand. The brand rides on the column's `_` config, not the builder object: `pgTable()` rebuilds each builder through drizzle's `BuildColumn`, which keeps only `Omit>` — drizzle's own extension point (`NotNull = T & { _: { notNull: true } }`). A brand on the object would be dropped and extraction would have nothing to read. - `extractEncryptionSchema` is generic over the table and returns `EncryptedTable & Cols`, the shape `encryptedTable()` itself returns. `EncryptedColumnsOf` recovers each brand and keeps only the branded columns — the type-level mirror of the existing runtime loop, which is unchanged. - `V3BuilderOf` short-circuits `any` to `AnyEncryptedV3Column`, so an untyped table keeps the widened-but-usable typing it had before rather than branding every column `any`. `__tests__/schema-extraction.test-d.ts` locks both paths: extracted and hand-authored must infer the same plaintext map, reject a field typed against the wrong domain, and pin `encrypt` to the addressed column. Six of its eight assertions fail without this fix. Two integration files relied on the old looseness and are corrected, not worked around: the dynamically-built matrix table names its model type explicitly (its rows are a pure index signature, so inference dropped the `| null` that `nullableTextEq` needs), and the bench comment claiming extraction returns `AnyV3Table` is now false and says so. Closes #589 --- .changeset/typed-extract-encryption-schema.md | 18 ++ packages/bench/src/drizzle/setup.ts | 9 +- .../__tests__/schema-extraction.test-d.ts | 165 ++++++++++++++++++ .../relational.integration.test.ts | 13 +- packages/stack-drizzle/src/column.ts | 121 ++++++++++--- .../stack-drizzle/src/schema-extraction.ts | 47 ++++- skills/stash-drizzle/SKILL.md | 2 + 7 files changed, 346 insertions(+), 29 deletions(-) create mode 100644 .changeset/typed-extract-encryption-schema.md create mode 100644 packages/stack-drizzle/__tests__/schema-extraction.test-d.ts diff --git a/.changeset/typed-extract-encryption-schema.md b/.changeset/typed-extract-encryption-schema.md new file mode 100644 index 000000000..e0967c60a --- /dev/null +++ b/.changeset/typed-extract-encryption-schema.md @@ -0,0 +1,18 @@ +--- +'@cipherstash/stack-drizzle': minor +'stash': patch +--- + +Type `extractEncryptionSchema` precisely: a Drizzle-extracted schema now preserves each column's concrete EQL v3 domain instead of widening to `AnyV3Table` (#589). + +`extractEncryptionSchema` is generic over the Drizzle table (`(table: T)`) and returns `EncryptedTable & Cols`, the same shape a hand-written `encryptedTable({...})` returns. Each column's builder is carried through `pgTable()` on a phantom brand and recovered by a mapped type, which also filters out the table's non-encrypted columns. + +What this fixes, along the documented flow `extractEncryptionSchema(table)` → `Encryption({ schemas })` → `bulkEncryptModels`: + +- `InferPlaintext` is a precise per-column plaintext map (`{ email: string; age: number }`) rather than an index signature. +- `encryptModel` / `bulkEncryptModels` check each schema field against its own domain's plaintext — a `string` written to an `IntegerOrd` column is now a compile error instead of an encrypt-time failure — and pass plain helper columns (`id`, a plain `text()`) through with their own types rather than typing them as encrypted. +- `schema.email` addresses the column at its concrete type, so `encrypt` / `encryptQuery` pin the value to that column's plaintext. + +**Runtime behaviour is unchanged** — the runtime already recovered each column's builder correctly, so this is a type-level fix only. It is `minor` rather than `patch` because code that previously compiled against the widened types can now fail to compile: a model field typed against the wrong domain, or a schema-derived type that relied on the old index signature. Rows whose shape is only known at runtime (a dynamically built table) should name their model type explicitly — `client.bulkEncryptModels(rows, schema)` — rather than being cast back to `AnyV3Table`. + +`skills/stash-drizzle` documents the preserved typing and warns against casting an extracted schema to `AnyV3Table` to make an insert compile. diff --git a/packages/bench/src/drizzle/setup.ts b/packages/bench/src/drizzle/setup.ts index b91d2ef0d..6f5bae123 100644 --- a/packages/bench/src/drizzle/setup.ts +++ b/packages/bench/src/drizzle/setup.ts @@ -41,10 +41,11 @@ export const encryptionBenchTable = extractEncryptionSchema(benchTable) * so `__unit__/seed-keys.test.ts` is the real drift guard, not the type. * Drizzle's `InferInsertModel` describes the ENCRYPTED column (the custom type's * `data` is the EQL envelope, not the plaintext) and degrades these three to - * optional `any`. Deriving from `encryptionBenchTable` is no better: - * `extractEncryptionSchema` returns the widened `AnyV3Table`, whose column map - * is an index signature — a type that admits `encTxt: 'x'` and `encText: 12345` - * alike, both of which the literal below rejects. + * optional `any`. `InferPlaintext` is precise per + * column since #589, and it does pin `encText`/`encInt` — but `encJsonb` is a + * `types.Json` column, whose plaintext is the whole `JsonDocument` union. The + * literal below says `{ idx: number; group: number }`, which is what the bench + * queries actually assume, so deriving would still lose the shape that matters. * * Update it by hand when `benchTable` changes; the unit test will tell you. */ diff --git a/packages/stack-drizzle/__tests__/schema-extraction.test-d.ts b/packages/stack-drizzle/__tests__/schema-extraction.test-d.ts new file mode 100644 index 000000000..3b6f91921 --- /dev/null +++ b/packages/stack-drizzle/__tests__/schema-extraction.test-d.ts @@ -0,0 +1,165 @@ +/** + * Type-level contract for {@link extractEncryptionSchema} (#589). + * + * A Drizzle-extracted schema must be typed as precisely as the hand-authored + * `encryptedTable({…})` it is equivalent to. Before #589 it was not: extraction + * returned the widened `AnyV3Table`, so `InferPlaintext` over an extracted + * schema collapsed to an index signature and every column on the insert path + * accepted (and produced) the union of every domain's plaintext instead of its + * own. There was no runtime bug — the runtime already recovers each column's + * concrete builder — which is exactly why this needs a type test to hold it. + * + * Every assertion below is written to be equally meaningful on BOTH paths, so + * the hand-authored surface cannot regress while the extracted one is fixed. + */ + +import { + type EncryptedIntegerOrdColumn, + type EncryptedTextEqColumn, + encryptedTable, + type InferPlaintext, + types as v3Types, +} from '@cipherstash/stack/eql/v3' +import type { Encrypted } from '@cipherstash/stack/types' +import { Encryption, type EncryptionClient } from '@cipherstash/stack/v3' +import { pgTable, serial, text } from 'drizzle-orm/pg-core' +import { describe, expectTypeOf, it } from 'vitest' +import { extractEncryptionSchema } from '../src/schema-extraction' +import { types } from '../src/types' + +// `id` and `nickname` are load-bearing, not scenery: extraction must keep only +// the encrypted (branded) columns, so the mapped type has to filter them out. +// They also cover the other half of the insert contract — a plain helper column +// must pass through the model types untouched rather than being treated as an +// encrypted one. +const users = pgTable('users', { + id: serial('id').primaryKey(), + nickname: text('nickname'), + email: types.TextEq('email'), + age: types.IntegerOrd('age'), +}) + +/** The hand-authored equivalent of `users`. Same columns, same domains. */ +const authored = encryptedTable('users', { + email: v3Types.TextEq('email'), + age: v3Types.IntegerOrd('age'), +}) + +const extracted = extractEncryptionSchema(users) + +/** The plaintext map both schemas must infer. */ +type UserPlaintext = { email: string; age: number } + +/** The model handed to `bulkEncryptModels`, plain helper columns included. */ +const plainRows = [ + { id: 1, nickname: 'ada', email: 'ada@example.com', age: 36 }, +] + +/** + * What `bulkEncryptModels` must resolve: the two encrypted columns become + * envelopes, the two plain ones pass through with their own types. + */ +type EncryptedUserRow = { + id: number + nickname: string + email: Encrypted + age: Encrypted +} + +describe('extractEncryptionSchema - plaintext inference (#589)', () => { + it('infers the same precise plaintext map as the hand-authored table', () => { + expectTypeOf< + InferPlaintext + >().toEqualTypeOf() + expectTypeOf< + InferPlaintext + >().toEqualTypeOf() + }) + + it('drops the table’s plain, non-encrypted columns', () => { + // The `toEqualTypeOf` above already forbids extra keys; this states the + // filtering intent directly so a failure names the actual cause. + expectTypeOf>().toEqualTypeOf< + 'email' | 'age' + >() + }) + + it('keeps each column addressable at its concrete domain type', () => { + // `encryptedTable()` returns `EncryptedTable & C`, so the schema doubles + // as a column accessor. Extraction returns the same intersection — that is + // what lets `client.encrypt(v, { table, column: schema.email })` pin `v` to + // the column's own plaintext rather than the union of every domain's. + expectTypeOf(extracted.email).toEqualTypeOf() + expectTypeOf(extracted.age).toEqualTypeOf() + expectTypeOf(authored.email).toEqualTypeOf() + expectTypeOf(authored.age).toEqualTypeOf() + }) +}) + +/** + * The documented Drizzle flow, end to end: + * `extractEncryptionSchema(table)` -> `Encryption({ schemas })` -> `bulkEncryptModels`. + * + * This is the path the issue reports as widened, so it is asserted on the real + * factory rather than on a hand-named client type — the whole point is that the + * concrete tuple survives from extraction through to the model types. + * + * These `it` bodies are type-checked, never executed: vitest's typecheck mode + * does not run `.test-d.ts` files, and they are outside the runtime suite's + * `include`. So `Encryption()` here needs no credentials. + */ +describe('extractEncryptionSchema - documented flow (#589)', () => { + it('carries the extracted schema into the client tuple', async () => { + const client = await Encryption({ schemas: [extracted] }) + expectTypeOf(client).toEqualTypeOf< + EncryptionClient + >() + }) + + it('types bulkEncryptModels precisely against an extracted schema', async () => { + const client = await Encryption({ schemas: [extracted] }) + const result = await client.bulkEncryptModels(plainRows, extracted) + if (result.failure) throw result.failure + + expectTypeOf(result.data).toEqualTypeOf() + }) + + it('types bulkEncryptModels precisely against the hand-authored schema', async () => { + const client = await Encryption({ schemas: [authored] }) + const result = await client.bulkEncryptModels(plainRows, authored) + if (result.failure) throw result.failure + + expectTypeOf(result.data).toEqualTypeOf() + }) + + it('rejects a field typed against the wrong domain on either path', async () => { + const onExtracted = await Encryption({ schemas: [extracted] }) + const onAuthored = await Encryption({ schemas: [authored] }) + const wrong = [{ email: 'ada@example.com', age: 'thirty-six' }] + + // This is the assertion with teeth. While extraction returned `AnyV3Table`, + // every column's plaintext was the union of every domain's, so a `string` + // in an `IntegerOrd` column type-checked and the mismatch only surfaced at + // encrypt time. If that widening ever returns, the suppression below goes + // unused and this test fails. + // @ts-expect-error - `age` is IntegerOrd: its plaintext is number, not string + onExtracted.bulkEncryptModels(wrong, extracted) + // @ts-expect-error - the hand-authored path has always rejected this; it must keep doing so + onAuthored.bulkEncryptModels(wrong, authored) + }) + + it('pins encrypt() to the addressed column’s own plaintext type', async () => { + const client = await Encryption({ schemas: [extracted] }) + + expectTypeOf(client.encrypt).toBeCallableWith('ada@example.com', { + table: extracted, + column: extracted.email, + }) + expectTypeOf(client.encrypt).toBeCallableWith(36, { + table: extracted, + column: extracted.age, + }) + // @ts-expect-error - the email column encrypts a string, not a number + client.encrypt(36, { table: extracted, column: extracted.email }) + }) +}) diff --git a/packages/stack-drizzle/integration/relational.integration.test.ts b/packages/stack-drizzle/integration/relational.integration.test.ts index 98df8335e..361c7969a 100644 --- a/packages/stack-drizzle/integration/relational.integration.test.ts +++ b/packages/stack-drizzle/integration/relational.integration.test.ts @@ -265,8 +265,19 @@ beforeAll(async () => { ) `) + // The model type is supplied explicitly. `MatrixPlainRow` is a pure index + // signature — this table's columns are generated from `V3_MATRIX` at runtime, + // so there are no statically-known keys to infer from — and inference would + // otherwise settle on the non-null `PlainValue`, which `nullableTextEq` + // (deliberately `null` on ROW_A) then fails. Naming `MatrixPlainRow` lets + // `V3ModelInput`'s nullability branch see the `| null` that is actually there. const encryptedRows = assertScopeKeys( - unwrapResult(await client.bulkEncryptModels(encryptedInsertRows(), schema)), + unwrapResult( + await client.bulkEncryptModels( + encryptedInsertRows(), + schema, + ), + ), ) await db.insert(matrixTable).values(encryptedRows) await db.insert(accountsTable).values([ diff --git a/packages/stack-drizzle/src/column.ts b/packages/stack-drizzle/src/column.ts index efad7635c..af408f0c3 100644 --- a/packages/stack-drizzle/src/column.ts +++ b/packages/stack-drizzle/src/column.ts @@ -125,7 +125,102 @@ function getSqlType(column: unknown): string | undefined { return typeof domain === 'string' ? qualifyDomain(domain) : undefined } -export function makeEqlV3Column(builder: C) { +/** + * Build the bare Drizzle carrier column for an EQL v3 domain. + * + * Split out of {@link makeEqlV3Column} for one reason: it gives the Drizzle + * builder type a NAME (`ReturnType`) without this + * module having to restate `customType`'s deeply-generic return type by hand — + * which would silently rot the day drizzle-orm changes it. + */ +function makeCarrierColumn(sqlType: string, name: string) { + // What is stored/inserted/selected is the ENCRYPTED EQL v3 jsonb envelope + // (produced by `client.encrypt` / `bulkEncryptModels`), NOT the column's + // plaintext. So `data` is the envelope type — an insert takes an already- + // encrypted `Encrypted`, and a select yields one, ready for `decryptModel`. + return customType<{ data: Encrypted; driverData: string | null }>({ + dataType() { + return sqlType + }, + toDriver(value: Encrypted): string | null { + return v3ToDriver(value) + }, + fromDriver(value: string | object | null | undefined): Encrypted { + // A present jsonb value round-trips to an envelope; the driver only + // reaches here for non-null values, so the SQL-NULL branch is a safety + // net rather than a live path (the boundary cast covers it). + return v3FromDriver(value) as Encrypted + }, + })(name) +} + +/** + * Key for the phantom v3-builder carrier on {@link EqlV3Column}. Declared, never + * defined — it exists only in the type layer. Deliberately NOT exported, so the + * brand is unnameable (and so uncounterfeitable) outside this module while + * staying just as inferrable through {@link V3BuilderOf}. Same trick, and the + * same reasoning, as the core's `domainCarrier` on `EncryptedV3Column`. + */ +declare const v3BuilderCarrier: unique symbol + +/** The unbranded Drizzle column {@link makeCarrierColumn} produces. */ +type CarrierColumn = ReturnType + +/** + * A Drizzle column for the concrete v3 builder `C`, carrying `C` at the type + * level so {@link V3BuilderOf} can recover it downstream. + * + * The brand rides on the builder's `_` config, NOT on the builder object type, + * because `pgTable()` does not hand its column *builders* through to the table — + * it rebuilds each one via drizzle's `BuildColumn`, which keeps only `_`. (The + * surviving part is `Omit>`, drizzle's + * own extension point — `NotNull = T & { _: { notNull: true } }` uses exactly + * this shape.) A brand on the builder object would be dropped by `pgTable()` and + * `extractEncryptionSchema` would have nothing left to read. + * + * The property is REQUIRED, not optional: an optional phantom is structurally + * satisfied by every column, so `types.TextEq(…)` and a plain `text()` would + * both match the recovery conditional and the non-encrypted columns could not be + * filtered out. Nothing constructs this property at runtime — it is type-only, + * which is why {@link makeEqlV3Column} asserts its return. + */ +export type EqlV3Column = CarrierColumn & { + _: { [v3BuilderCarrier]: C } +} + +/** + * `true` only for `any`. The usual idiom: `1 & T` collapses to `any` when `T` is + * `any` (and `0 extends any` holds), and to `1`/`never`/an intersection for + * every other `T` (where `0 extends …` does not). + */ +type IsAny = 0 extends 1 & T ? true : false + +/** + * Recover the concrete v3 builder branded onto a Drizzle column (or column + * builder) by {@link makeEqlV3Column}. Resolves to `never` for a plain, + * non-encrypted Drizzle column — which is what lets a mapped type filter a + * table's columns down to just the encrypted ones. The type-level mirror of + * {@link getEqlV3Column}'s runtime recovery. + * + * An `any` column short-circuits to the widened `AnyEncryptedV3Column` rather + * than falling through the conditional. A conditional type over `any` resolves + * to the UNION of both branches with `infer C` bound to `any`, so without this + * guard an untyped table (`const table: any`, the shape a dynamically-built + * adapter table has) would brand every column `any` and degrade the whole + * schema instead of landing on the widened-but-usable type it had before #589. + */ +export type V3BuilderOf = + IsAny extends true + ? AnyEncryptedV3Column + : Col extends { _: { [v3BuilderCarrier]: infer C } } + ? C extends AnyEncryptedV3Column + ? C + : never + : never + +export function makeEqlV3Column( + builder: C, +): EqlV3Column { const domain = builder.getEqlType() const name = builder.getName() @@ -156,28 +251,14 @@ export function makeEqlV3Column(builder: C) { } const sqlType = stripDomainSchema(domain) - // What is stored/inserted/selected is the ENCRYPTED EQL v3 jsonb envelope - // (produced by `client.encrypt` / `bulkEncryptModels`), NOT the column's - // plaintext. So `data` is the envelope type — an insert takes an already- - // encrypted `Encrypted`, and a select yields one, ready for `decryptModel`. - const column = customType<{ data: Encrypted; driverData: string | null }>({ - dataType() { - return sqlType - }, - toDriver(value: Encrypted): string | null { - return v3ToDriver(value) - }, - fromDriver(value: string | object | null | undefined): Encrypted { - // A present jsonb value round-trips to an envelope; the driver only - // reaches here for non-null values, so the SQL-NULL branch is a safety - // net rather than a live path (the boundary cast covers it). - return v3FromDriver(value) as Encrypted - }, - })(name) + const column = makeCarrierColumn(sqlType, name) writeBuilder(getCarrier(column), builder) writeBuilder(column as unknown as EqlV3ColumnCarrier, builder) - return column + // The brand is type-only — no runtime property backs it (see EqlV3Column) — + // so the carrier column is narrowed to the branded type here. The runtime + // carrier (`writeBuilder` above) is what `getEqlV3Column` actually reads. + return column as EqlV3Column } export function getEqlV3Column( diff --git a/packages/stack-drizzle/src/schema-extraction.ts b/packages/stack-drizzle/src/schema-extraction.ts index 7e0b4226f..c44d5412d 100644 --- a/packages/stack-drizzle/src/schema-extraction.ts +++ b/packages/stack-drizzle/src/schema-extraction.ts @@ -1,10 +1,10 @@ import { type AnyEncryptedV3Column, - type AnyV3Table, + type EncryptedTable, encryptedTable, } from '@cipherstash/stack/eql/v3' import type { PgTable } from 'drizzle-orm/pg-core' -import { getEqlV3Column } from './column.js' +import { getEqlV3Column, type V3BuilderOf } from './column.js' /** Drizzle stashes the SQL table name on this well-known symbol key. */ const DRIZZLE_NAME = Symbol.for('drizzle:Name') @@ -21,7 +21,37 @@ export function getDrizzleTableName(table: unknown): string | undefined { return typeof name === 'string' ? name : undefined } -export function extractEncryptionSchema(table: PgTable): AnyV3Table { +/** + * The v3 column map a Drizzle table carries: every branded encrypted column, + * keyed by its JS property name and mapped to the concrete v3 builder recovered + * from its brand; every other property (plain columns, and the `PgTable` + * members drizzle mixes in) dropped. + * + * This is the type-level mirror of {@link extractEncryptionSchema}'s runtime + * loop, and the whole reason extraction can be precisely typed: without it the + * return collapses to the widened `AnyV3Table`, and `InferPlaintext` over an + * extracted schema degrades to an index signature instead of a per-column + * plaintext map (#589). + */ +export type EncryptedColumnsOf = { + [K in keyof T as [V3BuilderOf] extends [never] + ? never + : K]: V3BuilderOf +} + +/** + * Rebuild a Drizzle table's encrypted columns as an eql/v3 {@link EncryptedTable}. + * + * The return type mirrors what `encryptedTable()` itself returns — + * `EncryptedTable & Cols` — so the result is both a schema for + * `Encryption({ schemas })` and a column accessor (`schema.email`), with each + * column's concrete domain preserved. That is what keeps `InferPlaintext` / + * `encryptModel` / `bulkEncryptModels` precisely typed against an extracted + * schema. + */ +export function extractEncryptionSchema( + table: T, +): EncryptedTable> & EncryptedColumnsOf { const tableName = getDrizzleTableName(table) if (!tableName) { throw new Error( @@ -46,5 +76,14 @@ export function extractEncryptionSchema(table: PgTable): AnyV3Table { ) } - return encryptedTable(tableName, columns) + // The runtime loop above is untyped by construction — it reads properties off + // a `PgTable` and recovers each builder dynamically — so the precise column + // map only exists at the type level (`EncryptedColumnsOf`). Narrow the + // widened runtime value to it here; the two are kept in step by the + // `.test-d.ts` assertions, which compare an extracted schema against the + // hand-authored `encryptedTable({…})` it must be equivalent to. + return encryptedTable(tableName, columns) as EncryptedTable< + EncryptedColumnsOf + > & + EncryptedColumnsOf } diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index 63d1c3ba8..1cb10347b 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -131,6 +131,8 @@ import { Encryption } from "@cipherstash/stack/v3" const usersSchema = extractEncryptionSchema(usersTable) ``` +The extracted schema keeps each column's concrete domain, so it types exactly like a hand-written `encryptedTable({...})`: `InferPlaintext` is a precise per-column plaintext map, non-encrypted columns (`id`, plain `text()` helpers) are excluded from it and pass through model operations untouched, and `usersSchema.email` addresses the column at its own type. Do not cast an extracted schema to `AnyV3Table` or annotate model rows with `Record` to make an insert compile — that discards the checking and hides real mismatches. + ### 2. Initialize the Encryption Client ```typescript From dd653652e19ef67be29765377ceb5593f67fa19f Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 30 Jul 2026 13:43:25 +1000 Subject: [PATCH 2/4] fix(stack-drizzle): retain widened extraction types --- .changeset/typed-extract-encryption-schema.md | 2 +- .../__tests__/schema-extraction.test-d.ts | 52 ++++++++++++++++++- .../stack-drizzle/src/schema-extraction.ts | 27 ++++++---- skills/stash-drizzle/SKILL.md | 7 +++ 4 files changed, 76 insertions(+), 12 deletions(-) diff --git a/.changeset/typed-extract-encryption-schema.md b/.changeset/typed-extract-encryption-schema.md index e0967c60a..8a1dfbc32 100644 --- a/.changeset/typed-extract-encryption-schema.md +++ b/.changeset/typed-extract-encryption-schema.md @@ -5,7 +5,7 @@ Type `extractEncryptionSchema` precisely: a Drizzle-extracted schema now preserves each column's concrete EQL v3 domain instead of widening to `AnyV3Table` (#589). -`extractEncryptionSchema` is generic over the Drizzle table (`(table: T)`) and returns `EncryptedTable & Cols`, the same shape a hand-written `encryptedTable({...})` returns. Each column's builder is carried through `pgTable()` on a phantom brand and recovered by a mapped type, which also filters out the table's non-encrypted columns. +`extractEncryptionSchema` is generic over the Drizzle table (`(table: T)`) and returns `EncryptedTable & Cols`, the same shape a hand-written `encryptedTable({...})` returns, when concrete column brands are available. Each column's builder is carried through `pgTable()` on a phantom brand and recovered by a mapped type, which also filters out the table's non-encrypted columns. Tables widened to `PgTable`, and ordinary `customType` columns recovered from their EQL SQL domain, retain the safe `AnyV3Table` fallback instead of incorrectly becoming an empty schema type. What this fixes, along the documented flow `extractEncryptionSchema(table)` → `Encryption({ schemas })` → `bulkEncryptModels`: diff --git a/packages/stack-drizzle/__tests__/schema-extraction.test-d.ts b/packages/stack-drizzle/__tests__/schema-extraction.test-d.ts index 3b6f91921..792270546 100644 --- a/packages/stack-drizzle/__tests__/schema-extraction.test-d.ts +++ b/packages/stack-drizzle/__tests__/schema-extraction.test-d.ts @@ -14,6 +14,7 @@ */ import { + type AnyV3Table, type EncryptedIntegerOrdColumn, type EncryptedTextEqColumn, encryptedTable, @@ -22,7 +23,13 @@ import { } from '@cipherstash/stack/eql/v3' import type { Encrypted } from '@cipherstash/stack/types' import { Encryption, type EncryptionClient } from '@cipherstash/stack/v3' -import { pgTable, serial, text } from 'drizzle-orm/pg-core' +import { + customType, + type PgTable, + pgTable, + serial, + text, +} from 'drizzle-orm/pg-core' import { describe, expectTypeOf, it } from 'vitest' import { extractEncryptionSchema } from '../src/schema-extraction' import { types } from '../src/types' @@ -47,6 +54,22 @@ const authored = encryptedTable('users', { const extracted = extractEncryptionSchema(users) +// Consumers can legitimately erase a table's concrete column shape at module +// boundaries. Runtime extraction still sees the columns in this case, but the +// private type brand is no longer recoverable. +const widenedUsers: PgTable = users +const widenedExtracted = extractEncryptionSchema(widenedUsers) + +// Runtime extraction also supports ordinary Drizzle custom columns by their +// EQL SQL domain, even though those columns never carried our private brand. +const eqlTextEq = customType<{ data: Encrypted }>({ + dataType: () => 'eql_v3_text_eq', +}) +const sqlTypeFallbackUsers = pgTable('sql_type_fallback_users', { + email: eqlTextEq('email'), +}) +const fallbackExtracted = extractEncryptionSchema(sqlTypeFallbackUsers) + /** The plaintext map both schemas must infer. */ type UserPlaintext = { email: string; age: number } @@ -96,6 +119,33 @@ describe('extractEncryptionSchema - plaintext inference (#589)', () => { }) }) +describe('extractEncryptionSchema - unbranded table fallback', () => { + it('preserves the widened schema type for a table annotated as PgTable', () => { + expectTypeOf(widenedExtracted).toEqualTypeOf() + expectTypeOf>().toEqualTypeOf< + string | number + >() + }) + + it('preserves the widened schema type for columns recovered by SQL type', () => { + expectTypeOf(fallbackExtracted).toEqualTypeOf() + expectTypeOf< + keyof InferPlaintext + >().toEqualTypeOf() + }) + + it('still types fields as encrypted through widened model operations', async () => { + const client = await Encryption({ schemas: [widenedExtracted] }) + const result = await client.bulkEncryptModels( + [{ email: 'ada@example.com' }], + widenedExtracted, + ) + if (result.failure) throw result.failure + + expectTypeOf(result.data).toEqualTypeOf>() + }) +}) + /** * The documented Drizzle flow, end to end: * `extractEncryptionSchema(table)` -> `Encryption({ schemas })` -> `bulkEncryptModels`. diff --git a/packages/stack-drizzle/src/schema-extraction.ts b/packages/stack-drizzle/src/schema-extraction.ts index c44d5412d..ab92ecd87 100644 --- a/packages/stack-drizzle/src/schema-extraction.ts +++ b/packages/stack-drizzle/src/schema-extraction.ts @@ -1,5 +1,6 @@ import { type AnyEncryptedV3Column, + type AnyV3Table, type EncryptedTable, encryptedTable, } from '@cipherstash/stack/eql/v3' @@ -39,6 +40,16 @@ export type EncryptedColumnsOf = { : K]: V3BuilderOf } +/** + * Keep concrete branded columns precise, but retain the pre-existing widened + * schema type when TypeScript cannot see any brands. The runtime extractor can + * still recover columns from a widened `PgTable` or from their EQL SQL domain; + * representing those successful paths as an empty table would be unsound. + */ +type ExtractedEncryptionSchema = keyof EncryptedColumnsOf extends never + ? AnyV3Table + : EncryptedTable> & EncryptedColumnsOf + /** * Rebuild a Drizzle table's encrypted columns as an eql/v3 {@link EncryptedTable}. * @@ -51,7 +62,7 @@ export type EncryptedColumnsOf = { */ export function extractEncryptionSchema( table: T, -): EncryptedTable> & EncryptedColumnsOf { +): ExtractedEncryptionSchema { const tableName = getDrizzleTableName(table) if (!tableName) { throw new Error( @@ -77,13 +88,9 @@ export function extractEncryptionSchema( } // The runtime loop above is untyped by construction — it reads properties off - // a `PgTable` and recovers each builder dynamically — so the precise column - // map only exists at the type level (`EncryptedColumnsOf`). Narrow the - // widened runtime value to it here; the two are kept in step by the - // `.test-d.ts` assertions, which compare an extracted schema against the - // hand-authored `encryptedTable({…})` it must be equivalent to. - return encryptedTable(tableName, columns) as EncryptedTable< - EncryptedColumnsOf - > & - EncryptedColumnsOf + // a `PgTable` and recovers each builder dynamically. Narrow to the branded + // column map when one is visible at the type level, or to `AnyV3Table` when + // runtime recovery succeeded without visible brands. The `.test-d.ts` + // assertions keep both paths in step with the runtime behavior. + return encryptedTable(tableName, columns) as ExtractedEncryptionSchema } diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index 1cb10347b..225bc46e8 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -133,6 +133,13 @@ const usersSchema = extractEncryptionSchema(usersTable) The extracted schema keeps each column's concrete domain, so it types exactly like a hand-written `encryptedTable({...})`: `InferPlaintext` is a precise per-column plaintext map, non-encrypted columns (`id`, plain `text()` helpers) are excluded from it and pass through model operations untouched, and `usersSchema.email` addresses the column at its own type. Do not cast an extracted schema to `AnyV3Table` or annotate model rows with `Record` to make an insert compile — that discards the checking and hides real mismatches. +That precision depends on passing the concrete table type produced by `pgTable()`. +If a table has already been widened to `PgTable`, or its encrypted columns are +ordinary `customType` columns detected from their EQL SQL domain, extraction +returns the safe widened `AnyV3Table` type because the concrete domain brands +are no longer available to TypeScript. Runtime extraction still discovers and +uses those encrypted columns. + ### 2. Initialize the Encryption Client ```typescript From 1def19b9c982621cc52f4e78f64986b7e9ef4a51 Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 30 Jul 2026 13:50:55 +1000 Subject: [PATCH 3/4] test(stack-drizzle): cover modifier survival and the json domain MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two gaps in the #589 type-level suite, both of which would have shipped green. Drizzle column modifiers. Every existing fixture declares bare columns, but real schemas almost always use `.notNull()`. The brand survives `pgTable()` only because drizzle's `BuildColumn` keeps `Omit | 'brand' | 'dialect'>`, and modifiers then intersect further into `_` (`NotNull = T & { _: { notNull: true } }`). That is an upstream compatibility contract, not one we control: a drizzle-orm change to either step could drop the brand for modified columns while bare ones kept working. The runtime would not notice — `getEqlV3Column` recovers builders from the runtime carrier and the SQL domain, never from the type — so extraction would silently type a `.notNull()` column as a plain pass-through field while still encrypting it. Verified the guard isolates that class: narrowing `V3BuilderOf` to match only `notNull: false` fails exactly the three new modifier assertions and leaves all fourteen pre-existing ones green. The json domain. `types.Json` is the only domain whose plaintext is a union (`JsonDocument`) rather than a scalar, which makes it the easiest to lose to a widened index signature without another assertion noticing. Covered on both the extracted and hand-authored paths, including a `@ts-expect-error` on a scalar written to a json column. Both blocks fail against the pre-fix source, so they cover the fix rather than passing incidentally. Test-only; the existing changeset already covers the behaviour. --- .../__tests__/schema-extraction.test-d.ts | 112 ++++++++++++++++++ 1 file changed, 112 insertions(+) diff --git a/packages/stack-drizzle/__tests__/schema-extraction.test-d.ts b/packages/stack-drizzle/__tests__/schema-extraction.test-d.ts index 792270546..d84ef4ebc 100644 --- a/packages/stack-drizzle/__tests__/schema-extraction.test-d.ts +++ b/packages/stack-drizzle/__tests__/schema-extraction.test-d.ts @@ -16,9 +16,11 @@ import { type AnyV3Table, type EncryptedIntegerOrdColumn, + type EncryptedJsonColumn, type EncryptedTextEqColumn, encryptedTable, type InferPlaintext, + type JsonDocument, types as v3Types, } from '@cipherstash/stack/eql/v3' import type { Encrypted } from '@cipherstash/stack/types' @@ -70,6 +72,33 @@ const sqlTypeFallbackUsers = pgTable('sql_type_fallback_users', { }) const fallbackExtracted = extractEncryptionSchema(sqlTypeFallbackUsers) +// Real schemas rarely declare a bare column — `.notNull()` in particular is +// near-universal. Modifiers are the one place the brand could be dropped for +// SOME columns while bare ones keep working, which is why they get their own +// fixture: every other table here is bare, so a modifier-only regression would +// otherwise ship green. See the `modifiers` describe below. +const modified = pgTable('modified', { + bare: types.TextEq('bare'), + notNull: types.TextEq('not_null').notNull(), + unique: types.IntegerOrd('uniq').unique(), + chained: types.IntegerOrd('chained').notNull().unique(), +}) +const modifiedExtracted = extractEncryptionSchema(modified) + +// `types.Json`'s plaintext is the whole `JsonDocument` union rather than a +// scalar, so it exercises the one domain where a widened index signature and +// the precise type are easiest to confuse. +const documents = pgTable('documents', { + id: serial('id').primaryKey(), + doc: types.Json('doc'), + label: types.TextEq('label'), +}) +const documentsExtracted = extractEncryptionSchema(documents) +const documentsAuthored = encryptedTable('documents', { + doc: v3Types.Json('doc'), + label: v3Types.TextEq('label'), +}) + /** The plaintext map both schemas must infer. */ type UserPlaintext = { email: string; age: number } @@ -119,6 +148,89 @@ describe('extractEncryptionSchema - plaintext inference (#589)', () => { }) }) +/** + * Regression guard for the brand's survival through Drizzle's column modifiers. + * + * `pgTable()` does not hand builders through — it rebuilds each one via + * drizzle's `BuildColumn`, which keeps only + * `Omit | 'brand' | 'dialect'>`. The + * brand rides in `_` precisely so it survives that (see `EqlV3Column`), and + * modifiers then intersect further into `_` (`NotNull = T & { _: { notNull: + * true } }`). Both steps have to preserve the symbol key. + * + * This is a drizzle-orm compatibility contract, not one we control: an upstream + * change to how `BuildColumn` or `NotNull` reshape `_` could drop the brand for + * modified columns while bare ones keep working. The runtime would not notice — + * `getEqlV3Column` recovers builders from the runtime carrier and the SQL domain, + * never from the type — so extraction would silently type a `.notNull()` column + * as a plain pass-through field while still encrypting it. Nothing else in CI + * catches that. + */ +describe('extractEncryptionSchema - drizzle column modifiers', () => { + it('keeps modified columns in the extracted schema', () => { + expectTypeOf< + keyof InferPlaintext + >().toEqualTypeOf<'bare' | 'notNull' | 'unique' | 'chained'>() + }) + + it('keeps each modified column at its own concrete plaintext type', () => { + expectTypeOf>().toEqualTypeOf<{ + bare: string + notNull: string + unique: number + chained: number + }>() + }) + + it('keeps modified columns addressable at their concrete domain type', () => { + expectTypeOf( + modifiedExtracted.notNull, + ).toEqualTypeOf() + expectTypeOf( + modifiedExtracted.chained, + ).toEqualTypeOf() + }) +}) + +/** + * `types.Json` is the only domain whose plaintext is a union rather than a + * scalar, so it is the easiest one to lose to a widened index signature without + * any other assertion noticing. + */ +describe('extractEncryptionSchema - json domain', () => { + it('infers JsonDocument on both paths', () => { + type DocumentPlaintext = { doc: JsonDocument; label: string } + + expectTypeOf< + InferPlaintext + >().toEqualTypeOf() + expectTypeOf< + InferPlaintext + >().toEqualTypeOf() + }) + + it('keeps the json column addressable at its concrete domain type', () => { + expectTypeOf(documentsExtracted.doc).toEqualTypeOf() + expectTypeOf( + documentsExtracted.label, + ).toEqualTypeOf() + }) + + it('rejects a scalar written to a json column', async () => { + const client = await Encryption({ schemas: [documentsExtracted] }) + + expectTypeOf(client.encrypt).toBeCallableWith( + { note: 'ok' }, + { table: documentsExtracted, column: documentsExtracted.doc }, + ) + // @ts-expect-error - JsonDocument is object | array | null, never a scalar + client.encrypt(36, { + table: documentsExtracted, + column: documentsExtracted.doc, + }) + }) +}) + describe('extractEncryptionSchema - unbranded table fallback', () => { it('preserves the widened schema type for a table annotated as PgTable', () => { expectTypeOf(widenedExtracted).toEqualTypeOf() From 9a4af723cc84a764351825328f2e823f125471cd Mon Sep 17 00:00:00 2001 From: Toby Hede Date: Thu, 30 Jul 2026 13:56:04 +1000 Subject: [PATCH 4/4] fix(stack-drizzle): widen mixed extracted schemas --- .changeset/typed-extract-encryption-schema.md | 4 +-- .../__tests__/schema-extraction.test-d.ts | 23 ++++++++++++++ packages/stack-drizzle/src/column.ts | 1 + .../stack-drizzle/src/schema-extraction.ts | 30 +++++++++++++++---- skills/stash-drizzle/SKILL.md | 11 +++---- 5 files changed, 56 insertions(+), 13 deletions(-) diff --git a/.changeset/typed-extract-encryption-schema.md b/.changeset/typed-extract-encryption-schema.md index 8a1dfbc32..5e1f31ce4 100644 --- a/.changeset/typed-extract-encryption-schema.md +++ b/.changeset/typed-extract-encryption-schema.md @@ -5,7 +5,7 @@ Type `extractEncryptionSchema` precisely: a Drizzle-extracted schema now preserves each column's concrete EQL v3 domain instead of widening to `AnyV3Table` (#589). -`extractEncryptionSchema` is generic over the Drizzle table (`(table: T)`) and returns `EncryptedTable & Cols`, the same shape a hand-written `encryptedTable({...})` returns, when concrete column brands are available. Each column's builder is carried through `pgTable()` on a phantom brand and recovered by a mapped type, which also filters out the table's non-encrypted columns. Tables widened to `PgTable`, and ordinary `customType` columns recovered from their EQL SQL domain, retain the safe `AnyV3Table` fallback instead of incorrectly becoming an empty schema type. +`extractEncryptionSchema` is generic over the Drizzle table (`(table: T)`) and returns `EncryptedTable & Cols`, the same shape a hand-written `encryptedTable({...})` returns, when concrete column brands are available. Each column's builder is carried through `pgTable()` on a phantom brand and recovered by a mapped type, which also filters out the table's non-encrypted columns. Tables widened to `PgTable`, and tables containing ordinary `customType` columns recovered from their EQL SQL domain, retain the safe `AnyV3Table` fallback instead of incorrectly becoming an empty or partial schema type. What this fixes, along the documented flow `extractEncryptionSchema(table)` → `Encryption({ schemas })` → `bulkEncryptModels`: @@ -15,4 +15,4 @@ What this fixes, along the documented flow `extractEncryptionSchema(table)` → **Runtime behaviour is unchanged** — the runtime already recovered each column's builder correctly, so this is a type-level fix only. It is `minor` rather than `patch` because code that previously compiled against the widened types can now fail to compile: a model field typed against the wrong domain, or a schema-derived type that relied on the old index signature. Rows whose shape is only known at runtime (a dynamically built table) should name their model type explicitly — `client.bulkEncryptModels(rows, schema)` — rather than being cast back to `AnyV3Table`. -`skills/stash-drizzle` documents the preserved typing and warns against casting an extracted schema to `AnyV3Table` to make an insert compile. +`skills/stash-drizzle` documents the preserved typing and warns against casting an extracted schema to `AnyV3Table` to make an insert compile. A matching update to the separately maintained CipherStash documentation site is required so its Drizzle schema-extraction guidance explains the precise branded typing and the widened fallback for incomplete runtime-recovered column maps. diff --git a/packages/stack-drizzle/__tests__/schema-extraction.test-d.ts b/packages/stack-drizzle/__tests__/schema-extraction.test-d.ts index d84ef4ebc..83ce4608c 100644 --- a/packages/stack-drizzle/__tests__/schema-extraction.test-d.ts +++ b/packages/stack-drizzle/__tests__/schema-extraction.test-d.ts @@ -72,6 +72,14 @@ const sqlTypeFallbackUsers = pgTable('sql_type_fallback_users', { }) const fallbackExtracted = extractEncryptionSchema(sqlTypeFallbackUsers) +// A single visible brand must not make the schema look complete when another +// runtime-encrypted column is only recoverable from its SQL domain. +const mixedUsers = pgTable('mixed_users', { + email: eqlTextEq('email'), + age: types.IntegerOrd('age'), +}) +const mixedExtracted = extractEncryptionSchema(mixedUsers) + // Real schemas rarely declare a bare column — `.notNull()` in particular is // near-universal. Modifiers are the one place the brand could be dropped for // SOME columns while bare ones keep working, which is why they get their own @@ -256,6 +264,21 @@ describe('extractEncryptionSchema - unbranded table fallback', () => { expectTypeOf(result.data).toEqualTypeOf>() }) + + it('widens a mixed branded and SQL-derived schema as a complete unit', async () => { + expectTypeOf(mixedExtracted).toEqualTypeOf() + + const client = await Encryption({ schemas: [mixedExtracted] }) + const result = await client.bulkEncryptModels( + [{ email: 'ada@example.com', age: 36 }], + mixedExtracted, + ) + if (result.failure) throw result.failure + + expectTypeOf(result.data).toEqualTypeOf< + Array<{ email: Encrypted; age: Encrypted }> + >() + }) }) /** diff --git a/packages/stack-drizzle/src/column.ts b/packages/stack-drizzle/src/column.ts index af408f0c3..a399d7e10 100644 --- a/packages/stack-drizzle/src/column.ts +++ b/packages/stack-drizzle/src/column.ts @@ -254,6 +254,7 @@ export function makeEqlV3Column( const column = makeCarrierColumn(sqlType, name) writeBuilder(getCarrier(column), builder) + // biome-ignore lint/plugin: makeCarrierColumn always returns an object; attaching the registry-symbol carrier directly is intentional and getEqlV3Column reads this exact property. writeBuilder(column as unknown as EqlV3ColumnCarrier, builder) // The brand is type-only — no runtime property backs it (see EqlV3Column) — // so the carrier column is narrowed to the branded type here. The runtime diff --git a/packages/stack-drizzle/src/schema-extraction.ts b/packages/stack-drizzle/src/schema-extraction.ts index ab92ecd87..cd59cea0a 100644 --- a/packages/stack-drizzle/src/schema-extraction.ts +++ b/packages/stack-drizzle/src/schema-extraction.ts @@ -4,6 +4,7 @@ import { type EncryptedTable, encryptedTable, } from '@cipherstash/stack/eql/v3' +import type { Encrypted } from '@cipherstash/stack/types' import type { PgTable } from 'drizzle-orm/pg-core' import { getEqlV3Column, type V3BuilderOf } from './column.js' @@ -40,15 +41,32 @@ export type EncryptedColumnsOf = { : K]: V3BuilderOf } +/** + * Columns whose Drizzle data is an encrypted envelope but whose concrete EQL + * builder is not visible in the type layer. This includes ordinary + * `customType<{ data: Encrypted }>` columns that runtime extraction recognizes + * from their SQL domain. If even one exists, the branded map is incomplete. + */ +type UnbrandedEncryptedKeys = { + [K in keyof T]-?: [V3BuilderOf] extends [never] + ? T[K] extends { _: { data: Encrypted } } + ? K + : never + : never +}[keyof T] + /** * Keep concrete branded columns precise, but retain the pre-existing widened - * schema type when TypeScript cannot see any brands. The runtime extractor can - * still recover columns from a widened `PgTable` or from their EQL SQL domain; - * representing those successful paths as an empty table would be unsound. + * schema type when the branded map is empty or incomplete. The runtime + * extractor can still recover columns from a widened `PgTable` or from their + * EQL SQL domain; representing those successful paths as an empty or partial + * table would be unsound. */ -type ExtractedEncryptionSchema = keyof EncryptedColumnsOf extends never - ? AnyV3Table - : EncryptedTable> & EncryptedColumnsOf +type ExtractedEncryptionSchema = [UnbrandedEncryptedKeys] extends [never] + ? keyof EncryptedColumnsOf extends never + ? AnyV3Table + : EncryptedTable> & EncryptedColumnsOf + : AnyV3Table /** * Rebuild a Drizzle table's encrypted columns as an eql/v3 {@link EncryptedTable}. diff --git a/skills/stash-drizzle/SKILL.md b/skills/stash-drizzle/SKILL.md index 225bc46e8..7ea416896 100644 --- a/skills/stash-drizzle/SKILL.md +++ b/skills/stash-drizzle/SKILL.md @@ -134,11 +134,12 @@ const usersSchema = extractEncryptionSchema(usersTable) The extracted schema keeps each column's concrete domain, so it types exactly like a hand-written `encryptedTable({...})`: `InferPlaintext` is a precise per-column plaintext map, non-encrypted columns (`id`, plain `text()` helpers) are excluded from it and pass through model operations untouched, and `usersSchema.email` addresses the column at its own type. Do not cast an extracted schema to `AnyV3Table` or annotate model rows with `Record` to make an insert compile — that discards the checking and hides real mismatches. That precision depends on passing the concrete table type produced by `pgTable()`. -If a table has already been widened to `PgTable`, or its encrypted columns are -ordinary `customType` columns detected from their EQL SQL domain, extraction -returns the safe widened `AnyV3Table` type because the concrete domain brands -are no longer available to TypeScript. Runtime extraction still discovers and -uses those encrypted columns. +If a table has already been widened to `PgTable`, or any encrypted column is an +ordinary `customType` column detected from its EQL SQL domain, extraction +returns the safe widened `AnyV3Table` type because the complete concrete domain +map is no longer available to TypeScript. This applies to mixed tables with +both `types.*` and ordinary `customType` encrypted columns. Runtime extraction +still discovers and uses all of those encrypted columns. ### 2. Initialize the Encryption Client