feat(appkit): embed typegen cache in generated .d.ts files - #501
Open
calvarjorge wants to merge 1 commit into
Open
feat(appkit): embed typegen cache in generated .d.ts files#501calvarjorge wants to merge 1 commit into
calvarjorge wants to merge 1 commit into
Conversation
The typegen cache lived in node_modules, which npm install wipes on every deploy — so a deployed app started cold and re-ran every workspace call (DESCRIBE / serving getOpenApi) at build time. Under the assumed-role authz model the Apps service principal running the build may lack access those calls need, which fails the deployment. Move the cache into the committed generated .d.ts files: a header comment holds one change-detector hash per entry, the body holds the rendered type block. On a later run, an entry whose source hash matches the header is a cache hit and its block is reused verbatim with no workspace call. A deploy with unchanged types therefore makes zero workspace calls; a genuinely changed entry is still described (and fails loudly if unreachable). - New embedded-cache module: render/parse header, split body into per-entry blocks (string/comment-aware), preserve timestamp on unchanged runs. - Serving re-keyed on a local identity hash (alias|endpointName) computed before fetching, so a hit skips getOpenApi; serving.d.ts is now committed and the runtime request-param allowlist reads it directly. - Remove the node_modules JSON caches. --no-cache stays as the force-refresh escape hatch. Co-authored-by: Isaac Signed-off-by: Jorge Calvar <jorge.calvar@databricks.com>
Contributor
📦 Bundle size reportCompared against
|
| dist | raw | gzip |
|---|---|---|
| JS (runtime) | 836 KB (+46 KB) | 292 KB (+16 KB) |
| Type declarations | 302 KB (+17 KB) | 103 KB (+5.9 KB) |
| Source maps | 1.6 MB (+99 KB) | 547 KB (+32 KB) |
| Other | 11 KB | 3.7 KB |
| Total | 2.8 MB (+162 KB) | 945 KB (+54 KB) |
Per-entry composition (own code — deps external (as shipped))
| Entry | Initial (gz) | Lazy (gz) | Total (gz) | node_modules (min) | Own code (min) |
|---|---|---|---|---|---|
. |
88 KB (+1.6 KB) | 2.5 KB | 90 KB (+1.6 KB) | external | 286 KB (+4.4 KB) |
./beta |
44 KB (+4.6 KB) | 429 B (+198 B) | 45 KB (+4.8 KB) | external | 129 KB (+9.5 KB) |
./type-generator |
20 KB (+1.1 KB) | 0 B | 20 KB (+1.1 KB) | external | 57 KB (+3.2 KB) |
Chunks:
| Entry | Chunk | Load | Size (gz) |
|---|---|---|---|
. |
index.js |
initial | 84 KB |
. |
utils.js |
initial | 4.0 KB |
. |
remote-tunnel-manager.js |
lazy | 2.5 KB |
./beta |
beta.js |
initial | 29 KB |
./beta |
stream-manager.js |
initial | 5.8 KB |
./beta |
wide-event-emitter.js |
initial | 3.2 KB |
./beta |
databricks.js |
initial | 3.0 KB |
./beta |
configuration.js |
initial | 2.1 KB |
./beta |
service-context.js |
initial | 1.3 KB |
./beta |
client-options.js |
initial | 220 B |
./beta |
supervisor-api.js |
lazy | 184 B |
./beta |
databricks.js |
lazy | 132 B |
./beta |
index.js |
lazy | 113 B |
./type-generator |
index.js |
initial | 20 KB |
@databricks/appkit-ui
npm tarball (packed): 305 KB (+101 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).
| dist | raw | gzip |
|---|---|---|
| JS (runtime) | 360 KB (+390 B) | 119 KB (+60 B) |
| Type declarations | 205 KB | 74 KB |
| Source maps | 686 KB (+390 B) | 224 KB (+66 B) |
| CSS | 16 KB | 3.3 KB |
| Total | 1.2 MB (+780 B) | 422 KB (+126 B) |
Per-entry composition (consumer bundle — deps bundled, peerDeps external)
| Entry | Initial (gz) | Lazy (gz) | Total (gz) | node_modules (min) | Own code (min) |
|---|---|---|---|---|---|
./js |
4.3 KB | 49 KB | 54 KB | 208 KB | 12 KB |
./js/beta |
20 B | 0 B | 20 B | 0 B | 0 B |
./react |
429 KB (+55 B) | 49 KB | 478 KB (+55 B) | 1.3 MB | 168 KB (+390 B) |
./react/beta |
20 B | 0 B | 20 B | 0 B | 0 B |
Chunks:
| Entry | Chunk | Load | Size (gz) |
|---|---|---|---|
./js |
index.js |
initial | 4.2 KB |
./js |
chunk |
initial | 120 B |
./js |
apache-arrow |
lazy | 49 KB |
./js/beta |
beta.js |
initial | 20 B |
./react |
index.js |
initial | 427 KB |
./react |
tslib |
initial | 2.1 KB |
./react |
apache-arrow |
lazy | 49 KB |
./react/beta |
beta.js |
initial | 20 B |
⚠️ Over budget: a package's shipped tarball, or a browser entry's consumer bundle (deps included), grew by more than 5% (and >10 KB). This check will fail — reduce the size, or acknowledge the increase by updatingbundle-size-baseline.json.
Contributor
🤖 AppKit PR bot🔬 Run evalsStart an eval for this PR from the evals-monitor app: Go to Evals Monitor → 📦 Try this PR's app templateScaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh run download 30559970662 -R databricks/appkit -n appkit-template-0.48.0-pr.d9a10c8-typegen-embedded-cache-501 -D appkit-pr-501 \
&& unzip -o "appkit-pr-501/appkit-template-0.48.0-pr.d9a10c8-typegen-embedded-cache-501.zip" -d "appkit-pr-501" \
&& databricks apps init --template "appkit-pr-501"The template pins |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Move the typegen cache into the committed generated files
Why
Typegen derives TypeScript types by calling the Databricks workspace at build time (
DESCRIBE QUERY,DESCRIBE TABLE EXTENDED, servinggetOpenApi). The cache that lets it skip those calls lived innode_modules/.databricks/appkit/*.json— whichnpm installwipes on every deploy. So a deployed app always started cold and re-ran every workspace call during the build.That's a latent source of nondeterministic deploys, and it becomes a hard blocker under the new Apps authz model: a user may reach the UC table/warehouse through an assumed role, but the Apps service principal running the build may not — so the build-time workspace calls fail and take the deployment down with them.
What
The cache now travels inside the committed generated
.d.tsfiles instead ofnode_modules:analytics.d.ts,metric-views.d.ts,serving.d.ts) carries a header comment listing one change-detector hash per entry; the body holds the rendered type block.Details:
embedded-cache.tsmodule: renders/parses the header and splits the body into per-entry blocks (string/comment-aware scanner so braces inside types don't corrupt boundaries). Preserves the generation timestamp when nothing changed, so unchanged regenerations are byte-identical (no git churn).sha256(alias|endpointName)) computed before fetching, so a hit skips thegetOpenApicall entirely — the previous spec-based hash couldn't, since it required fetching first.serving.d.tsis now committed (un-gitignored); the runtime request-parameter allowlist reads it directly.node_modulesJSON caches are removed.--no-cachestays as the force-refresh escape hatch for drift a local hash can't detect (e.g.SELECT *over a changed table, an altered metric view, or an upstream serving-model change).Migration
Old headerless committed files read as a version mismatch → full cache miss → re-described on the first run under a live warehouse, which rewrites them with the new header. No manual step required.
Testing
embedded-cache.test.tscovers the encode→decode round-trip, malformed/absent headers (→ safe all-miss), timestamp preservation, and__proto__-key safety.This pull request and its description were written by Isaac.