Shared storage: backend-agnostic state manager + pub/sub

.ai/ARCHITECTURE.md

@@ -726,6 +726,7 @@ Special handling for Colab:
 - `prevent_initial_callbacks` - Skip callbacks on load (default: `False`)
 - `background_callback_manager` - DiskcacheManager or CeleryManager
 - `on_error` - Global callback error handler
+- `shared_storage` - Cross-process state + pub/sub backend (default: `LocalSharedStorage`). Pass `None` to disable, or a `BaseSharedStorage` subclass/instance to swap. See [Shared Storage](#shared-storage).
 
 **WebSocket Callbacks:**
 - `websocket_callbacks` - Enable WebSocket for all callbacks (default: `False`). Requires FastAPI backend.
@@ -808,6 +809,183 @@ Supported components: Input, Dropdown, Checklist, RadioItems, Slider, RangeSlide
 | Share state across tabs | `dcc.Store` with `storage_type='local'` |
 | Session-only state | `persistence_type='session'` |
 
+## Shared Storage
+
+Backend-agnostic **server-side** state shared across worker processes: a
+cross-process key/value store plus an ordered, replayable publish/subscribe
+channel. Unlike `dcc.Store` (which lives in the browser), shared storage lives
+on the server and lets callbacks running in different workers see the same keys
+and topics without standing up an external service like Redis.
+
+Enabled by default on every app. It starts lazily on first use, so it costs
+nothing until touched.
+
+### Accessing It
+
+```python
+import dash
+from dash import Input, Output, callback
+
+@callback(Output("out", "children"), Input("btn", "n_clicks"))
+def handler(n):
+    store = dash.ctx.shared_storage      # inside a callback
+    store.set("clicks", n)
+    return store.get("clicks", 0)
+
+# Or off the app directly:
+app = dash.Dash()
+app.shared_storage.set("key", {"any": "json-compatible value"})
+```
+
+Values (and published messages) must be **JSON-compatible** — dict / list / str
+/ int / float / bool / None — the same constraint as `dcc.Store` and callback
+outputs. Messages are encoded with `msgspec` (msgpack), with a stdlib `json`
+fallback.
+
+### API
+
+`app.shared_storage` (and `dash.ctx.shared_storage`) is a `BaseSharedStorage`:
+
+| Method | Purpose |
+|--------|---------|
+| `get(key, default=None)` | Read a value |
+| `set(key, value)` | Write a value |
+| `delete(key)` | Remove a key |
+| `publish(topic, message)` | Append a message to a topic |
+| `subscribe(topic, replay_from=None)` | Return a `Subscription` |
+
+**Ordered pub/sub.** Each `publish` to a topic gets a monotonically increasing
+sequence number (per topic, starting at 1; `0` means "before the first
+message"). A subscriber receives every message published after it subscribed,
+in order:
+
+```python
+# Producer (one worker):
+store.publish("progress", {"pct": 50})
+
+# Consumer (another worker/callback):
+with store.subscribe("progress") as sub:
+    for message in sub:          # or: async for message in sub
+        print(message["pct"])
+```
+
+**Replay on reconnect.** Pass `replay_from=<last-seen-seq>` to resume after a
+drop; buffered messages since that cursor replay first, so a reconnecting
+consumer doesn't miss messages. If the consumer fell farther behind than the
+bounded buffer holds, the subscription raises `SharedStorageGap` (an explicit
+gap rather than a silent hole). A `Subscription` is iterable synchronously
+(`for`) or asynchronously (`async for`), is a context manager, and has
+`close()`.
+
+### Backends
+
+Three backends ship, differing only in **how far state reaches**. Pick by the
+deployment topology — critically, whether the app runs behind a load balancer as
+more than one container/pod (see [Deployment Topology](#deployment-topology-and-shared-storage) below).
+
+| Backend | Backing store | Reaches | Extra |
+|---------|---------------|---------|-------|
+| `LocalSharedStorage` *(default)* | in-memory, owner-elected socket | one **container** (all its workers) | none |
+| `DiskcacheSharedStorage` | a `diskcache.Cache` | one **host** (all processes sharing the dir) | `dash[diskcache]` |
+| `RedisSharedStorage` | Redis (Streams for pub/sub) | **any** process/container/pod | `dash[redis]` |
+
+All three implement the same `BaseSharedStorage` contract (KV + ordered,
+replayable pub/sub with `SharedStorageGap` on buffer overrun), so app code is
+identical across them — only the constructor differs.
+
+```python
+from dash import Dash
+from dash._shared_storage import (
+    LocalSharedStorage, DiskcacheSharedStorage, RedisSharedStorage,
+)
+
+Dash(shared_storage=LocalSharedStorage)                       # default
+Dash(shared_storage=DiskcacheSharedStorage(directory="/tmp/ss"))
+Dash(shared_storage=RedisSharedStorage(url="redis://host:6379"))
+Dash(shared_storage=None)                                     # disabled
+```
+
+When disabled, accessing `app.shared_storage` / `dash.ctx.shared_storage`
+raises `SharedStorageError`.
+
+**`LocalSharedStorage`** — in-memory, no external service. On first use every
+worker races to become the single **owner** for the machine by binding a stable
+address (`AF_UNIX` on POSIX, `127.0.0.1` loopback on Windows); the winner hosts
+the engine, losers proxy over the socket. The bind *is* the lease — when the
+owner dies the address frees and a survivor re-elects **cold** (state is not
+durable). A single-process deployment is its own owner and pays no socket
+overhead. Knobs: `namespace` (defaults to a hash of cwd + argv[0]),
+`buffer_size` (default 2048).
+
+**`DiskcacheSharedStorage`** — KV + an append-log pub/sub on a `diskcache.Cache`,
+the same store `DiskcacheManager` uses; pass an existing `cache` to share one.
+Every process on one host that opens the same directory shares state, so it
+gives multi-worker parity with the local backend without a socket. **Not for
+pods** — each pod has its own ephemeral disk.
+
+**`RedisSharedStorage`** — KV on Redis strings, pub/sub on a Redis Stream per
+topic (an atomic `INCR`+`XADD` script keeps sequences ordered; `MAXLEN` bounds
+the replay window; a subscriber past the trimmed floor gets `SharedStorageGap`).
+Redis is the single source of truth, so no owner election. Pass a `url`
+(defaults to `$REDIS_URL`) or an existing `client` to reuse a connection pool.
+This is the only backend correct for **horizontally-scaled, multi-pod**
+deployments.
+
+### Deployment Topology and Shared Storage
+
+The default `LocalSharedStorage` is **per-container**: its socket/loopback
+election only reaches processes in the same network + filesystem namespace.
+
+- **Single process / single pod** (e.g. Plotly Cloud apps): fine — one owner,
+  nothing to fragment.
+- **Multiple gunicorn workers in one container**: fine — they share the pod's
+  loopback/socket and elect one owner.
+- **Multiple pods behind a load balancer** (e.g. Dash Enterprise apps scaled by
+  an HPA, routed round-robin with no session affinity): **each pod elects its own
+  isolated owner**, so `set()`/`publish()` on one pod are invisible on another.
+  This works at 1 pod and fragments silently once it scales — use
+  `RedisSharedStorage` (one Redis shared by all pods) instead.
+
+### Custom and Out-of-Tree Backends
+
+`BaseSharedStorage` is the stable, public extension point. A backend — shipped
+in-tree or as a **separate package** — implements:
+
+- `get(key, default)` / `set(key, value)` / `delete(key)` — JSON-compatible values
+- `publish(topic, message)` and `subscribe(topic, replay_from=None) -> Subscription`
+- optional `start()` / `close()` (idempotent, called once per worker)
+
+and returns a `Subscription` (`__iter__` / `__aiter__` / `close`) that raises
+`SharedStorageGap` when the replay buffer overran. That trio —
+`BaseSharedStorage`, `Subscription`, `SharedStorageGap` — is exported from
+top-level `dash`; the poll-loop helpers (`PollResult`, the polling subscription)
+are private and not part of the contract.
+
+Core only ships backends whose dependency is already a Dash extra (`msgspec`
+base; `dash[diskcache]`; `dash[redis]`). Anything needing a heavier dependency
+belongs **out of tree** behind this same interface — e.g. a Postgres backend
+(`LISTEN`/`NOTIFY` for push pub/sub + a table for KV and replay) lives in its own
+package so a `psycopg` connection is never pulled into Dash core. It plugs in the
+same way as a built-in: `Dash(shared_storage=PostgresSharedStorage(...))`.
+
+### Module Layout
+
+| File | Responsibility |
+|------|----------------|
+| `_shared_storage/base.py` | Abstract interface: `BaseSharedStorage`, `Subscription`, `SharedStorageError`, `SharedStorageGap` |
+| `_shared_storage/_engine.py` | `StoreEngine`: authoritative in-memory kv map + per-topic ordered log with bounded replay buffer (thread-safe, transport-agnostic) |
+| `_shared_storage/local.py` | `LocalSharedStorage`: owner election; owner hosts the engine, clients proxy |
+| `_shared_storage/diskcache.py` | `DiskcacheSharedStorage`: KV + append-log pub/sub on a `diskcache.Cache` |
+| `_shared_storage/redis.py` | `RedisSharedStorage`: KV + Redis Streams pub/sub (atomic `INCR`+`XADD`) |
+| `_shared_storage/_polling.py` | `PollingSubscription`: shared poll-loop subscription for the diskcache/Redis backends |
+| `_shared_storage/_transport.py` | Length-prefixed, token-gated socket transport (local backend) |
+| `_shared_storage/_codec.py` | msgspec/json codec (data-only) |
+| `dash.py` | `shared_storage` constructor arg + lazy `app.shared_storage` property |
+| `_callback_context.py` | `dash.ctx.shared_storage` accessor |
+
+Requires `msgspec` (in `requirements/install.txt`); `DiskcacheSharedStorage`
+needs the `diskcache` extra and `RedisSharedStorage` the `redis` extra.
+
 ## Async Callbacks
 
 Dash supports `async def` callbacks for non-blocking execution.

.github/workflows/testing.yml

@@ -242,6 +242,77 @@
             tests/compliance/test_typing.py \
             tests/compliance/test_callback_typing.py
 
+  shared-storage:
+    name: Run Shared Storage Tests (Python ${{ matrix.python-version }})
+    needs: build
+    timeout-minutes: 20
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.9", "3.12"]
+
+    # Redis service for the RedisSharedStorage backend.
+    services:
+      redis:
+        image: redis:6
+        ports:
+          - 6379:6379
+        options: >-
+          --health-cmd "redis-cli ping"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+
+    env:
+      REDIS_URL: redis://localhost:6379
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: '24'
+          cache: 'npm'
+
+      - name: Install Node.js dependencies
+        run: npm ci
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: 'pip'
+          cache-dependency-path: requirements/*.txt
+
+      - name: Download built Dash packages
+        uses: actions/download-artifact@v4
+        with:
+          name: dash-packages
+          path: packages/
+
+      - name: Install Dash packages
+        run: |
+          python -m pip install --upgrade pip wheel
+          python -m pip install "setuptools<80.0.0"
+          find packages -name dash-*.whl -print -exec sh -c 'pip install "{}[ci,testing,dev,diskcache,redis]"' \;
+
+      - name: Setup Chrome and ChromeDriver
+        uses: browser-actions/setup-chrome@v1
+        with:
+          chrome-version: stable
+
+      - name: Verify Redis connection
+        run: python -c "import redis; redis.Redis(host='localhost', port=6379).ping(); print('Redis OK')"
+
+      - name: Build/Setup test components
+        run: npm run setup-tests.py
+
+      - name: Run shared-storage tests
+        run: pytest --headless tests/shared_storage -v
+
   background-callbacks:
     name: Run Background & Async Callback Tests (Python ${{ matrix.python-version }})
     needs: [build, changes_filter]

CHANGELOG.md

@@ -5,6 +5,7 @@ This project adheres to [Semantic Versioning](https://semver.org/).
 ## [Unreleased]
 
 ### Added
+- [#3930](https://github.com/plotly/dash/pull/3930) Shared storage: a backend-agnostic state manager available on every app via `dash.ctx.shared_storage` (and `app.shared_storage`). It offers a cross-process key/value store (`get`/`set`/`delete`) and ordered, replayable publish/subscribe (`publish`/`subscribe`) for sharing state between callbacks or across worker processes without an external service. Subscriptions resume from a caller's last-seen sequence out of a bounded buffer, and a buffer overrun surfaces as an explicit gap rather than a silent loss. Values must be JSON-compatible (like `dcc.Store`). Started lazily on first use, so it costs nothing until touched; pass `shared_storage=None` to `Dash(...)` to disable. Three backends ship, selected by passing a `BaseSharedStorage` instance to `shared_storage=`: the default `LocalSharedStorage` (in-memory, elects one owner process per machine over an `AF_UNIX`/loopback socket — correct within a single container, including across multiple gunicorn workers); `DiskcacheSharedStorage` (on a `diskcache.Cache`, shared by every process on one host — parity for single-machine deployments, not for multi-pod ones with ephemeral disks); and `RedisSharedStorage` (on Redis, using Redis Streams for the ordered pub/sub — the backend for horizontally-scaled deployments behind a load balancer, e.g. apps scaled across pods).
 - [#3646](https://github.com/plotly/dash/pull/3646) Experimental support for React 19. The default is still React 18.3.1; to use React 19 set the environment variable `REACT_VERSION=19.2.4` before running your app, or call `dash._dash_renderer._set_react_version("19.2.4")` inside the app. React 19 has no official UMD builds, so Dash serves the [`umd-react`](https://www.npmjs.com/package/umd-react) package, together with a compatibility shim loaded after react-dom and before any component package. The shim keeps component libraries built against React <=18 (e.g. dash-bootstrap-components, dash-mantine-components) working under React 19: it stubs the removed `ReactCurrentOwner` internals, redirects the legacy element `$$typeof` symbol so pre-bundled React 18 jsx-runtimes produce elements React 19 accepts (error #525), and exposes a global `react/jsx-runtime` (`window.ReactJSXRuntime`) that Dash's own component bundles externalize to. Component library authors adopting this convention should copy the defensive `jsxRuntimeExternal` webpack external from `components/dash-core-components/webpack.config.js` rather than a bare `'ReactJSXRuntime'` string: it falls back to a `React.createElement`-based runtime when the global is missing, so the same build also works on Dash versions older than this release.
 
 ### Removed

dash/__init__.py

@@ -43,6 +43,16 @@
 from ._patch import Patch  # noqa: F401,E402
 from ._jupyter import jupyter_dash  # noqa: F401,E402
 
+from ._shared_storage import (  # noqa: F401,E402
+    BaseSharedStorage,
+    DiskcacheSharedStorage,
+    LocalSharedStorage,
+    RedisSharedStorage,
+    SharedStorageError,
+    SharedStorageGap,
+    Subscription,
+)
+
 from ._hooks import hooks  # noqa: F401,E402
 
 ctx = callback_context
@@ -94,4 +104,11 @@ def _jupyter_nbextension_paths():
     "ctx",
     "hooks",
     "stringify_id",
+    "BaseSharedStorage",
+    "DiskcacheSharedStorage",
+    "LocalSharedStorage",
+    "RedisSharedStorage",
+    "SharedStorageError",
+    "SharedStorageGap",
+    "Subscription",
 ]

dash/_callback_context.py

@@ -210,6 +210,16 @@ def states_list(self):
     def response(self):
         return getattr(_get_context_value(), "dash_response")
 
+    @property
+    def shared_storage(self):
+        """The app's shared storage (state manager + pub/sub).
+
+        A backend-agnostic key/value + publish/subscribe store shared across
+        worker processes -- for cross-callback or session state without an
+        external service. Requires ``Dash(shared_storage=...)`` (on by default).
+        """
+        return get_app().shared_storage
+
     @staticmethod
     @has_context
     def record_timing(name, duration, description=None):

dash/_shared_storage/__init__.py

@@ -0,0 +1,28 @@
+"""Backend-agnostic shared state + pub/sub (state manager).
+
+Exposes the ``BaseSharedStorage`` interface and its backends:
+
+- ``LocalSharedStorage`` (default): in-memory, owner-elected -- one container.
+- ``DiskcacheSharedStorage``: on a ``diskcache.Cache`` -- one machine, not pods.
+- ``RedisSharedStorage``: on Redis -- across processes and containers.
+"""
+
+from .base import (
+    BaseSharedStorage,
+    SharedStorageError,
+    SharedStorageGap,
+    Subscription,
+)
+from .diskcache import DiskcacheSharedStorage
+from .local import LocalSharedStorage
+from .redis import RedisSharedStorage
+
+__all__ = [
+    "BaseSharedStorage",
+    "DiskcacheSharedStorage",
+    "LocalSharedStorage",
+    "RedisSharedStorage",
+    "SharedStorageError",
+    "SharedStorageGap",
+    "Subscription",
+]

dash/_shared_storage/_codec.py

@@ -0,0 +1,37 @@
+"""Codec for the shared-storage socket transport.
+
+Prefers ``msgspec`` (msgpack: fast, compact) and falls back to stdlib ``json``.
+Both are data-only, so bytes read off the socket cannot execute code. All
+workers in one deployment share a single install, hence a single codec, so the
+two ends always agree on the format.
+
+Payloads must be JSON-compatible (dict / list / str / int / float / bool /
+None) under either codec -- the same constraint as ``dcc.Store`` and callback
+outputs.
+"""
+
+from typing import Any
+
+try:
+    import msgspec
+
+    _encoder = msgspec.msgpack.Encoder()
+    _decoder = msgspec.msgpack.Decoder()
+
+    def encode(obj: Any) -> bytes:
+        return _encoder.encode(obj)
+
+    def decode(data: bytes) -> Any:
+        return _decoder.decode(data)
+
+    CODEC = "msgpack"
+except ImportError:  # pragma: no cover - exercised via the fallback install
+    import json
+
+    def encode(obj: Any) -> bytes:
+        return json.dumps(obj).encode("utf-8")
+
+    def decode(data: bytes) -> Any:
+        return json.loads(data.decode("utf-8"))
+
+    CODEC = "json"