Show host GPU driver on fleet instances - #4032
Open
peterschmidt85 wants to merge 1 commit into
Open
Conversation
|
This PR is stale because it has been open for 14 days with no activity. |
peterschmidt85
force-pushed
the
instance-gpu-driver
branch
from
July 30, 2026 09:54
894eaaa to
d6bf46a
Compare
Currently, the only way to know which GPU driver a fleet instance runs is to provision it and run a task there. Now shim detects the host GPU driver version on start (`nvidia-smi` for NVIDIA, `amd-smi` for AMD, `/sys/module/tenstorrent/version` for Tenstorrent) and reports it via the new `GET /api/instance/info` endpoint. The server stores it in `JobProvisioningData`, so that no migration is needed, and exposes it as `Instance.gpu_driver` and as a DRIVER column in `dstack fleet -v`. Notes: - Instance info reports facts observed by shim, as opposed to `/api/components`, which reports software managed by shim. - GPU hosts are asked on every instance check, as the facts change when shim restarts, e.g., after a driver upgrade, which the server does not necessarily observe. The provisioning data is only updated if the driver changed. This also fills the driver for instances provisioned before the server upgrade. - The driver stays unknown on hosts without GPUs, if detection fails, and on backends where shim does not run (runpod, vastai, kubernetes). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
peterschmidt85
force-pushed
the
instance-gpu-driver
branch
from
July 30, 2026 11:51
d6bf46a to
e81c7b9
Compare
peterschmidt85
marked this pull request as ready for review
July 30, 2026 12:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When
dstackprovisions instances, the accelerator driver installed on the host (NVIDIA/AMD/Tenstorrent) is not visible anywhere — anyone listing fleet instances cannot tell which driver a host runs, short of provisioning it and running a task there.Solution
Detect the driver on the host and show it on fleet instances. When it cannot be detected, the field stays
None.dstack-shimdetects the driver version once at startup, inside the existing GPU detection: an extradriver_versioncolumn in thenvidia-smiquery,--driveradded to the existingamd-smi staticcall (with a fallback retry ifamd-smirejects the option), and/sys/module/tenstorrent/versionfor Tenstorrent.GET /api/instance/infoendpoint reports facts observed by shim, as opposed to/api/components, which reports software managed by shim. The handler reads only the GPU list cached at startup, so it is constant-time.JobProvisioningData.gpu_driver(GpuDriverInfo {vendor, version}, no DB migration). GPU hosts are asked on every instance check, as the reported facts change when shim restarts — required after a driver upgrade, and not necessarily observed by the server. The provisioning data is only updated if the driver changed, which also fills it for instances provisioned before the upgrade.Instance.gpu_driverin the API and aDRIVERcolumn indstack fleet -v./api/instance/health, so old/new server, CLI, and shim combinations degrade toNone.Nonefor now;kubernetesandvastaicomputes carry TODOs with implementation notes. Intel is intentionally not detected.🤖 Generated with Claude Code