DennySORA
Search the site
Language：English

Language

Tools · Rust · Ratatui · updated 2026-10-11

dgxtop: designing a GPU terminal monitor that never shows a fake zero

dgxtop watches the CPU, GPUs, memory, disks, network and GPU processes of an NVIDIA DGX Spark, or any Linux host with NVIDIA GPUs, from a terminal. Version 2 is a complete rewrite around one rule: a value it cannot read is never shown as 0. This page explains the architecture behind that rule, and what each choice costs.

repository
github.com/DennySORA/dgxtop（external site）
stack
Rust 1.92 · Ratatui · crossterm · NVML · SQLite
platforms
Linux x86_64 / aarch64 (glibc) · tested on a DGX Spark (GB10) and on an RTX 5080 + RTX 3060 workstation
license
Apache-2.0
status
v2 on main · first v2 release not tagged yet
updated
dgx-spark-overview.webp1909×961
dgxtop overview on a DGX Spark: CPU and memory, the GB10 GPU at 92 % utilization with its RAM bar, the disk and network tables, and one Python process holding 69.8 GiB of GPU memory
Overview on a DGX Spark. A Python job keeps the GB10 at 92 %; it shares memory with the CPU, so the RAM held by GPU processes is drawn in its own colour inside system memory.

Why a rewrite

Version 1 sampled hardware inside the UI loop, so a slow driver call froze the screen. It kept data, history and UI state in one shared struct, chose the CPU temperature by the names and scan order of thermal zones, and showed values it could not read as 0. The v2 specification lists ten findings like these and gives each a design answer:

Finding in v1Answer in v2
Processes identified by PID aloneA ProcessKey (host, boot, PID namespace, PID, start time), a held pidfd and a two-step confirmation
Missing values became 0Read status, sample kind, quality and freshness on separate axes; a missing value is null with a reason
Unified-memory semantics mixed upSystem RAM, GPU framebuffer and per-process allocations kept apart
Bandwidth estimated from utilizationMeasured, derived and estimated values kept apart; no throughput without a byte counter
Interval and totals assumed, not measuredThe effective interval confirmed by the scheduler; real elapsed time and counter deltas
History windows misalignedTime buckets that carry the boot, the coverage and the gaps
One PID on several GPUs counted twiceHost CPU and memory sampled once per ProcessKey, then joined to each GPU
Installer did not verify downloadsChecksum and archive checks, with the previous binary kept for rollback
Sampling blocked the UIControl, sampling and rendering separated; drivers isolated in their own processes
Builds not reproducibleA committed Cargo.lock, a pinned toolchain and release gates

The rewrite was specified before it was written: 40 requirements, 15 work packages and 40 acceptance cases. It was then built as an eight-crate Rust workspace against a fixed core contract, roughly 157,000 lines of Rust with 1,373 tests.

What it shows

Four tabs: Overview, GPU, History and Settings. These screenshots come from the two machines it was tested on.

workstation-overview.webp1906×966
Overview on a workstation with two GPUs, an RTX 5080 and an RTX 3060, listing the processes that hold GPU memory
Two GPUs on a workstation. The process table lists only processes that use a GPU, marked C or G as nvidia-smi prints them.
dgx-spark-gpu.webp1922×960
GPU view of the GB10: state, clocks, unified memory, PCIe, NVLink-C2C, thermal thresholds, power, CUDA, driver, device and topology
The GPU view of the GB10, read from NVML and the CUDA toolkit files without running a command. A value the GPU does not report gets no row.
workstation-gpu.webp1906×976
GPU view of an RTX 5080: clocks, VRAM and BAR1, PCIe Gen5, fans and thresholds, power limits, CUDA, driver, firmware and the P2P path to the second GPU
The same view on an RTX 5080: VRAM, PCIe Gen5 x16, fans, power limits and the path to the second GPU. Serial number and UUID are masked.
dgx-spark-history.webp1906×956
History view of the DGX Spark over one hour: CPU, memory, GPU utilization and memory, temperature, power with energy, disk and network
One hour of a GPU job on the DGX Spark: panels in pairs on one time axis, each column a mean coloured by level, with now, average and maximum.
workstation-history.webp1907×971
History view of the two-GPU workstation with the mean and peak of each GPU over the last 1, 6, 12 and 24 hours
Below the axis, each GPU’s last 1, 6, 12 and 24 hours: mean and peak utilization, memory, temperature and power, and the energy drawn.
dgx-spark-settings.webp1916×957
Settings view listing the saved value, effective value, origin and restart requirement of each setting, beside the display-module toggles
Settings shows each value as saved and as in effect, where it came from, and whether it applies live or after a restart.

Six components, two paths

Every value travels the data path, and every user intent travels the control path. The Manager is the only writer: it alone holds the store writer and the process-action port, so neither the Viewer nor the Controller can change data or send a signal.

Architecture: six components, two pathsSolid accents mark the data path, dashed ones the control path. The Viewer never reads the store; it draws the view model the Manager publishes.

data path

  1. Linux + NVIDIA driver/proc · sysfs · NVML
  2. Adapter hostsone process each
    • procfs
    • hwmon
    • thermal
    • powercap
    • NVML
    • network
    • filesystem
  3. Collectornormalize · de-duplicate · arbitrate · rates
  4. Managerthe only writer
  5. Storesnapshots · history · config · audit
  6. Managerprojection
  7. Viewerimmutable view model · Ratatui

control path

  1. Controllerkeys · mouse · CLI → typed command
  2. Managervalidate · authorize
  3. config · scheduler · action adapterpidfd for signals
  4. audit + resultdurable before any signal
  5. Store → Viewerthe outcome is shown
ComponentIts one jobMust not
AdapterTalk to one driver or kernel interface; return native values with their source, timing and errorsPick an authoritative value, average sources, write the store, render
CollectorNormalize units, de-duplicate proven aliases, arbitrate, derive ratesCall driver internals, run SQL, change settings, send signals
ManagerSchedule sampling, own policy and configuration, authorize commands, commit data, project view modelsLay out the screen, or wait on a blocking driver call
StoreKeep snapshots, history, configuration and the audit logChoose sensors, run commands
ViewerLay out display modules and draw immutable view modelsSample, read or write the store, compute rates, authorize anything
ControllerTurn keys, clicks and flags into typed commands with stable IDsDecide sources, send signals, draw

Ten display modules register at compile time and receive only their view model and their area of the screen. A module that panics is isolated as a module error instead of taking sampling down with it.

A hung driver stalls one host, not the screen

Runtime: one parent, resident hostsHere the NVML host has timed out. CPU, memory and the screen keep updating from the other hosts.

dgxtop parent process

  • terminal · input and render, at most 4 fps
  • Manager · 2 ms control slices
  • supervisor · pipes and reaping
  • collector workers × 2
  • hot store · single writer
  • history queries × 2
  • config · audit · archive · log
  • process action worker

adapter hosts · one process each

  • procfsfresh
  • hwmonfresh
  • thermalfresh
  • powercapfresh
  • NVMLtimeout
  • networkfresh
  • filesystemfresh

↔ anonymous pipes · length-prefixed JSON · ≤ 1 MiB per frame

a host that keeps failing

  1. degraded
  2. circuit open
  3. backoff 1 s · 2 s · 4 s … 60 s
  4. probe
  5. warmup

not reaped → quarantined · at most 8 hosts

A timeout says a result is late. It does not cancel the call: a thread stuck inside a vendor library cannot be stopped safely. So every live adapter runs in its own resident child process, the same binary started as an adapter host, and talks to the parent over anonymous pipes in length-prefixed JSON frames of at most 1 MiB.

A host that misses its deadline turns degraded. After three failures a circuit breaker opens and retries back off 1, 2, 4 … 60 seconds. A host is replaced only after it has been reaped; one that cannot be reaped is quarantined, and there are never more than eight.

The parent runs a fixed set of threads and no async runtime: nothing here does network I/O, and spawn_blocking would only hide the same uncancellable call behind a future. One 256 MiB budget covers all managed data, and every queue has a byte limit as well as an item limit, because a bounded channel counts messages, not bytes.

Explainable sources, never an average

Linux can expose one sensor through several interfaces, and a label does not prove what it measures: temp1 is not necessarily the CPU. dgxtop first decides what a reading means, as a metric key of host, boot, entity, metric and scope, and only then compares readings of the same key.

Arbitration: nine steps per metric keyEvery candidate, the reason it was kept or dropped, its age and its origin travel with the committed value.
A · hwmon · package 068 °C
B · thermal zone42 °C
  1. map meaning
  2. normalize units
  3. validate
  4. check freshness
  5. de-duplicate origins
  6. rank candidates
  7. detect conflict
  8. hysteresis · 3 wins · 5 s
  9. commit + explanation

possible results

  • selected · 68 °C · hwmon
  • source conflict · A 68 °C / B 42 °C
  • unsupported
  • read failed
  • stale
  • 55 °C · never an average

four separate axes, never one trust score

  • read status
  • sample kind
  • quality
  • freshness
Synthetic example. A: hwmon package 0 reads 68 °C. B: a thermal zone reads 42 °C.
SituationWhat dgxtop doesWhat you see
B is the SoC zoneDifferent metrics; nothing to compareCPU package 68 °C, SoC 42 °C
B is a proven alias of AOne sensor reached two ways, counted onceOne value, with a note on the mismatch
A and B are independent, equal and aligned26 °C apart, beyond tolerance: a conflict, value nullSource conflict with both readings, never 55 °C
B reports a faultB excluded, A selected68 °C from A, with B marked faulty

A switch to another source needs three consecutive wins and at least five seconds, so a value does not flap between sensors. Read status, sample kind, quality and freshness stay four separate axes instead of one trust score.

Exact history

History: counters, bridges and gapsThe bridge crosses a minute boundary and is counted once, in any window that holds all of it; the gap is drawn as a gap, not as zero.
History: counters, bridges and gaps. The bridge crosses a minute boundary and is counted once, in any window that holds all of it; the gap is drawn as a gap, not as zero.12:00–12:0112:01–12:0212:02–12:03gapbridgeΔ counter ÷ Δtdots are samples; each line is one exact interval

A rate is the difference of two u64 counters divided by the real elapsed time on CLOCK_BOOTTIME, which keeps counting through suspend. A counter reset, a new boot or a source change starts a new segment instead of producing a negative or invented rate.

A one-minute rollup cannot know when inside an interval the bytes moved. Totals therefore count only intervals that lie fully inside a window, and an interval that crosses a minute boundary is kept as an exact bridge and counted once, never split in proportion.

Gauges are time-weighted, a value counts only until its source’s stale deadline, and gaps stay gaps. History lives in history.sqlite3 for seven days, at most 1 GiB, so earlier runs are charted too. The archive writes with synchronous=NORMAL, the action audit with FULL: losing the last second of a chart is acceptable, losing the record of a signal is not.

Protected process termination

Process termination: at most one SIGTERMEvery refusal sends nothing, and the action can be cancelled until the signal is being sent.
  1. Kselect a GPU process
  2. preparesame user · no root · not PID 1, dgxtop or its hosts
  3. pidfd_openverify the /proc identity
  4. confirm10 s single-use token · y
  5. audit intentSQLite synchronous=FULL
  6. pidfd_send_signalSIGTERM

signal sent

  • exit observed
  • still running

crash after the intent → unknown, never resent

SIGKILL off unless allow_force

Terminating the wrong process is the worst thing a monitor can do, so termination is a two-step, audited action on a ProcessKey (host, boot, PID namespace, PID and start time), never on a row index or a bare PID.

Prepare checks the user, refuses root sessions, PID 1, dgxtop and its own hosts, opens a pidfd and re-checks the identity in /proc. The confirmation is a single-use token valid for 10 seconds. The intent is written with synchronous=FULL before the signal, and the signal goes through the pidfd held since preparation, so a reused PID cannot be hit.

The result says signal sent apart from exit observed. A crash between the durable intent and the outcome is recorded as unknown and never resent: a database commit and a system call share no transaction, so the honest guarantee is at most once. SIGKILL stays off unless actions.allow_force is set, and --read-only turns every action off.

Budgets and overload

Every allocation dgxtop manages draws on one 256 MiB ledger, split by purpose. These are admission limits, not measured memory: allocator overhead, thread stacks and the NVIDIA library come on top, which is why the specification sets a separate 384 MiB target for the whole process tree.

The 256 MiB data budget (defaults)
PartitionMiBHolds
Catalog and metadata24Names, tombstones, arbitration state
Raw candidates3260 s of evidence for source diagnostics
Hot state16512 series, 256 processes, 1,024 GPU links
Full-resolution history32The last 300 s
Minute aggregates9624 h of rollups, source changes and gaps
Ingress8Every live IPC and decoding buffer
Presentation12View models and the formatting cache
History queries16Snapshots pinned by running queries
Control and reserve20Commands, archive journal, log, safety reserve
Overload statesEntered by queue bytes at 75 % or slow collection (pressured) and memory at 90 % (throttled).
  1. normalfull rate
  2. pressuredoptional work stops · 2 fps
  3. throttledqueries limited · period stretched
  4. recoveringone step per 30 s

Under load dgxtop degrades visibly instead of silently. Pressure stops optional work and draws at most 2 fps; throttling limits queries and stretches the sampling period, which a banner shows as effective against configured. Recovery takes one step per 30 healthy seconds, and missing samples are never back-filled.

Boundaries and verification

Workspace: one contract crate at the centreConcrete implementations meet only in the composition root.
dgxtopcomposition root + CLI
  • dgxtop-runtimehosts · IPC · workers→ dgxtop-core only
  • dgxtop-adaptersprocfs · NVML · pidfd→ dgxtop-core only
  • dgxtop-collectorsarbitration · rates→ dgxtop-core only
  • dgxtop-storehistory · SQLite→ dgxtop-core only
  • dgxtop-manageruse cases · sessions→ dgxtop-core only
  • dgxtop-uiRatatui · keymap→ dgxtop-core only
dgxtop-coretypes · units · ports · commands · byte budget

enforced in CI · check_boundaries.py

The workspace has eight crates. Each implementation crate depends only on dgxtop-core, and only the composition root sees them all; a CI script reads cargo metadata and fails the build if a crate reaches another. The core never depends on NVML, libc, SQLite or the UI.

Format, lint, 1,373 tests (1,372 on aarch64), the dependency boundaries and the real-hardware suites pass on a Debian 13 workstation with an RTX 5080 and an RTX 3060, and on a DGX Spark. The acceptance report tracks 40 cases: 12 pass, 24 are not run yet (mostly opt-in end-to-end suites), and 4 are blocked on a tagged release, three 30-minute benchmark runs and a 72-hour soak. Until those exist, no platform is called validated, and the performance numbers stay targets.

Trade-offs

  • A process per adapter, not a thread. More resident memory and IPC, in exchange for a hung driver that cannot freeze anything else.
  • Fixed threads and crossbeam channels, not an async runtime. There is no network I/O to justify one.
  • A compile-time registry, not plugins. No unstable Rust ABI and no new attack surface for adapters and display modules.
  • Conflicts shown, not averaged. Less tidy, but never a number that no sensor reported.
  • At most one signal, not retries. An unknown outcome is shown as unknown.
  • Redraw on change, at most 4 fps. A monitor does not need 60 fps, and a slow terminal must never block input.