Dynamic Margin Systems

Know what your GPUs do to the grid.

DMS is the measured record of what AI compute does to the grid. It reads every GPU on your own servers as fast as the card's sensor actually updates, a rate it measures rather than assumes, builds the load profile planners are now asking for, and checks it against your meters and against the models planners use. On a shared floor it splits the record by tenant, so the site answering to the grid can see whose load moved. When the grid calls for less power, it cuts at the cards in steps and records exactly what happened. Jobs run slower and keep going.

Two A100sreplaying a real run
Power draw of a two-GPU node over 65 minutes Node power from 05:50 to 06:55 UTC, stepping up when jobs start and dropping when they stop, with a dashed budget line that falls to 380 watts during a shaded thirty-minute event.
Node draw, 2-second means of 10 Hz readings Budget the daemon held, lowered to 380 W in the shaded event

Two A100s running a scripted mix of training and inference. Every time a job started or stopped, the node moved by about 300 W within seconds.

  • 3 of 1,800seconds over the limit during a 30-minute grid event, never by more than 70 W
  • 80%of full speed for a training job under that cap, and it never stopped. Scripted load.
  • 476 Wthe biggest jump in any 5 seconds across a 10.7-hour run. That's 79% of what the node is rated for.
  • 26 / 25job starts from idle and drops to idle in 10.7 hours, each one on the record with its time and size.

Grid operators have started asking about your GPUs

ERCOT told its board in September that dynamic models for large loads "have proven to be inaccurate and thus are inadequate for studying their grid impacts." NERC's board adopted its first standards for computational loads in October, and they go to FERC by the end of the year. Every site of 50 MW and up connected at 100 kV or more will register, give the grid operator modeling data on how its computing equipment is used, check that data again after changes, and follow the operator's instructions. On a colo or neocloud floor, the site that registers answers for GPUs its tenants run. PJM has proposed that large computational loads recover to 90% within 2 seconds after a voltage dip, ERCOT already requires it, MISO has filed ramp limits, and Southern Company caps ramps at 20 MW a minute.

Most sites can't answer yet, because nobody has measured it. DMS measures it on your own servers and checks it against your meter, without touching your customers' jobs. The result is a record a planner, a utility or an aggregator can check line by line.

Where DMS sits on a site A one-line diagram. The grid feeds a substation and the site's revenue meter, where the grid operator measures. The site bus feeds cooling and networking on one branch and the GPU servers on the other. DMS runs on the servers, reading each GPU and lining that up with the meter. A dashed line shows meter data and grid instructions reaching DMS, and signed records going back. M Grid Substation Revenue meter and PMU. The grid operator measures here. Site bus Cooling, networking GPU servers thousands of these Meter data and grid instructions in, records out DMS reads each GPU at its sensor's real rate and lines it up with the meter

What to read

If you run more than 50 MW of GPUs, these six pieces cover the rules headed your way. Newest is first.

What you get

The profile

Hardware-tested on H100 and A100

Run DMS read-only on one node for 48 hours. You get back how fast the load climbs when a job starts, how far it falls when one ends, and the biggest swing it made over any window between half a second and a minute.

One node can't tell you what a whole site does, so the profile gives two numbers. One assumes every node moves at once, like a single huge training run. The other assumes the jobs have nothing to do with each other and partly cancel out. Your site lands somewhere in between.

The check

Simulation-tested

Run DMS next to a rack or server meter and it scores how closely the GPU and CPU readings track the meter, on data the check wasn't tuned on. Hand it a planner's simulated power curve and it shows where the model and the real load disagree: peaks, ramps, the size of every swing.

If a UPS or battery sits between your servers and the grid, every report says so, because then the GPUs aren't what the grid sees.

The tenant report

Simulation-tested

Tell DMS which nodes or GPUs belong to which tenant. It gives the site and every tenant their biggest swing in any 5 seconds and their biggest rise and fall inside any minute, scaled to the site's size and checked against the limits that apply. For each of the site's worst swings it shows which tenant moved and by how much.

When the voltage dips, it shows how long each tenant's load took to get back to 90%. When a driver, firmware or card changes, it logs the change and measures before against after.

The control

Power limit hardware-tested on A100. Ramp limit simulation-tested.

Set a power limit for the node and DMS keeps it under, and you can cap how fast it's allowed to climb. When the grid calls, it lowers the cards a step at a time and writes a record of the cut. If DMS itself crashes, every card drops to a safe cap you picked ahead of time.

First page of a sample DMS planning profile for a two-GPU A100 node

What a profile looks like

This is the first page of a real one, built from the two-A100 run below. Levels, the largest swings by window, every job start and stop, and the limits of the data, all on the page.

Open the full sample profile (PDF)

Compliance with a grid rule is judged by the operator's meter at the grid connection. DMS doesn't replace that meter. It explains what's behind it.

What DMS does

Twenty pieces, each marked with how it was tested. Hardware-tested means it has run on real GPUs. Simulation-tested means it passes its full test suite against simulated GPUs and meters and has not yet run on hardware.

Everything DMS does, with how each piece was tested
  • Measuring
  • Checking and records
  • Control
  • Readings at the sensor's real rateEvery GPU's power through the vendor's own library, NVIDIA or AMD. At install DMS measures how often each card's power reading and energy counter actually change and reads at that rate, using the driver's own sample buffer where it exists; the rest of the telemetry runs ten times a second, with gaps counted instead of smoothed over.Hardware-tested at 10 Hz on H100 and A100; the measured-rate reader is simulation-tested until the next run
  • Sensor checkMeasures how often each card's power reading really changes and how much of the time it covers, and prints that on every report.Simulation-tested
  • Recorder filesReads the site recorder's COMTRADE files, hundreds of readings a second, and lines them up with every node.Simulation-tested
  • Planning profileRamps, drops and dips, swings by window and by frequency, scaled to a site.Hardware-tested, A100
  • CPU readingsCPU package power next to the GPUs, so the record covers more of the server.Simulation-tested
  • Server and rack metersReads a PDU, a server's own meter or a site meter beside the cards, including meters that read many times a second.Simulation-tested
  • Meter checkFits the GPU and CPU readings to a meter and reports the error on data held back from the fit.Simulation-tested
  • Model checkLines a planner's simulated power curve up against what was measured and reports peaks, ramps and overall error.Simulation-tested
  • Site attributionLines a site meter up with every node and shows which GPUs moved each swing.Simulation-tested
  • Tenant reportSwing and ramp for the site and each tenant, and who moved in each of the worst swings.Simulation-tested
  • Recovery after a dipHow long the site and each tenant took to get back to 90% after a voltage dip.Simulation-tested
  • Change logRecords every driver, firmware and card change, and measures the load before against after.Simulation-tested
  • UPS and battery settingEach report states whether a UPS or battery sits between the GPUs and the grid.Simulation-tested
  • Signed reportsEach report hashed, signed with a key that never leaves the node, and chained to the one before it.Simulation-tested
  • Audit exportOne command zips every report and ledger since a date you pick.Simulation-tested
  • Power budgetKeeps the node under a line, cutting the cards you rank lowest first.Hardware-tested, A100
  • Events and schedulesBudgets with a start and an end, from a file, a weekly schedule or an OpenADR 3 server.Hardware-tested, A100 (OpenADR simulation-tested)
  • Safe cap on stopIf DMS stops for any reason, the cards fall back to a cap you chose.Hardware-tested, A100
  • Rise limit and swing budgetHow fast draw may climb, and how far it could fall if every job died at once.Simulation-tested

What we've measured

Node power when a job starts and when a job endsTwo panels of node power at ten readings a second on two A100 GPUs under scripted load. Left: power climbs from about 140 watts to about 400 watts within a second as a job starts, after about half a second above 600 watts. Right: power falls from about 390 watts to about 140 watts within a second as a job ends. A job starts 0 W 200 W 400 W 600 W -5 s 0 +5 s +10 s +15 s A job ends 0 W 200 W 400 W 600 W -5 s 0 +5 s +10 s +15 s
Ten readings a second on two A100s, the night of October 26. When a job starts, the node jumps from about 140 W to over 300 W in under a second, and spends about half a second above 600 W on the way. When a job ends, it falls about 250 W just as fast. Scripted load, one node.
Largest change in node power within each windowBar chart for a 10.7 hour run on two A100 GPUs under scripted load: the largest change within half a second was 475 watts, within one second 475 watts, within five seconds 476 watts, and within sixty seconds 571 watts, against a node rating of 600 watts.0 W200 W400 W600 WNode rating, 600 W0.5 s: 475 W475 Wwithin 0.5 s1 s: 475 W475 Wwithin 1 s5 s: 476 W476 Wwithin 5 s60 s: 571 W571 Wwithin 60 s
The biggest change in node power inside each window, across the whole 10.7-hour run. Most of a node's swing happens in under half a second. Scripted load, one node.

How DMS measures

A record is only worth something if someone else can check it. The method document says what DMS reads and how often, what every field in a report means, how each check is computed, and where the numbers stop being valid. That includes the limits: ten readings a second sees up to 5 Hz, on A100 and H100 each reading covers 25 ms of every 100, a GPU sensor isn't a revenue meter, and a battery between the servers and the grid changes what the grid sees. Faster swings come from the site's own recorder, which DMS reads.

Read the method (PDF)

Your free profile

You run DMS on one NVIDIA or AMD node for 48 hours, no root needed. It reads each GPU through the vendor's own library (NVML, the same one nvidia-smi uses, or AMD's amdsmi) and saves power readings and settings to a folder you pick. It doesn't look at what jobs are running. It also opens no ports and never touches the network, so you can unplug the node and it runs the same.

Want to read the code first? Ask and I'll send it with its SHA-256, so you can confirm the copy you run is the one you read. Send the folder back and you'll have your profile in three days.

Try it on one node.

Liam Bronson, founder