Skip to content
NexusIgniteDiscuss your custom solution
All insights

AI Infrastructure

Choose the workload before the GPU count

For universities, the right AI system starts with the research and teaching it must support. That connects quality, response time and concurrent use to memory, networking, power and an operating plan.

An MI325X-inspired accelerator package with adjacent memory, framed in silver and blue refracted light.

“How many GPUs should the university buy?” sounds like the first design question. It is usually an output of several earlier decisions: which research and teaching must run, what quality is acceptable, how quickly work must finish and how many people use it at once.

A useful specification describes that service first, in terms a research office, a faculty and an IT team can all agree on. Then hardware options can be tested against the same requirement.

Describe the service target

A campus research assistant and an overnight analysis job value time differently. For the assistant, waiting for the first response and the pace of the answer matter. For a research group’s batch work, the deadline and total completed work may matter more.

In practice · illustrative example

Two targets that lead to different tests

Interactive
A research assistant answers representative questions at the agreed quality, with the first words appearing in about a second and a steady reading pace, while 200 researchers and staff use it during the working day.
Batch
A research group summarizes 400,000 archive documents between 8 p.m. and 6 a.m., including loading and writing results. That is a sustained rate of about 11 documents every second, all night.

The first target is limited by response time; the second by total throughput. The same hardware can pass one and fail the other, so neither can be reduced to a GPU count without testing the actual workload.

MLPerf Inference's workload scenarios distinguish different ways requests arrive and results are measured. Use a benchmark that resembles your intended service; a high throughput result under another scenario is not a response-time promise.

Ask what must fit in memory

Model weights are only part of memory demand. Runtime overhead, context and concurrent requests also matter. For many language-model deployments, the key-value cache grows with the work being served; Hugging Face's cache documentation explains the trade-offs among cache strategies.

The arithmetic is worth seeing once. Take an openly published 70-billion-parameter model, Llama 3.1 70B, whose configuration lists 80 layers and 8 key-value heads of 128 dimensions. At 16-bit precision, every token held in context costs about 0.33 MB of cache (2 × 80 × 8 × 128 × 2 bytes).

141 GBModel weights at 16-bit precision
86 GBCache for a class of 32 students, each with 8,192 tokens in context
≈ 227 GBBefore runtime overhead, for that one service
≈ 114 GBThe same service with 8-bit weights and cache

One class of 32 students needs more than half as much memory as the model itself. One researcher working through a 128,000-token thesis or set of papers needs about 43 GB of cache alone. Lower precision roughly halves both numbers, but it is a quality decision to test, not a free saving. The same model can therefore call for very different hardware for a teaching lab, a single research group or a campus-wide service.

Record the model version, precision, context lengths and concurrency used in a test. A smaller model that meets the quality target may permit a simpler system. A model that fits at one context length may not serve the intended users at another.

Choose the topology from the workload

A workload might fit on one accelerator, require several within a server or span multiple servers. Serving separate model replicas differs from splitting one model across devices. vLLM's parallelism guidance explains why memory and communication affect that choice.

Multi-server designs bring networking, storage and operational dependencies. Specify the fabric and test the communication pattern instead of assuming that adding GPUs produces proportional improvement.

Bring the facility into the same conversation

Check rack space, weight, delivery access, electrical feeds and cooling in the campus data center or colocation site before finalizing the configuration. Distinguish equipment ratings, planning allowances and measured consumption; they answer different questions.

Published MLPerf data-center results can provide useful comparative evidence within their stated conditions. They do not replace a facility review or a test of your exact software and workload.

Price the service and its next step

Include hardware, networking, storage, software, facility work and support in the comparison. Identify the next constraint as more courses and research groups join: memory, response time, storage throughput, power or staff capacity.

A defensible configuration explains why each major component is present, what was measured and what remains an assumption. That gives the IT team, the research office and the budget holder a clearer basis for approving the quote.

Five answers make the first sizing conversation far more productive:

  1. The model, its version and the quality bar it must meet, with how that quality will be checked.
  2. The response-time target for interactive use, or the deadline for batch work.
  3. Peak concurrent users, from a single research group to a full class, and the context lengths they actually need.
  4. Where it will run, with the available space, power and cooling.
  5. What the service should do next academic year, so the first design leaves room for it.

Share this perspective

Share the public article. Your message and audience are up to you.

The conversation continues.

Follow NexusIgnite on LinkedIn for ideas, updates, and perspectives.

Follow on LinkedIn