{"id":15601,"date":"2026-10-30T15:05:10","date_gmt":"2026-10-30T07:05:10","guid":{"rendered":"https:\/\/ai-stack.ai\/?p=15601"},"modified":"2026-10-30T15:23:28","modified_gmt":"2026-10-30T07:23:28","slug":"what-is-llm-benchmark","status":"publish","type":"post","link":"https:\/\/ai-stack.ai\/en\/what-is-llm-benchmark","title":{"rendered":"LLM Benchmarking in Ten Minutes: Latency, Throughput, and Load Models"},"content":{"rendered":"\n

2026\/10\/22 \u00a0 INFINITIX Product Dep.
Introduction: How Many Users Can This Server Handle at Once?<\/strong><\/p>\n\n\n\n

Your team has deployed an open-source large language model (LLM) on a GPU server. You send a prompt with curl, and text starts streaming back in just over a second. It looks fast. But “looks fast” does not answer the questions that matter: Can the system handle 10 concurrent users, or 200? How many seconds may pass before the first word of the reply appears and the experience still counts as acceptable? How fast must streamed text appear for the output to feel smooth?<\/p>\n\n\n\n

Answering questions like these requires an LLM benchmark, that is, a performance benchmark test. This article does not cover tool installation. It focuses on the logic behind the test: which metrics to measure, why a single number is never enough, how to apply load, and which details can distort the results. Along the way, NVIDIA’s open-source AIPerf tool is used as an example of how these concepts appear in practice.<\/p>\n\n\n\n

1. What Is an LLM Benchmark?<\/strong><\/h1>\n\n\n\n

The term “LLM benchmark” is commonly used in two senses. Capability benchmarks<\/strong> (question sets such as MMLU and HumanEval) measure whether a model answers correctly; they do not evaluate hardware performance. Inference performance benchmarks<\/strong> measure how fast a given model responds, and how many users it can serve concurrently, once it is deployed on specific hardware and a specific inference engine. This article is about the second kind, which is what most teams actually want to know.<\/p>\n\n\n\n

These tests, however, cannot simply reuse the approach of a conventional HTTP load test, because LLM inference differs from a traditional API in several important ways:<\/p>\n\n\n\n

    \n
  • Responses stream back one token at a time.<\/strong> When the first token arrives (Time to First Token, TTFT), the interval between subsequent tokens (Inter-Token Latency, ITL), and when the full response completes (End-to-End Request Latency, E2E latency) are three different times.<\/li>\n\n\n\n
  • The amount of work per request varies.<\/strong> The model can stop generating at different points, and longer outputs generally occupy the GPU for longer. If two test runs have different output-length distributions, their throughput cannot be compared directly.<\/li>\n\n\n\n
  • Computation has two phases: prefill and decode.<\/strong> Prefill processes the entire input prompt; decode generates output tokens step by step. The two phases have different resource demands and different performance bottlenecks.<\/li>\n\n\n\n
  • Requests affect one another.<\/strong> The inference engine may use continuous batching to schedule the decode work of multiple requests together. As a result, the token generation speed of any single request also depends on the other requests running at the same time.<\/li>\n\n\n\n
  • GPU memory can become the bottleneck first.<\/strong> Every in-flight request occupies KV cache, and the requirement grows with sequence length. When memory approaches or reaches its limit, latency can deteriorate abruptly rather than gradually.<\/li>\n<\/ul>\n\n\n\n

    Every benchmark run should fix at least five things up front: the test target (the API endpoint, the model, and whether streaming is enabled); the workload (prompt source, Input Sequence Length (ISL), and Output Sequence Length (OSL)); the load model (how many concurrent requests, sent at what cadence); the metrics to measure; and the pass criterion, the SLO (Service Level Objective). Treat the benchmark as a controlled experiment and change only one variable at a time, so that the cause of any change in the results is easy to identify.<\/p>\n\n\n\n

    2. Why a Single Number Is Not Enough<\/strong><\/h1>\n\n\n\n

    2.1 Four Core Metrics on One Timeline<\/h2>\n\n\n\n

    All metrics come from the same timeline: t\u2080 is the moment the request is sent, t\u2081 is the moment the first token arrives; more tokens follow until the last token arrives at t\u2099.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Figure 1. Timeline of a streaming request as observed by the client, and the interval each core metric measures<\/p>\n\n\n\n

      \n
    • TTFT (time to first token) = t\u2081 \u2212 t\u2080.<\/strong> This interval includes the network round trip, server-side queueing, and the prefill time needed to process the whole prompt. It reflects whether users feel that “the system is responding.” It is highly sensitive to prompt length and is often the first metric to deteriorate as concurrency rises.<\/li>\n\n\n\n
    • ITL (inter-token latency) = (request latency \u2212 TTFT) \u00f7 (OSL \u2212 1).<\/strong> This is the average interval between tokens after the first one arrives, and it reflects how smoothly decode output flows. TPOT (time per output token) is a closely related term common in the industry. The denominator is OSL \u2212 1 because the wait for the first token is already counted in TTFT.<\/li>\n\n\n\n
    • Request latency = t\u2099 \u2212 t\u2080.<\/strong> This is the total time to complete the full response. Two requests with the same total latency can still deliver very different user experiences: one may “wait a long time, then stream quickly,” while the other “starts quickly, then streams slowly.” Total latency should therefore always be read together with TTFT and ITL.<\/li>\n\n\n\n
    • Per-user output speed = 1 \u00f7 ITL.<\/strong> This is roughly how many tokens a single user receives per second. When it drops to single-digit tokens per second, users usually notice that output feels slow.<\/li>\n<\/ul>\n\n\n\n

      2.2 Throughput: The System’s Overall Capacity<\/h2>\n\n\n\n
        \n
      • Benchmark duration:<\/strong> the time from when the first request is sent to when the last response is received. This duration is the denominator in throughput calculations.<\/li>\n\n\n\n
      • Output token throughput = total output tokens \u00f7 duration.<\/strong> The total number of output tokens the entire system produces per second (Tokens Per Second, TPS).<\/li>\n\n\n\n
      • Request throughput = completed requests \u00f7 duration.<\/strong> The number of requests the system completes per second (Requests Per Second, RPS). How it converts to token throughput depends on the average output length per request.<\/li>\n<\/ul>\n\n\n\n

        The most commonly confused pair is “each user receives 20 tokens per second” and “the system produces 2,000 tokens per second.” These numbers describe different things and can move in opposite directions: as concurrency increases, batching may raise the system’s total output while each decode step slows down, so the speed seen by an individual user falls. Whenever you see the unit tokens\/s, first confirm whether it means per user or for the whole system.<\/p>\n\n\n\n

        2.3 Why the Average Is Not Enough<\/h2>\n\n\n\n

        Latency distributions are typically right-skewed with a long tail: most requests complete quickly, but a few can take several times longer because of queueing or memory pressure. The mean is pulled up by these slow requests, yet it does not show how bad the tail actually is. In an example from the AIPerf methodology documentation, p99 is more than 70% higher than the mean. In general, p50 describes the latency of a typical request; p95 means 95% of requests finish within that value; p99 means 99% of requests finish within that value, with only the slowest 1% beyond it. SLOs are usually defined at p95 or p99. A good benchmarking tool therefore keeps the raw record of every request, so that any percentile can be recomputed afterward.<\/p>\n\n\n\n

        2.4 Goodput: Measuring Speed and Service Quality Together<\/h2>\n\n\n\n

        Latency and throughput usually pull against each other. To answer “how many users can the system handle,” both must be judged against the same criterion: goodput = requests that meet all SLOs \u00f7 duration<\/strong>. For example, you might set TTFT < 500 ms and ITL < 50 ms; only requests that meet every condition count toward the goodput numerator, so goodput can never exceed request throughput. Choosing the wrong metric for capacity planning can change the answer dramatically: in an example in the AIPerf documentation, a throughput-based estimate calls for 38 servers, while a goodput-based estimate calls for 135. The latter is the number that actually means “the work gets done, and the user experience meets requirements.”<\/p>\n\n\n\n

        Metric<\/strong><\/td>Formula<\/strong><\/td>What it tells you \/ where it matters<\/strong><\/td><\/tr>
        TTFT<\/td>t\u2081 \u2212 t\u2080<\/td>Whether users feel the system is responding<\/td><\/tr>
        ITL (TPOT)<\/td>(request latency \u2212 TTFT) \u00f7 (OSL \u2212 1)<\/td>Whether streamed text flows smoothly<\/td><\/tr>
        Request latency<\/td>t\u2099 \u2212 t\u2080<\/td>Batch jobs and agent workflows<\/td><\/tr>
        Output token throughput<\/td>\u03a3 OSL \u00f7 duration<\/td>Operating cost and GPU utilization<\/td><\/tr>
        Goodput<\/td>Requests meeting SLO \u00f7 duration<\/td>Capacity planning and service quality<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

        2.5 Why TTFT and ITL Cannot Be Measured Without Streaming<\/h2>\n\n\n\n

        When a request does not use streaming, the server returns the whole response at once after it is complete. The client’s timeline then shows only t\u2080 and t\u2099: the first and last tokens arrive together with the full response. TTFT becomes equal to request latency, and ITL can no longer be meaningfully calculated. TTFT, ITL, and per-user output speed are therefore metrics that apply only to streaming output<\/strong> (AIPerf labels them streaming-only). Conversely, if a product does not use streaming at all, such as back-end batch summarization, these metrics may not matter. Choose metrics based on how the product is actually used.<\/p>\n\n\n\n

        Two settings are especially likely to distort measurements.<\/strong> First, uncontrolled output length: the model may hit an end-of-sequence token early and stop, making the actual OSL shorter than intended and skewing throughput and ITL. A more rigorous test uses parameters such as ignore_eos or min_tokens to make the model generate the configured number of tokens. Note that some services still use the field name max_tokens, and some silently ignore unsupported parameters. Second, a misconfigured tokenizer: the client uses a tokenizer to count ISL and OSL, and if that tokenizer does not match the model, every token-based metric may be off.<\/strong><\/p>\n\n\n\n

        3. Load Models: How You Apply Load Determines What You Measure<\/strong><\/h1>\n\n\n\n

        A load model is not just a “size” setting. More importantly, it is the mechanism that decides when requests are sent and by what rule.<\/p>\n\n\n\n

        \"\"<\/figure>\n\n\n\n

        Figure 2. Closed-loop (fixed concurrency) vs. open-loop (fixed arrival rate) load models<\/p>\n\n\n\n

        Concurrency (fixed concurrency, closed loop):<\/strong> simulates N virtual users. Each user sends one request, waits for the complete response, and then sends the next. At any moment, the number of in-flight requests never exceeds N. This creates a feedback effect: the slower the system, the later the next request is sent, so the overall send rate drops automatically. It is well suited to measuring the steady-state capacity the system can sustain when N users use it continuously. Results are stable and easy to reproduce, but the load is relatively gentle compared with bursty traffic.<\/p>\n\n\n\n

        Request rate (fixed arrival rate, open loop):<\/strong> sends \u03bb requests per second on a schedule, regardless of whether earlier requests have finished. If the system cannot keep up, unfinished requests accumulate, and latency can rise rapidly. This approach is closer to real traffic arriving at a fixed rate and is suited to verifying whether latency still meets the SLO under expected peak load. It is, however, very sensitive to the value of \u03bb: slightly below the system’s capacity, everything may look fine; slightly above it, the queue may keep growing.<\/p>\n\n\n\n

        The two methods answer different questions. “What is the maximum number of users we can serve at once?” calls for a closed-loop test; “What is p99 latency at 50 requests per second?” calls for an open-loop test. A common misuse goes like this: a closed-loop test shows “TTFT p99 is only 800 ms at 100 concurrency,” yet once in production, 100 requests per second arrive, and the system cannot cope. The reason is that when the system slows down, 100 concurrency in a closed-loop test may actually be sending only 30 requests per second.<\/p>\n\n\n\n

        Request arrival patterns:<\/strong> an open-loop test also has to decide the spacing between requests. Constant mode sends one request every 1\/\u03bb seconds, with perfectly regular spacing. In Poisson mode, inter-arrival times follow an exponential distribution; the mean interval is also 1\/\u03bb, but bursts of requests and quiet gaps occur naturally. It is the mathematical model for independent users sending requests at random, and it is AIPerf’s default mode. With constant spacing, the queue is less likely to build up suddenly; in Poisson mode, five requests may arrive within the same second, creating a latency tail. As a result, p99 measured with constant mode alone is usually more optimistic than results from tests that resemble real traffic.<\/p>\n\n\n\n

        3.1 Load Sweeps and the Knee<\/h2>\n\n\n\n

        A single load point only tells you how the system behaves at that load. To answer “how much load can the system take at most,” increase concurrency step by step, for example, on a logarithmic scale of 1, 2, 4, 8, and so on, run a full benchmark at each load point, and plot throughput and latency curves.<\/p>\n\n\n\n

        \"\"<\/figure>\n\n\n\n

        Figure 3. Concurrency sweep: throughput saturates while latency can surge (left); the intersection with the SLO threshold determines capacity (right)<\/p>\n\n\n\n

        As concurrency increases, throughput usually keeps rising, but by smaller and smaller increments, and beyond a certain point it may flatten or even dip slightly. Latency may barely change at first and then climb sharply past a certain point. This transition region is called the knee. A good operating point usually sits a little before the knee. If you add the SLO latency threshold to the chart, the point where the latency curve crosses the threshold line gives the highest concurrency that meets the SLO; combined with the goodput at that load, it provides a more complete answer to the performance question.<\/p>\n\n\n\n

        3.2 Warm-up and Avoiding Interference Between Tests<\/h2>\n\n\n\n

        Warm-up:<\/strong> for the first few tens of seconds after an inference service starts, its behavior can differ greatly from steady-state operation, because of loading model weights, compiling CUDA kernels for the first time, allocating the KV cache memory pool, and establishing connections. In an example from the AIPerf methodology documentation, TTFT during the first 10 seconds is about 41% higher than at steady state. Including this period in the statistics can make p99 look worse than actual steady-state performance, and the size of the effect also changes with the benchmark’s duration. A better practice is to run a warm-up phase first, mark those requests as warm-up data and exclude them, and only then begin the formal measurement.<\/p>\n\n\n\n

        Do not run multiple tests at once:<\/strong> suppose two engineers each run a benchmark against the same endpoint, each at 32 concurrency. The server may actually be handling 64 in-flight requests. These requests share the same GPU, the same batch scheduler, and the same KV cache pool, so latency in both reports may be worse than a standalone 32-concurrency test, and the cause is hard to see from either report alone. Run only one benchmark at a time against a given endpoint, and confirm before starting that no other test is running, rather than relying on verbal reminders. The same principle applies to the client machine generating the load: if it runs short of CPU, exhausts available ports, or has its event loop blocked by JSON parsing, the measured latency will include client-side bottlenecks and no longer reflect server performance alone.<\/p>\n\n\n\n

        Three more factors affect whether results are comparable.<\/strong> The first is sample size: collect at least several hundred requests at each load point for p99 to be meaningful; with only 100 requests, p99 is essentially the single slowest request. The second is workload characteristics: synthetic data (fixed ISL\/OSL) gives controllable, reproducible capacity numbers, while real conversation logs often have long-tailed length distributions that can expose problems synthetic data does not. Both are worth testing, but their results should not be compared directly. The third is reproducibility: fixing the random seed makes prompt content and Poisson arrival intervals reproducible, but latency itself still varies naturally. When comparing two configurations, run each several times and compare medians, and record the model, inference engine version, hardware, tokenizer, load model, warm-up settings, and SLO thresholds; otherwise the same report will be hard to reproduce later.<\/p>\n\n\n\n

        4. AIPerf: How These Concepts Appear in a Real Tool<\/h1>\n\n\n\n

        AIPerf is an open-source LLM inference performance benchmarking tool from the NVIDIA Dynamo team. It is licensed under Apache 2.0 and is the successor to GenAI-Perf. Rather than walking through how to use it, this section maps three of its design choices to the concepts above.<\/p>\n\n\n\n

        \"\"<\/figure>\n\n\n\n

        Figure 4. AIPerf’s three planes: the control plane sets the load pattern, the data plane measures, and the analysis plane computes statistics<\/p>\n\n\n\n

        Multiple processes and credits (request permits):<\/strong> AIPerf runs about ten services, each in its own process, communicating over ZeroMQ. The system is divided into a control plane (which decides when and how many requests to send), a data plane (where workers send the actual HTTP requests and record timestamps), and an analysis plane (which computes metrics and exports results). One of the cores of AIPerf’s load control is the credit, which can be thought of as a “request pass”: a worker can send a request only after obtaining a credit, and cannot send one without it. In fixed-concurrency mode, the Timing Manager issues N credits at the start and issues a new one only each time one is returned; this is the closed loop. In request-rate mode, credits are issued on a Poisson arrival schedule without waiting for earlier ones to be returned; this is the open loop. In other words, the core of a load model is deciding under what conditions the next pass is issued.<\/p>\n\n\n\n

        Timestamps and the tokenizer:<\/strong> workers record when each request is sent and when each streamed chunk arrives at the client. TTFT, ITL, and request latency are all computed by subtracting timestamps; benchmark duration is computed from the actual elapsed time of the whole test. ISL and OSL are counted by the tokenizer, so a tokenizer must be specified when running the benchmark, and it must match the model the server has actually loaded.<\/p>\n\n\n\n

        Sweeps and adaptive search:<\/strong> a grid sweep tests eight concurrency points on a logarithmic scale by default: 1, 3, 7, 19, 52, 139, 373, and 1000. Each point can also be repeated several times to estimate confidence intervals. To find the highest concurrency that meets the SLO directly, use a search recipe such as max-concurrency-under-sla. The default algorithm first raises the load step by step until it finds the first point that exceeds the threshold, narrowing the boundary to an interval. It then applies isotonic regression to reduce measurement noise, interpolates the crossing point, and runs additional tests near the boundary until the interval is narrower than 5%. The whole process takes about 13 to 25 iterations. By comparison, testing only the fixed eight points may only tell you that the capacity boundary lies somewhere between 139 and 373. If the goal is instead to find the maximum throughput while staying within the threshold, AIPerf can use Bayesian optimization: it builds a Gaussian process model that predicts throughput as a function of concurrency, and at each step tests a point that is either predicted to perform well or has high uncertainty; if several consecutive steps bring no improvement, the search stops. As the number of parameter dimensions to explore grows, this approach gains an advantage over a fixed grid sweep.<\/p>\n\n\n\n

        Conclusion<\/h1>\n\n\n\n

        A performance question can only be answered once it is restated in a form that can be measured and verified: “For the specified model, inference engine, and ISL\/OSL workload, using a closed-loop concurrency sweep with warm-up data excluded, what is the highest concurrency at which TTFT p95 stays below X ms and ITL p95 stays below Y ms? And what is the goodput at that point?” The answer can then be presented as a single chart: a throughput-and-latency curve, an SLO threshold line, and the point where they intersect.<\/p>\n\n\n\n

        Before you start testing, confirm the following:<\/p>\n\n\n\n

          \n
        1. Is streaming enabled? This determines whether TTFT and ITL can be measured.<\/li>\n\n\n\n
        2. Is output length controlled? Does the tokenizer match the model?<\/li>\n\n\n\n
        3. To determine system capacity, use a closed-loop concurrency sweep; to verify latency under peak traffic, use an open-loop fixed arrival rate. Open-loop tests can use the Poisson arrival pattern.<\/li>\n\n\n\n
        4. Are warm-up requests excluded? Is any other benchmark running against the same endpoint?<\/li>\n\n\n\n
        5. Beyond the mean, check percentiles such as p95 and p99; beyond throughput, check goodput. Keep the complete test configuration and the per-request records.<\/li>\n<\/ol>\n\n\n\n

          Tools will be updated, and GPUs will be replaced, but this checklist will still apply.<\/p>\n\n\n\n

          References<\/h1>\n\n\n\n

          Zhong et al., “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving”, OSDI 2024 (goodput)<\/p>\n\n\n\n

          AIPerf source code and documentation: https:\/\/github.com\/ai-dynamo\/aiperf<\/a> (including the architecture, metrics-reference, comprehensive-llm-benchmarking, timing-modes-reference, and search-recipes documents)<\/p>\n\n\n\n

          Yu et al., “Orca: A Distributed Serving System for Transformer-Based Generative Models”, OSDI 2022 (continuous batching)<\/p>\n","protected":false},"excerpt":{"rendered":"

          LLM benchmark is commonly used in two senses. Capability benchmarks (question sets such as MMLU and HumanEval) measure whether a model answers correctly; they do not evaluate hardware performance. Inference performance benchmarks measure how fast a given model responds, and how many users it can serve concurrently, once it is deployed on specific hardware and a specific inference engine. This article is about the second kind, which is what most teams actually want to know.<\/p>\n","protected":false},"author":253372376,"featured_media":15618,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[96987570,96987598],"tags":[96987685,96987655],"class_list":["post-15601","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technical-support","category-product-features","tag-ai-stack-features-2","tag-llm-en"],"blocksy_meta":[],"acf":[],"jetpack_shortlink":"https:\/\/wp.me\/ph344V-43D","jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/i0.wp.com\/ai-stack.ai\/wp-content\/uploads\/2026\/10\/63bc990e.png?fit=1920%2C1080&quality=100&ct=202603031250&ssl=1","_links":{"self":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts\/15601","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/users\/253372376"}],"replies":[{"embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/comments?post=15601"}],"version-history":[{"count":1,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts\/15601\/revisions"}],"predecessor-version":[{"id":15622,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/posts\/15601\/revisions\/15622"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/media\/15618"}],"wp:attachment":[{"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/media?parent=15601"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/categories?post=15601"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ai-stack.ai\/en\/wp-json\/wp\/v2\/tags?post=15601"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}