Amazon Web Services introduced the SageMaker JumpStart Model Leaderboard on November 18, 2026. The company says it covers more than seventy language models in SageMaker Studio, presenting three quality benchmarks alongside latency, token throughput and estimated instance cost. AWS runs the quality tests and measures the deployment configurations. A common platform and visible settings can reduce the work of collecting unlike numbers from separate model announcements. The result is most useful as a structured starting point for evaluation, with each score tied to what was tested and each deployment metric tied to the configuration that produced it. AWS launch announcement

Consider a team choosing a model for an internal help desk. It needs correct answers, an output format its application accepts and a first response before employees assume the page has stalled. It also has a budget that runs while the endpoint is available, including quiet periods. The model with the largest general knowledge score might suit the task, but so might a smaller one on a different machine. Putting several measurements in one view makes that decision easier to frame. The crucial comparison then moves from which name sits first to which combination meets the team's quality and service requirements at an acceptable cost.

Three tests keep their own meaning#

The quality columns at launch are BIG-Bench Hard, GPQA Diamond and MMLU-Pro. AWS displays them separately. BBH comes from a set of challenging reasoning tasks in BIG-Bench. Its original research showed that prompting choices changed how much of a model's ability the test revealed. That history is a reason to value a shared setup inside a table. It is also a reason to preserve the setup when comparing the same model elsewhere. A score belongs to a model responding under a particular prompting and generation procedure. The benchmark name identifies the questions, and the method determines how the model encounters them. AWS benchmark description · BBH's original paper

GPQA's original research introduced expert-written questions in science and investigated how difficult they were for knowledgeable nonexperts. Diamond is a specific subset used in many model comparisons. It can provide evidence about answering difficult, constrained questions. A help-desk model may need some of the same reasoning, yet the live task also asks for the correct company policy, a current document and appropriate handling of an ambiguous request. A high science-question score supplies a useful signal among others. A test set drawn from the team's actual questions can establish how far that signal transfers to its own service. GPQA paper

MMLU-Pro expands the challenge of broad multiple-choice evaluation, including more reasoning and a larger choice set than the original MMLU. The authors also studied sensitivity to prompt styles. Its spread of subjects gives another view of capability, with the score defined by that task format. A company deploying a system may care about extracting a field, producing valid structured output or choosing when to ask for clarification. Those requirements can be added to a local test alongside the general score. A benchmark that reaches across subjects can narrow the field while leaving room for measures of the exact work the model is being asked to do. MMLU-Pro paper

Keeping the three columns separate is a useful design choice. It avoids assigning one universal weight to scientific questions, broad knowledge and reasoning puzzles. A reader can examine the part closest to their needs. It also makes a difference in ranking visible. A model could lead one column and trail another, with neither result describing all possible work. A team can give its business requirements their own weights, decide that some conditions are mandatory and then compare candidates that remain. That produces an explicit decision a colleague can review, with priorities that remain visible when the choice is revisited.

The shared method has boundaries#

AWS says it uses the same evaluation harness and settings across models, but the launch also identifies an important qualification. Reasoning and non-reasoning models use different generation parameters, and GPQA Diamond results for reasoning models are averaged across ten passes. The company calls comparisons between those types approximate. The extra effort can improve a model's chance of a correct answer and increase the compute used to obtain it. A reader who wants to compare the result with another table should check which attempt is scored, how many times it was run and what budget the model received. Those conditions are part of the result. AWS's testing qualifications

Ten passes can describe several kinds of statistics, so the publication of the exact aggregation matters. An average of individual attempts answers a different question from whether any attempt succeeded. A product with one chance to answer a user might be closer to the first situation. A service that can spend more resources, check candidates and choose one has another process to evaluate. The launch's wording should be carried along until the detailed run procedure provides more information. Even with the same dataset, a reader cannot recover a comparable measurement by choosing an aggregation that happens to make the numbers look similar.

Close results also need a sense of variation. A stochastic model can produce different answers across repeated runs. A fixed test set contains a particular mix of questions, which may weigh heavily on a small difference. AWS advises treating small gaps as ties. That is a useful starting rule when deciding where to spend further test time. A clear difference on relevant work is easier to act on than a tiny general-score edge accompanied by a much larger difference in latency or spending. Confidence intervals and a repeatable local sample would help determine which differences remain when the setup is rerun.

The provider is also the operator of the platform being measured. It has an advantage in testing configurations that a customer can deploy, with a commercial interest in making that path useful. Those two facts can be held together. The record becomes stronger when methodology, model revisions and serving settings are accessible enough for another evaluator to understand the comparison. AWS's launch links to JumpStart documentation for more detail. The linked material read for this article gives general model access and deployment guidance, leaving a full independent replication dependent on a more complete set of run details and access to the configurations. Linked JumpStart documentation

The menus choose a machine as well as a metric#

Two controls above the leaderboard change the performance and cost view. The use-case profile selects the test workload closest to an application, with Max Context as the default. The optimisation goal selects one deployment configuration for that model and workload. The public quality scores stay the same as the menus change. The interface allows a deployment measurement to change while the general benchmark columns remain fixed. A team selecting a particular row needs to retain the profile and configuration with the model name. Otherwise it has recorded only part of the object it compared. AWS explanation of profiles and goals · Use-case-based deployment guide

Balanced chooses the configuration with the lowest estimated cost per token. Cost optimized chooses the lowest hourly instance price. A larger instance can cost more per hour and generate enough useful output to reduce the cost of each token under the test load. A cheaper instance can be appropriate for a small workload while processing requests more slowly. The names describe different decisions, and the definitions are more informative than the labels. They give an engineer a way to explain why choosing the lowest hourly line might still lead to a less economical service at sustained volume. AWS's optimisation definitions

Latency optimized chooses the configuration with the lowest time to first token. Throughput optimized chooses the highest throughput and can use speculative decoding. The distinction maps to different work. A person in a chat needs a prompt first sign that the system is answering. A scheduled process handling a large queue can trade some first-response delay for more finished work. If the same endpoint serves both, a single configuration may be a compromise. A useful production test should therefore include the mix that the actual application receives and retain the service requirements that motivated the choice. AWS's performance goal definitions

A configuration is also a reproducibility record. Model revision, instance type, serving software and generation settings are the pieces a team should preserve when it tests a candidate. A later revision may alter response behaviour or capacity, even if the model's broad name is unchanged. The team can then decide whether a new result follows a model change, a hardware change or its own prompt. The leaderboard creates an accessible first comparison. Saving the configuration turns that comparison into the beginning of an experiment and makes a temporary position in the table understandable later.

Waiting starts before the first token#

Time to first token describes how long a request waits before a stream begins. It is useful for a person watching a chat box, but it is only one point on the request's timeline. The application might need a complete structured object before it can display anything useful. A long response can start quickly and finish later than a short one that starts more slowly. A model spending time on internal reasoning can produce a different pattern again. The service should measure the delay to the output a user can actually use. First token, last token and successful task completion can all be recorded without pretending they answer the same question.

Prompt length changes that timeline. A question alone and a question accompanied by a large set of documents require different input processing. Repeated requests can benefit from caching if the deployed system supports it and the workload produces suitable repeated prefixes. Traffic bursts can introduce a queue before generation. Network travel and retrieval add time outside the model endpoint. Standardised profiles help create a baseline for those effects. A local test with representative input lengths and concurrent requests reveals how much of the baseline carries into the service that people will see.

Tail latency deserves its own record. A typical response might look fine while a smaller group waits much longer. For an interactive service, that smaller group can dominate complaints. A sample showing a median and a high percentile gives a clearer account than a single average. Failures and timeouts should stay visible with the latency measurements, because dropping slow failures can make an endpoint look better by removing its worst behaviour. A repeated test over the workload's normal variation would help distinguish a stable configuration from one that worked well during a short quiet run.

Tokens per second have a denominator#

AWS describes throughput per request in tokens per second. It provides one view of generation speed. Capacity for an application also depends on simultaneous requests, prompt processing, output length and the point at which queues form. A configuration may perform well for a single request and handle a busy service differently. For an overnight queue, a team might care about documents finished per hour at an accepted error rate. For a chat, it might care about simultaneous sessions meeting the response target. The token metric helps compare a common test, and task-level capacity connects it to the work that needs paying for. AWS's metric definitions

The tokenizer introduces another difference. Two models can use different numbers of tokens to express the same content, especially across languages or in code. A higher token rate can therefore correspond to a smaller difference in words or useful records completed. The output can also include varying amounts of explanation. If an extraction service needs ten fields, a verbose model may generate more tokens without completing more work. Measuring valid, accurate outputs per unit time and cost helps a team compare those behaviours. The general throughput column remains a useful baseline with its unit intact.

Speculative decoding is a serving technique that can change speed while preserving the intended output behaviour under its algorithmic conditions. Its benefit depends on the models and workload used by the deployment. If a throughput-selected configuration uses it, the customer should test the same arrangement it intends to run. The example illustrates a broader point about cloud model selection. The model and the serving implementation jointly produce the service. Comparing only the model name can hide an optimisation that makes one configuration appealing or a requirement that the team's eventual environment cannot provide.

The hourly price keeps running#

The launch says the cost column is an estimate of SageMaker instance cost at the default instance's on-demand hourly rate. It excludes transfer, storage, inference component charges and negotiated or committed discounts. A reader can use that estimate to compare the configurations shown, while a full budget still needs the other services. The team must also decide how many hours the endpoint is provisioned and how its demand varies. An hourly compute price can be predictable and leave a substantial idle cost in a service with occasional requests. A configuration's low unit cost under sustained testing says less about a lightly used endpoint. AWS's cost scope

Suppose two candidates satisfy the quality threshold. One needs more hourly spending but finishes a queue faster. The other has a lower hourly rate and runs longer. To compare the queue cost, account for the time each is active and any minimum or continuing deployment requirements. To compare an always-available chat service, use the expected number of serving hours and the capacity needed at busy periods. These can produce different choices from the same rows. A team makes the comparison more honest by writing down the operational schedule before turning a test's cost per token into its monthly forecast.

The meaning of a successful request belongs in that cost. An inexpensive model that often needs a second attempt or a human correction can be costly per completed job. A more accurate model can still be an unsuitable purchase if the required accuracy gain concerns work the team never does. A useful evaluation counts correct accepted outcomes and records retries, escalation and review time. This lets the team see whether lower inference spending saves money in the process as a whole. The leaderboard's instance estimate begins the arithmetic. It takes the application test to complete it.

Requirements come before rank#

The tool shares JumpStart's search and filters, including licence, provider and input or output modality. AWS recommends filtering hard requirements first. That is a sound order. A model needs terms compatible with the intended use and the inputs the application can send. Its deployment must also fit the team's region, capacity and access policies. A row with a strong score can remain irrelevant to a system that needs a different modality or cannot accept its licence. Filtering those conditions early reduces the number of quality tests a team needs to pay for, leaving candidates it can plausibly deploy. AWS leaderboard filters · JumpStart's licence guidance

Coverage has its own boundary. The launch focuses on language models and AWS says every JumpStart model has yet to receive scores. A missing score therefore identifies missing measurement in this view. It offers no direct evidence that the model would perform badly for the team. An organisation with a specialised or previously qualified candidate can keep it in a local comparison even if the public table omits it. The provider catalogue is useful for discovering deployable options, while the evaluation design should preserve candidates that make sense for the actual requirement.

From a shortlist to an answer#

A representative test begins with the application's real distribution of tasks. For the help desk, that includes frequent clear questions, ambiguous requests, long documents, changed policies and cases where the correct next step is clarification. Expected answers can be written before looking at the model's result. Reviewers can check factual support, output format, completeness and whether the response lets the employee act. A dataset with those categories gives the team an error map. It can then ask which failures a prompt or retrieval change might fix and which persist across reasonable settings.

Language is another part of that distribution. An organisation serving French and English questions may need the same policy answered consistently in both. Code, abbreviations and a name written in another script can change token use and retrieval. A general knowledge score does little to establish the exact terms a local service requires. Including those cases before choosing a model helps avoid discovering the gap in the first customer week. The team can preserve aligned examples and have qualified reviewers examine the resulting meaning. An answer that keeps the format while changing an important qualification should remain a failed answer in the record.

The test should keep the comparison repeatable. The same task set, model revisions and documented prompts allow each configuration to receive the same work. Repeated attempts reveal variation while retaining the distinction between average success and choosing the best of several answers. A separate sample held back from prompt tuning can indicate whether improvements transfer beyond examples already used in development. Local testing can focus on a credible account of whether this application meets its requirements and how certain the team is about that finding.

After deployment, the same record can become a small regression set. A model version change, a new retrieval source or a different serving configuration can be checked before it reaches all users. The live service should also retain aggregate information about response times, accepted results and failure categories under its own data policies. That helps distinguish a new model issue from missing source material or a traffic spike. Comparing later results with the original choice makes the selection an ongoing engineering decision. The most useful outcome is a service whose quality and cost remain understood as the workload and the underlying products change.

AWS already offers model evaluation integrations in Studio through its foundation model evaluation tools. The documentation covers tasks such as summarisation, question answering and classification, with availability varying across models. Those tools can form part of the next stage after the leaderboard. A team may also need custom checks for its own response format, policy and service operation. A dashboard comparison becomes materially more useful when it leads into evidence the team's maintainers can inspect and rerun. The deployment choice can then draw on a documented test record that gives the ranking practical meaning. AWS's Studio evaluation documentation

The new view improves the starting position for an AWS customer by joining quality scores with deployment measurements and stated cost scope. Its value will grow if the configurations, methods and revisions remain easy to inspect as coverage expands. The next evidence for a buyer comes from its own workload on the selected configuration, with quality, waiting time and complete cost measured together. A leaderboard can make the first decision much faster. A saved and repeatable application result is what makes the final choice defensible.