When an agent reads a dense form, it often starts with a low-resolution overview and spends a limited number of high-resolution crops on important regions. A crop has value partly because of what it lets the agent do next. A routing field may identify the relevant table when two reads remain. With one read left, the same field can consume the last opportunity without exposing the answer. A paper posted on November 18 proposes a protocol for testing whether visual agents account for that changing budget. It holds the public evidence and region mapping fixed while changing the remaining reads or the value of an action. Budget-paired visual-agent audit

The method also tracks the consequences of each observation through a complete reading history. Under a declared model of the document and the read tool, it computes the best possible continuation value after each observation. The difference between that value and the agent’s selected crop gives one measure of acquisition loss. A separate terminal measure evaluates the final answer after observation stops. This accounting offers a way to locate where a reading policy loses accuracy in the tested task model.

The authors apply the protocol to controlled routing tasks, multi-step hierarchies and 30 scanned forms from FUNSD. Their results draw attention to a practical distinction. A model can make a plausible initial choice and still continue with a fixed selection habit when the value of a region changes. In natural forms, apparent gains from crop selection vary with the answer reference, and multi-crop pipelines can trail whole-image reading while consuming more input tokens.

One crop can route the next#

The controlled task uses five document regions. One region is a selector, two are candidate answer branches, and two are distractors. A public binding says which role is attached to each physical region. The question’s rule defines how the hidden bits determine the answer. Each read returns a region value under a specified observation channel. The evaluator knows the prior and the channel for these controlled cases, so it can calculate the expected accuracy of every legal read sequence exactly.

In the switch task, the selector tells the agent which answer branch contains the relevant value. When the agent has two reads remaining, opening the selector first is best because it directs the second crop to the active branch. When there is one read remaining, opening either answer branch gives the answer directly. The selector has no time to route a follow-up. The optimal first action therefore changes when only the budget changes.

Stylized paired diagrams of an identical five-region reading map: with two read tokens the path follows the selector; with one token an optimal dashed switch to a branch is missed while a rigid path returns to the selector.

The paper constructs pairs with identical public evidence and the same region-role bindings. It changes the remaining budget or moves a task parameter that determines which action has greater expected value. A fixed role preference can look correct in the two-read case. The paired one-read case tests whether the policy adapts to the horizon. Observing the pair separates two explanations that an isolated accuracy score cannot resolve. A model may know what the selector means and still fail to account for the remaining read budget.

The same idea moves the action-value boundary. A branch may be more informative when the prior probability of an answer changes or the selector becomes unreliable. The experiment can move this boundary while preserving the rest of the scenario. A policy that always opens the selector may do well near one parameter setting, then overuse it when a direct branch becomes more valuable. A policy that responds to task value should change its choice near the point where expected continuation values cross.

This test relies on an explicit task model. The paper declares the prior over hidden document state and the distribution of region reads, then enumerates the possible outcomes. The results describe optimal decisions under those declarations. That lets the authors compute an exact reference for each read, and it also defines the interpretation boundary. A fielded document may contain uncertain region bindings, OCR noise, page context or correlated errors that differ from the controlled model. Those factors call for measured read models or additional audits.

Accuracy alone hides the route#

An isolated question-and-answer test records whether the final answer is correct. It does not show which region the agent selected, whether that choice was valuable at the time, or whether the policy adapted after receiving a return. The paper computes continuation values for every history. The remaining budget, the read history and the declared model jointly determine which action can preserve the highest expected terminal accuracy.

For each acquired region, the audit records the value of the best possible continuation and the value of the action the agent actually chose. Their difference is stepwise acquisition loss. Once the read budget is exhausted, the evaluator computes the best answer given the acquired evidence. The gap between that best terminal answer and the model’s response is terminal decision loss. Adding these terms across the complete trajectory decomposes the expected accuracy gap between an optimal policy and the observed policy.

That decomposition distinguishes two problems. An agent may collect a weak set of evidence by choosing low-value regions. It may also gather useful evidence and then answer incorrectly. A final error rate combines these cases. The trajectory record separates them under the evaluator’s declared prior and read model. The result can show where a policy deviated from an optimal continuation, while retaining the exact observations its own choices created.

The method gives a concrete example. With two reads remaining, the selector is optimal because it routes to the relevant branch. If the agent opens the selector and receives the value that makes one gate active, the next choice should open that gate. The recorded Qwen3-VL-8B rollout instead opens a distractor. That action loses 25 percentage points of conditional value in the constructed task. It then reads another leaf and leaves the gate unopened. The posterior can remain tied, so the final answer may be wrong despite having zero terminal decision loss under the model. The audit attributes that loss to acquisition rather than to the final choice.

The distinction between best-case posterior accuracy and observed correctness also needs care. If the acquired history leaves two answers equally probable, either terminal answer has zero decision loss relative to the best available choice, while the best possible accuracy remains only 50 percent. An agent can make the best terminal decision with insufficient evidence. The audit reports evidence sufficiency separately from its decision quality. That can be useful for visual tasks where the model’s observation policy and its answer policy are both under evaluation.

The paired results show a persistent selector habit#

The study tests five open-weight vision-language models in forty paired routing families each. Under fair priors and noiseless reads, the branch is optimal with one read left and the selector is optimal with two. Each suboptimal action in these anchor cases loses 25 percentage points. None of the five open-weight systems makes an optimal branch-to-selector switch in its forty pairs.

Several models choose the selector in every paired condition. Qwen3-VL-32B does so across all 2,320 confirmation acquisition queries. Other models distribute some choices across branches or distractors, but still make no correct switch across the forty anchors. Their mean loss on those carefully chosen, high-gap anchors is 12.50 to 15.62 percentage points. On other near-boundary states the mean loss is only 0.59 to 0.85 points. A low average over less diagnostic examples can conceal a rigid selection policy. The counterfactual pair makes the failure visible by putting the action boundary under pressure.

The behaviour is attainable. Hosted GPT-5 mini makes the optimal switch in 48 of 48 pairs when given an image plus map, and in 47 of 48 with the map alone. GPT-4.1 makes 44 of 48 with map-only input, while its image-plus-map protocol retains the selector and has zero transitions. These hosted results use their own dated endpoints and evaluation protocols. They show that a budget-sensitive choice is possible in the task. They do not form a matched comparison of hosted and open models.

The paper also changes the task value boundary by varying priors and selector reliability. At reliability 0.74, four profiles put the selector advantage at different values. Two favour a branch and two favour the selector. GPT-4.1 chooses the selector across all 864 decisions in those profiles. Low-effort GPT-5 mini makes many optimal choices overall, yet keeps selecting the same role across most families in each profile. A predictor based on reliability alone fits its choices better than a predictor based on signed action-value advantage in the reported held-out families.

This result is about the tested policy behaviour and the formal task model. The authors do not infer that a model has no concept of budget or document structure. A model may follow a simple selector-first heuristic that performs well under many ordinary conditions. The paired cases are selected to distinguish that heuristic from a policy sensitive to remaining reads and action value. That distinction matters when the deployment environment imposes a strict crop budget.

Loss grows after the first read#

The authors extend the controlled task into a hierarchy with eight regions. A root routes to one of two gates, each gate leads to one of four answer leaves, and one region is a distractor. The agent has three reads. This creates a longer sequence in which a good first crop must be followed by useful decisions based on each return.

Across 160 episodes per checkpoint, the first-read loss is small, between 0.18 and 0.96 percentage points. Combined second- and third-read loss ranges from 22.20 to 26.76 points. Terminal decision loss adds 5.73 to 14.30 points. Realised accuracy ranges from 53.75 to 57.50 percent. Most of the measured loss occurs after the initial observation, where the agent must use new information to select its next crop.

The pattern persists in the balanced, noiseless profile. First-read loss remains between zero and 3.12 points. Later observations lose 29.38 to 36.88 points. In a control where all legal first actions have equal value, later loss remains measurable. These results point toward the continuation policy as the largest issue in the tested hierarchies. A one-step decision can look strong while the complete rollout performs far below its exact reference.

The evaluator’s exact values depend on the task model. In controlled settings, region bindings, priors and read channels are specified, and the evaluator can enumerate possible states. In open scenes, the prior and observation model would need to be learned or approximated. Any error in their values would affect the attribution of loss. The authors identify this as a limitation. The accounting is exact relative to the declared model, while the declared model may be a simplification of a deployed document workflow.

The task’s reference can also define what counts as useful. If the selector’s label is noisy, or region bindings are uncertain, a theoretically optimal route under the supplied map may not be optimal in a real scanning system. The paper isolates selection by using verified bindings for controlled tasks. That makes policy comparisons cleaner. A production evaluation should also test localization and read quality, because selecting the right role does not help if the crop misses its text.

Real-form results change with the answer key#

The natural-document panel uses 30 pages from FUNSD. Each page has a low-resolution overview and six high-resolution crops in a fixed 3-by-2 layout. The study compares choosing one crop against a shared uniform reader. It also compares a two-crop sequential policy with native whole-image reading, which processes the page in one call. References are defined in two ways: the original labels and a curated version used by the authors.

Under curated scoring, Qwen3-VL-8B’s one-crop selection reaches 24 of 30 exact-match pages, a 24.44-point gain over the uniform reader. Against the original reference text, the same saved selections gain 10.56 points, with an interval that crosses zero. The reference wording changes whether the extracted answer is judged correct. For example, an original form reference merges two adjacent fields, and removing the neighbouring-field text from the curated reference changes one saved answer from incorrect to correct.

The full two-crop pipeline does not outperform native image reading in this panel. Qwen3-VL-8B answers 21 of 30 pages with two crops versus 25 of 30 with the whole image. The two-crop route uses three calls and about 1,662.8 input tokens per page. Native reading uses one call and about 824.8 tokens. InternVL3.5-8B also answers fewer pages with two crops than with the native whole-image path. The crop policy can spend additional observations without increasing task accuracy.

Scoring definitions add uncertainty to the apparent selection gain. On the same saved outputs, the paper crosses original versus revised reference text with two normalization rules. For Qwen3-VL-8B, revising the reference text alone increases the one-crop gain by 12.22 points. Changing punctuation and spacing normalization alone adds 10.00 points. Their interaction reduces the combined change by 8.33 points. The outcome underscores the need to publish reference text and normalization rules when answer matching is exact.

These form results are small and reference-sensitive. The intervals for several comparisons cross zero. The authors do not treat them as a broad proof that crop selection improves document understanding. The reference curation was AI-assisted and did not receive independent human adjudication, and public FUNSD pages may have appeared in model pretraining. The result is useful as an audit of the evaluation setup as much as a comparison of observation policies.

What the protocol can and cannot support#

Budget-paired audits can distinguish two policies that choose the same crop under one budget but should diverge when the remaining reads change. Exact continuation values can identify the cost of a selected action in controlled tasks. Complete-trajectory accounting can locate whether expected accuracy was lost during evidence acquisition or in the final answer. The approach gives researchers a vocabulary for evaluating sequential observation, rather than reporting only terminal accuracy.

The method requires a declared prior over the hidden scene, a model of how reads return evidence and verified bindings between task roles and image regions for exact value calculation. The controlled experiments are designed around those assumptions. The natural-form section uses empirical scoring instead of claiming an exact posterior model. Open documents introduce uncertain localization, OCR noise and correlations among fields. Extending exact value accounting to those settings would require an observation model whose estimation error can be measured.

Model coverage is also limited. Five open-weight models share Qwen-derived language components. They answer structured text states under constrained decoding without an explicit reasoning phase. The two hosted models use dated endpoints and their own protocols, so they establish that the desired behaviour can occur rather than provide a controlled head-to-head result. Other model families, free-form reasoning and learned visual observation channels remain to be tested.

The main evidence supports a specific claim. In the reported controlled tasks, open models often keep reading a fixed selector even when a paired budget makes a branch more valuable. In longer hierarchies, much of the exact-model loss occurs after the first read. In scanned forms, gains depend on how references are curated, and crop systems need comparison with whole-image accuracy and token cost. These observations are bounded by the task construction and model set.

The controlled panels also separate a policy from the observation interface. The main routing audit provides a map of region roles and structured region returns. That makes it easier to test the decision rule after evidence arrives, but it leaves visual localization and crop quality outside the central counterfactual. The natural-form panel brings images and crops back into the comparison, with different sources of uncertainty. A model can choose the valuable field yet miss its text in the crop. It can read the field correctly and then reason over an ambiguous label. Separate controls are needed to attribute each failure.

The paper’s exact reference counts continuation values from a declared probability model. For a given history, it estimates the best achievable terminal accuracy when a particular number of reads remains. It then compares an agent’s chosen region with that action value. This is a useful accounting device in synthetic tasks, because every possible state and return can be enumerated. For natural forms, the hidden state and observation channel are not known exactly. The researchers therefore use empirical answer accuracy and paired document comparisons, rather than claiming the same exact value guarantee for those pages.

This difference between the two panels matters for interpreting the contribution. Controlled tasks give a clean causal contrast because public evidence and bindings remain fixed while a budget or value parameter changes. Real scanned forms establish whether crop choices help on a small document panel. They do not establish an exact expected-value ranking for every crop, because the reference labels and visual read model carry their own uncertainty. The work presents both types of evidence, and each answers a different question.

A deployment decision needs complete-path evidence#

For a deployed document agent, every crop has a marginal cost. It can add tokens, a model call, latency and the possibility of introducing a misread. An evaluation that reports accuracy after an unlimited number of reads will not show whether the policy remains useful under a fixed budget. A test at one budget will not reveal whether the agent changes its strategy when fewer observations remain. The paired protocol supplies a way to expose that policy sensitivity under controlled conditions.

The complete-path audit asks the more operational question. Starting with the overview and a fixed crop budget, what happens at every choice, after each returned crop and at final answer time? Stepwise acquisition loss can point to a region choice that consumed a read without enough expected benefit. Terminal decision loss can reveal a problem with combining the evidence the system collected. Reporting them alongside final accuracy makes a low score more actionable and helps compare alternate interfaces, crop policies or state presentations.

There are costs to making this kind of test credible. The evaluator needs to define which regions are legal, how returns are generated or read, which answer rule applies, and what counts as remaining budget. For a natural form, references need review, image preprocessing must be recorded, and the whole-image baseline must use comparable model conditions. A crop pipeline should report its extra calls and input area, not just its accuracy. These requirements add work to an evaluation but make the conclusion easier to audit.

The studied open models use constrained output formats and do not have an explicit reasoning phase. In a production interface, models may have different image encoders, tool policies, caching, region proposal systems or hidden reasoning settings. Those differences can change observation costs and the value of returning a crop. A new deployment should evaluate its own complete configuration. The current comparison is a diagnostic of specified policies and panels, not a universal ranking of visual agents.

For product teams, the lesson is to evaluate the full observation policy. A visual agent should be tested on which region it reads, what it does with the returned evidence, how it changes strategy as the remaining budget shrinks and whether the final answer is reliable. Reference wording, crop definitions, whole-image baselines and input cost should be visible. A top-line answer score cannot reveal the route that produced it.

The paper provides a detailed protocol for diagnosing that route in controlled settings. Its most surprising result is not a higher accuracy claim, but a gap between plausible first actions and effective sequential policies. The first crop may be sensible. Later choices can ignore the information already obtained and consume the remaining budget. Measuring each step gives evaluators a way to find that loss before deploying a visual agent into document work where every extra read has a cost.