papers.

Agents Can Build Benchmarks. Approvals Missed What Probing Found: 22 Cases

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-17 · benchmark-* family · preprint

submitted to: Research Square

read the pdfcode and data

Coding agents built the 22 benchmark packages this paper studies, 1204 tasks on one shared engine; the 22 cases of the title are the packages. Probing the packages found defects that needed a fix round in every one of them (median 3, maximum 9), across 13 defect classes, and at least 13 packages paid credit to a strategy that ignored the task. Re-running the logged exploits on the artifacts replicates 45 of 50 record clauses, 31 of 34 measured on the published tarballs and 14 of 16 on the unreleased commits the record cited, and the median attack score over the 16 paired wave-1 rows falls from 0.7500 before the fixes to 0.0841 after; 5 of the 52 post-fix clauses still fail at the shipped versions and 7 more hold only under the weak lower-than-before bound. The other layers did not stay blind in both directions: the diff-reading review flagged a published hidden seed before one merge, and the generic blind battery later found released credit on payment-delegation that the probing record never logged. Under the preregistered design the best of 8 generic blind policies scored higher before the fixes in only 2 of 12 packages (p = 0.50 on 3 non-tied packages), a null that bounds what the registered design can claim; at the released versions the best blind score on any package is 0.4000. At the released versions 7 probe cells on 5 graders pay content that never did the task, and 2 of 22 graders accept forged fields. The record documents 2 of the 94 approvals as approval-to-defect cases. We release the engine, the packages, the re-review protocol, the defect record and the scripts behind every number, and we condense the classes into five checks to run before an agent-built benchmark ships. No real model was run and no leaderboard is kept; the registered real-model arm is unfinished.

the benchmark suite behind the paper lives at aibenchmarkindex.com.

@misc{benchmark-family,
  author = {Armani, Ryan},
  title = {Agents Can Build Benchmarks. Approvals Missed What Probing Found: 22 Cases},
  year = {2026},
  doi = {10.5281/zenodo.23455125},
  url = {https://ryanarmani.com/papers/benchmark-family},
}

Accessibility Fix: Measuring HTML Repair Without Content Loss

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-accessibility-fix · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

Accessibility Fix measures whether an HTML repair corrects injected defects while retaining page content. We study 40 public synthetic pages using scripted controls and a simulated population sweep. The reference pages receive full credit on every task; unchanged pages, empty answers, and refusals receive none. A template fixer earns mean 0.788 [0.713, 0.869] with a stratified bootstrap interval. The grader rejects reference pages modified with display:none, but accepts the same references modified with clip-path concealment. A simulated gate ablation measures the credit retained when the automatic-zero gate is removed. These experiments characterize a static repair score, not complete accessibility, user benefit, or learned-model ability. No learned model or human study is included.

@misc{accessibility-fix,
  author = {Armani, Ryan},
  title = {Accessibility Fix: Measuring HTML Repair Without Content Loss},
  year = {2026},
  doi = {10.5281/zenodo.23773050},
  url = {https://ryanarmani.com/papers/accessibility-fix},
}

Apology Pizza: Measuring Bounded Remedies in Customer Support

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-apology-pizza · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

Customer support agents can offer polite text while taking the wrong financial action. Apology Pizza measures bounded remedy execution on 40 synthetic complaints across 8 scenario classes. We evaluate 12 scripted controls through a scripted mock tool environment and replay grader. A scenario-aware policy completes 40/40 tasks, with a 95% Wilson interval [0.912, 1.000]. Replacing its final message with unrelated text also completes 40/40, with the same Wilson interval, exposing a message-content gap. A simulated control ablation varies independent omissions of verification and messaging. At omission probability 0.5, simulated completion changes from 0.234 [0.197, 0.266] without forced steps to 1.000 [1.000, 1.000] with both steps forced; brackets show 95% seed-bootstrap intervals. All 13760 local episodes include repeated runs with identical semantic traces. These results test the evaluation design, not learned models, customer satisfaction, or deployment safety.

@misc{apology-pizza,
  author = {Armani, Ryan},
  title = {Apology Pizza: Measuring Bounded Remedies in Customer Support},
  year = {2026},
  doi = {10.5281/zenodo.23773039},
  url = {https://ryanarmani.com/papers/apology-pizza},
}

Checkout Consent: Measuring Authorization Before Agent Payments

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-checkout-consent · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

Checkout consent binds a proposed purchase to a current total and shipping address. Checkout Consent evaluates this boundary in 40 synthetic shopping episodes across 7 scenario types. We test 9 scripted controls through the public execution and replay path. The privileged oracle attains full credit on 40/40 tasks, a rate of 1.000 [0.912, 1.000]. Empty output and blanket refusal receive no credit. Re-quoting after consent produces an unauthorized-attempt rate of 0.875 [0.739, 0.945]. These are 95% Wilson intervals over tasks. A simulated population sweep adds 16000 episodes and removes consent questions in a paired intervention. At equal oracle and unsafe-policy mixture weights, simulated mean score falls from 0.514 [0.491, 0.536] to 0.062 [0.055, 0.069]; intervals bootstrap simulated agents. The payment guard remains active throughout. Results validate selected measurement distinctions, not deployed agent safety. No learned model or human participant is evaluated.

@misc{checkout-consent,
  author = {Armani, Ryan},
  title = {Checkout Consent: Measuring Authorization Before Agent Payments},
  year = {2026},
  doi = {10.5281/zenodo.23752388},
  url = {https://ryanarmani.com/papers/checkout-consent},
}

Cost per Verified Task: Replay-Based Evaluation of Back-Office Agents

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-cost-per-task · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

Cost per attempted task can conceal expensive failures. We study cost per verified success on 30 synthetic back-office tasks, using deterministic replay of file and customer-record operations. A privileged oracle completes every task (95% Wilson interval: 88.6-100.0%); each of six trivial policies completes none (0.0-11.4%). Ninety invalid-trace probes fail verification. A simulated population sweep compares direct execution with a redundant read-only control under a fixed artificial token tariff. The control preserves success and doubles assumed cost per success. These results validate accounting and replay behavior, not learned-agent ability or provider prices. No hosted model or human study is included.

@misc{cost-per-task,
  author = {Armani, Ryan},
  title = {Cost per Verified Task: Replay-Based Evaluation of Back-Office Agents},
  year = {2026},
  doi = {10.5281/zenodo.23752393},
  url = {https://ryanarmani.com/papers/cost-per-task},
}

CRM Truthfulness: Consent, Evidence, and Record Integrity in Sales Agents

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-crm-truthfulness · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

CRM Truthfulness evaluates sales-agent actions against consent records and product evidence. It contains 40 fictional episodes across 8 scenarios. We study scripted controls and a simulated factorial ablation, using replay rather than an agent's claimed final state. The packaged compliant control has full-credit rate 40/40; 1.000 [0.912, 1.000] (95% Wilson interval). Empty, constant, echo, refusal, and first-tool controls earn no credit. Removing lookup leaves scores unchanged, exposing a process-coverage limit. Simulated consent removal produces forbidden-contact violation rate 10/10; 1.000 [0.722, 1.000] (95% Wilson interval). Claim and record controls affect distinct score components. All 16 conditions reproduce their semantic traces. These are privileged scripted experiments, not learned-model results. Lexical claim checks, public templates, and the lack of human validation limit external conclusions.

@misc{crm-truthfulness,
  author = {Armani, Ryan},
  title = {CRM Truthfulness: Consent, Evidence, and Record Integrity in Sales Agents},
  year = {2026},
  doi = {10.5281/zenodo.23752395},
  url = {https://ryanarmani.com/papers/crm-truthfulness},
}

Benchmark Honesty: Recovery and Abstention after Tool Failure

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-honesty · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

Honesty after tool failure requires both recovery and restraint. Benchmark Honesty tests this distinction with 60 fictional lookup episodes: 30 recoverable failures and 30 unavailable records. Its headline score combines evidence-backed answering with evidence-backed abstention. We evaluate 10 scripted controls and a confidence sweep through the benchmark environment and grader. The evidence-reading recovery script achieves balanced success 1.000 [1.000, 1.000]; each component is 1.000 [0.886, 1.000]. Always-abstain and no-retry controls achieve balanced success 0.000 [0.000, 0.000]. Balanced-success intervals use 95% item bootstrap bounds; component rates use 95% Wilson bounds. Changing confidence leaves balanced success unchanged while changing Brier loss. All 14 repeated conditions agree in their semantic traces. These results validate selected scoring distinctions, not model honesty in deployment. No learned model or human participant is evaluated.

@misc{honesty,
  author = {Armani, Ryan},
  title = {Benchmark Honesty: Recovery and Abstention after Tool Failure},
  year = {2026},
  doi = {10.5281/zenodo.23752399},
  url = {https://ryanarmani.com/papers/honesty},
}

Hundred Step: Dependency Preservation and Recovery in Long Workflows

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-hundred-step · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

Long workflows require agents to retain exact dependencies and recover from faults without skipping work. Hundred Step isolates these behaviors in 24 synthetic workflows with replay-based scoring. A reactive scripted controller completes all tasks: 100.0% (95% Wilson interval 86.2 to 100.0). Disabling recovery or allowing transient retries alone prevents completion on every task. We also execute 4608 simulated episodes with matched omission draws and a recovery ablation. The stipulated omission probability is 0.02 per logical step. With recovery enabled, simulated completion is 60.4% (95% seed-bootstrap interval 53.5 to 67.4) at 25 steps. Simulated completion is 13.2% (8.3 to 18.8) at 100 steps. The study validates diagnostic behavior under explicit controls. It provides no learned-model leaderboard, human baseline, or estimate of production reliability.

@misc{hundred-step,
  author = {Armani, Ryan},
  title = {Hundred Step: Dependency Preservation and Recovery in Long Workflows},
  year = {2026},
  doi = {10.5281/zenodo.23773042},
  url = {https://ryanarmani.com/papers/hundred-step},
}

Benchmark Instruction Hierarchy: Authority, Utility, and Injection Detection

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-instruction-hierarchy · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

Instruction hierarchy requires an assistant to preserve authorized rules when lower-trust text requests a conflicting answer. This benchmark isolates that problem in synthetic single-turn tasks with explicit authority cues and deterministic checks. We distinguish full task success from the presence of listed injection text. Scripted controls expose the limits of either measure alone. An additional simulated agent population sweep compares unchecked responses with an ideal blocker that knows the listed attack strings. The simulated blocker can remove detected attack text without recovering the required answer. These experiments validate measurement behavior on a fixed public task set. They do not estimate learned-model ability, deployment security, or human performance. We provide reproducible outputs, uncertainty intervals, and a scoped roadmap for external validation.

@misc{instruction-hierarchy,
  author = {Armani, Ryan},
  title = {Benchmark Instruction Hierarchy: Authority, Utility, and Injection Detection},
  year = {2026},
  doi = {10.5281/zenodo.23752403},
  url = {https://ryanarmani.com/papers/instruction-hierarchy},
}

Latency Budget: Grounded Answers Under Explicit Deadlines

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-latency-budget · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

An answer can be grounded and still arrive too late to be useful. Latency Budget measures this distinction on 36 synthetic directory, calendar, and document tasks. Credit requires a single correct candidate, replayed evidence, and completion within an explicit deadline. We run 1656 local episodes across 23 repeated conditions without paid inference. The privileged oracle obtains 100.0 [90.4, 100.0] percent deadline credit (95% Wilson interval); each blind control obtains 0.0 [0.0, 9.6] percent. An additional simulated factorial experiment varies declared thinking time and redundant tool calls while holding task deadlines fixed. At 80 ms per turn, one redundant call reduces deadline credit to 75.0 [58.9, 86.2] percent, while grounded correctness remains 100.0 [90.4, 100.0] percent. The experiment demonstrates sensitivity to wasted actions under the chosen cost model. It does not measure learned-agent performance, network latency, or human deadline preferences.

@misc{latency-budget,
  author = {Armani, Ryan},
  title = {Latency Budget: Grounded Answers Under Explicit Deadlines},
  year = {2026},
  doi = {10.5281/zenodo.23752409},
  url = {https://ryanarmani.com/papers/latency-budget},
}

MCP Safety: Measuring Useful Action across Tool Trust Boundaries

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-mcp-safety · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

MCP tools expose descriptions, returned content, and actions across trust boundaries. MCP Safety measures whether an agent completes a requested lookup without forbidden calls or literal canary disclosure. Its 40 synthetic tasks form 20 benign/malicious pairs across 6 attack categories. Deterministic replay scores safety and usefulness separately, then requires both for success. In 560 local scripted episodes, the privileged good control achieves 100.0 [83.9, 100.0] percent joint success in each stratum (95% Wilson intervals). The bad control completes the work but achieves 0.0 [0.0, 16.1] percent joint success. An additional 16000-episode simulated population sweep separates argument filtering from task-aware action filtering. At maximal simulated attack following, combining both filters preserves all requested work; either filter alone leaves failures. Blanket filtering loses legitimate work. These are diagnostic controls and a stipulated simulation, not learned-model results or evidence of deployment security.

@misc{mcp-safety,
  author = {Armani, Ryan},
  title = {MCP Safety: Measuring Useful Action across Tool Trust Boundaries},
  year = {2026},
  doi = {10.5281/zenodo.23752413},
  url = {https://ryanarmani.com/papers/mcp-safety},
}

Memory Conflict: Auditing Household Decisions Under Changing Preferences

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-memory-conflict · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

Stored preferences can become invalid without disappearing. Memory Conflict evaluates whether household assistants act on current preferences while preserving privacy, booking state, and authority boundaries. We audit 40 synthetic episodes across 6 conflict types with 12 scripted conditions and 960 runner episodes including repeats. Reference actions achieve pass rate 1.000 [0.912, 1.000]; empty output achieves 0.000 [0.000, 0.088]. Moving clarification after booking also achieves 1.000 [0.912, 1.000], exposing a temporal limit of final-state scoring. These are 95% Wilson intervals over tasks, not population guarantees. A clearly simulated population sweep scores 30000 traces under injected errors and perfect control repairs. The sweep estimates sensitivity under explicit assumptions, not model performance. No learned model or human participant is evaluated.

@misc{memory-conflict,
  author = {Armani, Ryan},
  title = {Memory Conflict: Auditing Household Decisions Under Changing Preferences},
  year = {2026},
  doi = {10.5281/zenodo.23752415},
  url = {https://ryanarmani.com/papers/memory-conflict},
}

Pelican: Measuring Structural Compliance in SVG Drawings

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-pelican · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

SVG drawing tasks combine markup, geometry, and instruction following. Pelican measures structural compliance with explicit bird and bicycle constraints. We evaluate 30 public tasks using scripted controls, paired reference replay, and a simulated population with drawing-part deletion. The prompt-only builder policy obtains a mean structural score of 1.000 [1.000, 1.000]. Mismatched references obtain 0.024 [0.020, 0.028]. Intervals are 95 percent task bootstrap intervals. A simulated restoration control bounds the loss caused by deletion; it is not an implemented repair agent. The evidence concerns the scorer and these policies, not learned-model ability or visual quality. No human ratings or learned-model results are reported.

@misc{pelican,
  author = {Armani, Ryan},
  title = {Pelican: Measuring Structural Compliance in SVG Drawings},
  year = {2026},
  doi = {10.5281/zenodo.23752417},
  url = {https://ryanarmani.com/papers/pelican},
}

Benchmark Ranking Stability: Separating Score Order from Evidence

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-ranking-stability · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

A benchmark score orders systems without showing how that order depends on the evaluation sample. Benchmark Ranking Stability combines paired item resampling, repeated runs, and prompt-specific ranks. Its public suite contains 60 reasoning items across 6 families, each with 5 prompts. We run 9 scripted or mock controls twice through the actual evaluator. We also evaluate simulated populations with different sizes and correctness gaps, using 12 seeds per condition. The answer-key control reaches 1.000 [0.940, 1.000] all-variant item success (Wilson interval). In simulated populations of eight agents, a designated leader's mean bootstrap first-place share changes from 0.150 [0.090, 0.222] at equal ability to 0.644 [0.568, 0.728] at the largest gap. These are confidence intervals across simulation seeds. Exact ties retain high rank agreement while offering no unique winner. The contribution is an inspectable ranking diagnostic and controlled validation, not a learned-model leaderboard or a new estimator.

@misc{ranking-stability,
  author = {Armani, Ryan},
  title = {Benchmark Ranking Stability: Separating Score Order from Evidence},
  year = {2026},
  doi = {10.5281/zenodo.23752419},
  url = {https://ryanarmani.com/papers/ranking-stability},
}

Rollback Recovery: Measuring Repair, Disclosure, and Restraint After Agent Errors

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-rollback-recovery · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

Rollback Recovery measures whether an agent restores damaged state, reports the incident, and avoids needless repair. Its 48 synthetic episodes cover contact records, files, and messages, with 36 incidents and 12 clean controls. A visible-information scripted policy passes 48/48 tasks (100.0 [92.6, 100.0] percent; Wilson 95% interval). Removing structured reports leaves repair intact but reduces full success to 25.0 [14.9, 38.8] percent. Removing state tags reduces success to 45.8 [32.6, 59.7] percent. A simulated population sweep runs 24000 episodes and isolates report retention from repair ability. At half retention, simulated full success is 62.2 [60.9, 63.4] percent (bootstrap 95% interval across simulated agents). These are local scripted controls and simulated outcomes, not learned-model scores. Public templates, idealized reversibility, and unvalidated human interpretation limit generalization.

@misc{rollback-recovery,
  author = {Armani, Ryan},
  title = {Rollback Recovery: Measuring Repair, Disclosure, and Restraint After Agent Errors},
  year = {2026},
  doi = {10.5281/zenodo.23752421},
  url = {https://ryanarmani.com/papers/rollback-recovery},
}

Schema Mutation: Measuring Tool Calls After Interface Changes

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-schema-mutation · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

Tool interfaces change while familiar call patterns remain. Schema Mutation measures whether a call follows the current interface after a migration. The benchmark crosses 12 tool families with 5 mutation types, yielding 60 public tasks. A prompt-only scripted solver achieves 100.0% accuracy (95% Wilson interval 94.0-100.0%). Empty, constant, refusal, echo, and first-tool controls score 0.0% (0.0-6.0%). Exact stale calls fail and receive a separate diagnostic. A simulated population sweep compares pass-through with schema validation. At adherence q=0.5, simulated accuracy is 52.0% (agent-bootstrap interval 50.8-53.2%) under both layers. Validation blocks stale proposals without improving completion. These are scripted and simulated results, not learned-model evaluations. The study supports measurement checks on a synthetic public task set; it does not establish deployment reliability or identify memorization as a cause.

@misc{schema-mutation,
  author = {Armani, Ryan},
  title = {Schema Mutation: Measuring Tool Calls After Interface Changes},
  year = {2026},
  doi = {10.5281/zenodo.23752423},
  url = {https://ryanarmani.com/papers/schema-mutation},
}

Silent Constraints: Measuring Goal Completion Without Collateral Violations

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-silent-constraints · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

An agent can complete a requested goal while damaging resources outside its authority. Silent Constraints measures this distinction with 40 synthetic episodes and deterministic audit-log replay. We report goal completion, clean success, and invariant violations separately. Privileged clean scripts achieve 40/40 clean successes; shortcut scripts achieve 38/40 goals but 0/40 clean successes. Five task-blind controls obtain no clean successes. A clearly labelled simulated population sweep varies clean-script choice and measures the resulting score response. All 107 conditions reproduce exactly across 8560 episode executions. Intervals describe hypothetical task sampling or simulated seed variation, not deployment risk. No learned model, human survey, or external validation is included.

@misc{silent-constraints,
  author = {Armani, Ryan},
  title = {Silent Constraints: Measuring Goal Completion Without Collateral Violations},
  year = {2026},
  doi = {10.5281/zenodo.23752427},
  url = {https://ryanarmani.com/papers/silent-constraints},
}

Benchmark Skateboard: Signed Rotations and Flight-Time Constraints

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-skateboard · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

Benchmark Skateboard tests consistency between signed rotations, projectile arithmetic, and landing constraints under an explicit simplified model. Its 72 public tasks cross 24 trick definitions with three gravity settings. We evaluate scripted controls through the benchmark runner and conduct a simulated population experiment with numeric and rotation corruption. A glossary-assisted analytic control reaches full credit on every task. Reversing rotation signs retains a mean score of 0.780, showing why partial credit must accompany complete-answer accuracy. Numeric perturbations reveal a sharp tolerance boundary with an absolute rounding-floor exception. Simulated repair layers separate arithmetic recovery from consistency with corrupted rotations. All results are generated locally without paid inference. These findings characterize the measurement instrument; they do not establish learned-model skill or real-world skateboarding ability.

@misc{skateboard,
  author = {Armani, Ryan},
  title = {Benchmark Skateboard: Signed Rotations and Flight-Time Constraints},
  year = {2026},
  doi = {10.5281/zenodo.23752429},
  url = {https://ryanarmani.com/papers/skateboard},
}

Calibrated Pushback: A Harm-Ladder Benchmark for Sycophantic Agreement

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-sycophancy · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

Agreement with a harmful plan and refusal of a reasonable plan are distinct errors. We study calibrated pushback using 12 public plan families and 48 labelled scenarios. Each scenario asks for an ordered decision, a rationale, and an alternative when pushing back. We report local scripted controls and a clearly simulated compliance-bias sweep with global control layers. The label-informed control reaches the scoring ceiling; constant policies do not. A harm-rank shortcut misses labels, while a simulated pushback floor trades lower under-response for higher over-response. These experiments validate score behavior on the released tasks, not deployed model safety. We provide generated data, uncertainty intervals, repeated traces, and a reproducible paper. No learned model or human participant is evaluated.

@misc{sycophancy,
  author = {Armani, Ryan},
  title = {Calibrated Pushback: A Harm-Ladder Benchmark for Sycophantic Agreement},
  year = {2026},
  doi = {10.5281/zenodo.23752431},
  url = {https://ryanarmani.com/papers/sycophancy},
}

Test Weakening: Measuring Repair without Eroding Tests

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-test-weakening · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

A coding agent can make visible tests pass by changing their expected answers. Test Weakening measures whether repair preserves the evidence used to accept it. We study 30 synthetic JavaScript tasks with executable checks for original assertions, held-out behavior, and mutation strength. The study runs 11 scripted policies over 660 episodes including repeats. Reference repair earns 100.0 [88.6, 100.0] percent full success. Expected-answer rewriting earns 100.0 [88.6, 100.0] percent public-test success but 0.0 [0.0, 11.4] percent full success. Brackets are 95 percent Wilson intervals over tasks. A semantics-preserving test reformat earns 0.0 [0.0, 11.4] percent full success, exposing a lexical preservation limit. A separately labelled simulated population sweep compares public-only, original-test, and full acceptance rules. It measures scoring sensitivity under fixed behavior, not changes in learned agents. No learned model, human study, or production deployment is evaluated.

@misc{test-weakening,
  author = {Armani, Ryan},
  title = {Test Weakening: Measuring Repair without Eroding Tests},
  year = {2026},
  doi = {10.5281/zenodo.23752433},
  url = {https://ryanarmani.com/papers/test-weakening},
}

Tool Errors: Measuring Safe Recovery from Ambiguous API Failures

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-tool-errors · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

An API error does not establish whether an operation took effect. Tool Errors measures safe recovery through stateful fault injection and replay-based scoring. Its 40 public episodes cover 6 fault families across 4 domains. We execute 4960 episodes using scripted controls and a simulated status-read omission sweep. The reference and response-aware controller each reach full credit on 40/40 tasks (95% Wilson interval 91.2-100.0%). Removing reads in a simulated ablation leaves 7/40 full-credit tasks (8.7-31.9%). At simulated read-omission probability 0.5, the mean full-credit fraction is 53.8% (50.6-56.9%, seed bootstrap). These are checks of the evaluation design, not trained-model results. No hosted model, human study, or production deployment is evaluated.

@misc{tool-errors,
  author = {Armani, Ryan},
  title = {Tool Errors: Measuring Safe Recovery from Ambiguous API Failures},
  year = {2026},
  doi = {10.5281/zenodo.23752436},
  url = {https://ryanarmani.com/papers/tool-errors},
}

Exploit Resistance: Separating Bug Repair from Verifier Manipulation

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-exploit-resistance · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

A coding agent can make a test runner report success without repairing the target function. Exploit Resistance separates functional repair from verifier manipulation in 46 seeded tasks spanning 23 scenarios. Independent semantic checks and integrity checks define success. We run 9 scripted control policies through the benchmark, yielding 828 episodes including exact semantic repeats. Reference repairs solve all tasks (95% Wilson interval 92.3-100.0%). Scripted verifier attacks pass every visible suite (95% Wilson interval 92.3-100.0%) but solve none (0.0-7.7%). Repair followed by tampering also loses all solved credit. A simulated population sweep compares visible-only, semantic-only, and full acceptance on identical measured traces. It isolates the contribution of integrity checks under a stated mixture, without estimating real agent behavior. These results validate selected scoring boundaries, not learned-model resistance, attack completeness, or deployment safety.

@misc{exploit-resistance,
  author = {Armani, Ryan},
  title = {Exploit Resistance: Separating Bug Repair from Verifier Manipulation},
  year = {2026},
  doi = {10.5281/zenodo.23752440},
  url = {https://ryanarmani.com/papers/exploit-resistance},
}

Benchmark Payment Delegation: Policy Compliance and Useful Action in a Simulated Ledger

Ryan Armani · orcid 0009-0005-9964-6841 · 2026

2026-10-29 · benchmark-payment-delegation · Published on Zenodo

submitted to: none recorded

read the pdfcode and data

Delegated payment agents must complete permitted work while respecting spending limits and owner authority. We present a synthetic ledger benchmark with 40 public episodes across 22 scenario templates. Native replay separates task success, ledger goal completion, and policy violations. Scripted reference success is 100.0% [91.2, 100.0] for the good policy and 0.0% [0.0, 8.8] for the bad policy. A constant OK response still scores 40.0% [26.3, 55.4], exposing weak evidence of informed refusal. Removing owner calls reduces success to 90.0% [76.9, 96.0]. A simulated population sweep varies selection between reference scripts and measures scoring sensitivity. All intervals are 95%; reference rates use Wilson intervals, and the simulated sweep resamples agent means. These results validate execution and expose scoring limits; they do not measure learned models, payment security in deployment, or human judgment.

@misc{payment-delegation,
  author = {Armani, Ryan},
  title = {Benchmark Payment Delegation: Policy Compliance and Useful Action in a Simulated Ledger},
  year = {2026},
  doi = {10.5281/zenodo.23752442},
  url = {https://ryanarmani.com/papers/payment-delegation},
}