Benchmark Radar™
RSS Contact Star —

Daily AI benchmark brief: 2026-09-22

595evidence observations
12sources represented
20public-attention signals

Daily briefing

  1. The new Curation-Bench release evaluates whether coding agents can improve training-data selection through an iterative workflow: agents inspect data, implement selection policies, run a fixed training and evaluation pipeline, observe noisy results, and revise their policies while the model and training recipe remain fixed. This isolates the agent’s data-curation decisions from unrelated model changes. [E003] Why it matters: If you are deciding whether an agent can automate data preparation, this benchmark tests the operational loop rather than asking for a one-shot policy recommendation. It can distinguish agents that merely propose plausible rules from those that improve policies using measured feedback. [E003] Evidence: E003. High confidence.
  2. The new XYEval release converts existing benchmarks into tests of whether agents resist the “XY problem,” where a user proposes a plausible but misguided solution instead of explaining the underlying need. It evaluates five models across six suites and reports performance drops after this mutation. [E016] Why it matters: If you evaluate assistants that advise users, ordinary task accuracy may miss harmful agreement with a faulty premise. XYEval provides a way to reuse an existing suite while adding a communication stress test, making resistance to misleading advice a separate selection criterion. [E016] Evidence: E016. High confidence.
  3. The new VibeMemBench release tests persistent memory—the ability to reuse prior experience—through executable work on 111 coding targets from 90 software repositories, backed by 3,634 historical trajectories. It covers bug fixes, feature requests, interface changes, and configuration work rather than scoring memory through recall questions alone. [E018] Why it matters: If you are choosing a memory system for a coding agent, this design measures whether stored experience improves completed repository changes. That separates useful memory from systems that retrieve relevant-looking history without improving the resulting code. [E018] Evidence: E018. High confidence.
  4. The new ChartJudgeBench release evaluates vision-language judges—models that inspect both images and text—used to score chart-to-code systems that recreate charts as programs. Its diagnostics include 1,003 pairwise visual-comparison cases and 650 reasoning-judgment cases. [E014] Why it matters: If model-generated judge scores drive regression tests or reward-based training, the judge itself becomes part of the measurement system. ChartJudgeBench lets teams test visual comparison and reasoning separately before treating those scores as reliable optimization targets. [E014] Evidence: E014. High confidence.
  5. The new BabelArena release extends agent evaluation beyond English with 16,146 multilingual instances derived from 702 canonical tasks. Its BabelFlow process analyzes runtime dependencies, preserves task structure during translation, and applies automated verification plus human review to retain task and scoring meaning. [E019] Why it matters: If you are selecting an agent for multilingual workflows, translated prompts alone may alter tools, dependencies, or expected outputs. BabelArena offers aligned tasks intended to support language-by-language comparisons without silently changing what the agent must accomplish. [E019] Evidence: E019. Medium confidence.
  6. The Full-text extraction pipeline for atomic layer deposition process parameters received an update that preserves its earlier frozen scorer while adding a corrected scorer for three identified defects, including comparisons of temperatures reported in kelvin without conversion. [E108] Why it matters: If you reproduce or compare results from this extraction pipeline, the scorer version is now an evaluation variable. Keeping both versions allows teams to audit earlier numbers while using corrected measurement logic for new runs instead of mixing incompatible results. [E108] Evidence: E108. High confidence.

What arrived

What benchmarks, datasets, or evaluation methods did the radar first see today?

high confidence

Within this captured feed, first observations included AgentVidBench for multi-step video questions, Curation-Bench for automated data curation, ChartJudgeBench for chart-code judges, XYEval for misleading user advice, VibeMemBench for coding-agent memory, BabelArena for multilingual agents, and mini-CommonForms for form-field detection.

Other notable arrivals covered surgical-video segmentation, medical imaging, camera-motion analysis, face recognition, game play, computer-aided design, video generation, retrieval, security alerts, and agent payment authorization. New evaluation approaches included transforming existing tests into misleading-advice scenarios, measuring checker quality through injected faults, and comparing evaluation protocols themselves.

Takeaway: The registry confirms a large set of artifacts first observed by the radar today, but these were releases, discoveries, and updates rather than one uniform class of new publication. Category labels overlap, and all findings apply only to this keyword-filtered feed.

Another reading: First observed by the radar does not mean newly created or released. BIRD-History arrived as an update, while APort Vault and CADWorld were discovery records, so collection timing is a strong competing explanation for their appearance today.

  • S023 artifacts first observed by the radar today: 426

Which of today's arrivals document how they score an answer?

medium confidenceNot enough evidence

The clearest documented cases are DISCERN, with deterministic grading outputs and adjudication records; DualLoop Evaluation, with gold answers generated by a pixel-based reference process; the Jev retrieval benchmark, with per-query artifacts and calibration tables; and Measuring the Checker, which scores test procedures by whether they detect injected faults.

Other auditable candidates include the Hemmingway quantization benchmark, which publishes blind-judge results, reliability information, and a record-selection policy, and APort Vault, which reports separate evaluation events instead of hiding them inside one combined result.

Takeaway: These arrivals expose more of the path from an output to a verdict than records that merely advertise a benchmark. The registry provides no computed count for this property, and the supplied summaries do not establish an exhaustive list across every arrival.

Another reading: A summary can mention grading artifacts without exposing the complete rubric, judge prompt, tie handling, or aggregation rule. ChartJudgeBench, for example, describes judgment tasks but its supplied summary does not show enough scoring detail to place it confidently on the confirmed list.

  • S023 artifacts first observed by the radar today: 426

What is still moving

Which artifacts the radar already tracked moved measurably, and over what span?

medium confidenceNot enough evidence

The movement registry identifies hf-benchmarks/transformers, IntelligenceLab/LHTB-leaderboard, RoboDojo-Benchmark/RoboDojo, vedangfake/chess-slm-benchmark, alexshpunt/explicit-edit-benchmark, vava22684/song-jury-leaderboard, career-ops-hq/career-ops, and AlphaDojo/dojo_benchmark_kline. Each statistic covers its stated full tracked span.

Downloads increased for most listed datasets and decreased for the LHTB and song-jury leaderboards. The career-ops repository gained stars. These are cumulative changes between each artifact’s first and latest tracked observations, not one-day changes.

Takeaway: These entries show the largest registered movements in this captured, keyword-filtered feed. No certified comparison window exists, so they should not be described as broader trends. Artifact-level E citations were not supplied, preventing a fully evidenced artifact report.

Another reading: The tracked-artifact packet contains additional nonzero changes without corresponding movement statistics, so the registered list is not exhaustive. Missing artifact-level E identifiers also prevents independent citation of the named records.

  • S026 downloads change for hf-benchmarks/transformers: 22,812.0 downloads
  • S027 downloads change for IntelligenceLab/LHTB-leaderboard: -15,587.0 downloads
  • S028 downloads change for RoboDojo-Benchmark/RoboDojo: 11,758.0 downloads
  • S029 downloads change for vedangfake/chess-slm-benchmark: 7,929.0 downloads
  • S030 downloads change for alexshpunt/explicit-edit-benchmark: 7,668.0 downloads
  • S031 downloads change for vava22684/song-jury-leaderboard: -3,936.0 downloads
  • S032 stars change for career-ops-hq/career-ops: 2,716.0 stars
  • S033 downloads change for AlphaDojo/dojo_benchmark_kline: 1,984.0 downloads

Which of that movement is corroborated by more than one data source?

high confidence

None of the registered artifact-level movement is corroborated by more than one data source. Every listed download or star change comes from a single platform.

Another source did not independently report the same metric change for any listed artifact. Seeing an artifact in multiple places is not enough; both sources would need to measure the same movement.

Takeaway: Treat all registered movement as single-source measurement within this captured feed. It may be useful as a platform-specific signal, but it is not cross-source confirmation.

Another reading: Some tracked artifacts were independently sighted by more than one source, but the registry explicitly distinguishes that from corroboration of a particular metric. Thus, multi-source visibility does not contradict the absence of corroborated movement.

  • S025 tracked artifacts today seen by more than one data source: 5
  • S026 downloads change for hf-benchmarks/transformers: 22,812.0 downloads
  • S027 downloads change for IntelligenceLab/LHTB-leaderboard: -15,587.0 downloads
  • S028 downloads change for RoboDojo-Benchmark/RoboDojo: 11,758.0 downloads
  • S029 downloads change for vedangfake/chess-slm-benchmark: 7,929.0 downloads
  • S030 downloads change for alexshpunt/explicit-edit-benchmark: 7,668.0 downloads
  • S031 downloads change for vava22684/song-jury-leaderboard: -3,936.0 downloads
  • S032 stars change for career-ops-hq/career-ops: 2,716.0 stars
  • S033 downloads change for AlphaDojo/dojo_benchmark_kline: 1,984.0 downloads

What it means

What should someone building or evaluating AI systems do differently today?

medium confidence

Within this captured feed, benchmark and evaluation records are prominent, with several new releases targeting practical gaps in agent testing rather than establishing broad performance gains.

Add tests for misleading user advice, persistent-memory coding work, multilingual workflows, and automated judge reliability. Newly released XYEval, VibeMemBench, BabelArena, and ChartJudgeBench each address one of these gaps, but their appearance here does not validate them independently.

Takeaway: Use the relevant releases as candidates for an internal pilot, reproduce their setup, and compare results with existing task-specific tests before changing deployment or procurement decisions.

Another reading: These are newly released research artifacts represented mainly by author summaries. They may not match a particular production environment, and the feed provides no independent replication showing that adopting them improves real-world decisions.

  • S018 records tagged benchmark: 378 count (multi-label)
  • S019 records tagged evaluation: 316 count (multi-label)
  • S021 records tagged agentic: 81 count (multi-label)

What does today's evidence fail to show, and what would change the reading?

high confidence

The feed does not establish a field-wide trend, broad adoption, benchmark quality, or reliable momentum. Today lacks a certified comparison window, and tracked metric movements are single-source rather than corroborated.

Differences from earlier captures could reflect collection changes. Download or star movement spans each artifact’s full tracked period, not one day, and cannot by itself show use, quality, or improvement.

Takeaway: The reading would strengthen with a certified comparable window, stable connector coverage, independent sources confirming the same metric movement, and external replications or audits of the highlighted evaluations.

Another reading: The breadth of captured benchmark and evaluation records may still help discover candidates, but discovery volume is not evidence that the field changed or that any artifact is decision-ready.

  • S025 tracked artifacts today seen by more than one data source: 5
  • S026 downloads change for hf-benchmarks/transformers: 22,812.0 downloads
  • S027 downloads change for IntelligenceLab/LHTB-leaderboard: -15,587.0 downloads
  • S028 downloads change for RoboDojo-Benchmark/RoboDojo: 11,758.0 downloads
  • S029 downloads change for vedangfake/chess-slm-benchmark: 7,929.0 downloads
  • S030 downloads change for alexshpunt/explicit-edit-benchmark: 7,668.0 downloads
  • S031 downloads change for vava22684/song-jury-leaderboard: -3,936.0 downloads
  • S032 stars change for career-ops-hq/career-ops: 2,716.0 stars
  • S033 downloads change for AlphaDojo/dojo_benchmark_kline: 1,984.0 downloads

Evidence sources

Where this came from, and what it does not cover

Built from the validated daily snapshot and its 595 evidence records.

Briefing model: gpt-5.6-sol.

It read 155 of 595 records.

This keyword-filtered feed is not representative of the AI field. The briefing received 155 selected evidence records from a 595-record corpus, so it did not inspect the full captured set. Brave and OpenReview were unavailable, and there is insufficient comparable history to infer category-share movement. Descriptions of new artifacts come from their authors and were not independently validated.