How Far Are We From True Auto Research?

1. Introduction

Recent years have seen a wave of auto-research systems, including Auto Research [1], AI Scientist [2], and Analemma's FARS [3]. These projects have shown impressive progress in automated end-to-end scientific research. Rather than proposing another complex framework, we study a more direct setting: whether widely used general-purpose CLI agents (e.g., Claude Code, Codex, Kimi Code) can carry out research with only minimal guidance. Moreover, there is no comprehensive evaluation of the quality of agent-conducted research across diverse domains or under varying computational constraints (e.g., CPU-only vs. GPU environments), making it difficult to systematically assess the true capabilities and limitations of such agents in realistic research scenarios.

We evaluate three off-the-shelf CLI agents — Claude Code with Opus 4.6 [4, 5], Codex with GPT-5.4 [6, 7], and Kimi Code with K2.5 [8, 9] — using a minimal scaffold on 13 diverse CS domains (5 CPU-only + 8 GPU environments). We employ a peer-review protocol where three CLI agents evaluate each paper alongside its code, and additionally use the Stanford Agentic Reviewer [10] for external validation.

Key Findings

Each CLI agent develops a distinct research persona: Claude Code is the full-stack researcher — balancing both strategies (46% empirical, 33% novel methods) with the longest, most comprehensive papers. Codex is the empirical scientist — 87% empirical studies, highest integrity, but lower novelty. Kimi Code is the system builder — 79% of its papers are framed as novel methods with acronym-heavy names, yet it still scores low on novelty because many of its ideas sound more like repackaged existing work than genuinely new contributions.

Stanford Agentic Review (SAR) (0–10, ICLR scale):

• Among automatic systems: Claude Code (5.45) > FARS (5.06) > Codex (4.93) > Kimi Code (4.24). Our minimal scaffold with Claude Code (5.45) surpasses Analemma's FARS (5.06) using only a $200 Max subscription, compared to Analemma's $186,000 budget.
• Compared to 200 human-authored ICLR 2025 papers, Claude Code (5.45) scores above the weighted average of human-authored submissions based on the acceptance rate (5.42), sitting between accepted papers (5.59) and rejected papers (5.34).
• But human-annotated accept rates reveal a wide gap: ICLR Accepted 76.0%, Claude Code 41.0%, FARS 21.6%, Codex 12.8%, Kimi Code 5.1%.

Peer Review (PR) (0–10, ICLR scale):

• Claude Code (4.59) > Codex (4.51) > Kimi Code (3.38). Our PR is stricter than SAR because it gives the reviewer access to experiment code and execution logs, enabling systematic checks for fabricated or unsupported results. This may sometimes be overlooked even by the human expert.
• The major weakness for CLI agents are experimental rigor and significance. Agents perform relatively well on reference integrity (Codex: 8.41/10) and clarity (Codex: 7.47/10).
• Performance varies substantially across domains — Claude Code ranges from 5.33 (Probabilistic Methods) to 3.78 (Privacy in ML). Using stronger GPUs (RTX A6000→H100) shows no overall improvement, and in some cases even leads to lower scores, which further validates the major weaknesses.
• Manual integrity checks reveal sharp divergence between agents: Kimi Code has both results + setting mismatches in 77% of papers and fake references in 72%; Codex is the most trustworthy (5% / 8%); Claude Code sits in between (31% / 36%).

Human Inspection :

2. Framework Overview

Each CLI agent follows a standardized pipeline:

1. Ideation — Generate a research idea and experiment plan; self-review for up to 3 iterations.
2. Experiments — Write and execute code, collect results; self-review for up to 3 iterations.
3. Paper Writing — Produce a paper; self-review for up to 3 iterations.
4. Review — Evaluate via Stanford Agentic Reviewer and triple peer review (all three agents review each paper alongside its code).

3. Experiment Setup

CPU seeds: Causal Learning, Compiler Optimization, Data Integration & Cleaning, Operating System Design, Probabilistic Methods

GPU seeds: AI for Biology, Computer Vision, Datasets & Benchmarks, Generative Models, Interpretability, NLP, Privacy in ML, Supervised Representation Learning

Hardware: CPU-only or 1× NVIDIA RTX A6000 (48GB). Each agent gets 4 CPUs, 60GB RAM. We also conduct experiments for GPU experiments using NVIDIA H100 (80GB) GPUs.

Peer review (PR): Every paper is reviewed by all three agents (Claude Code, Codex, Kimi Code), scoring on 9 dimensions (novelty, soundness, significance, clarity, reproducibility, experimental rigor, references, reference integrity, results integrity) on a 0–10 ICLR scale.

Stanford Agentic review (SAR): Every paper is reviewed by Stanford Agentic Reviewer, which provides ICLR-calibrated scores on a 0–10 scale.

4. Main Results on Stanford Agentic Review

Automated Research Systems Comparison

We evaluate the performance of three CLI agents on the Stanford Agentic Reviewer (SAR). For comparison, we also collect scores from Analemma's FARS, which reports SAR scores for a total of 102 papers.

Figure 1: Score distributions of automated research systems on the SAR. Claude Code has the highest median and tightest distribution above 5.0.

Comparison with Human-Authored ICLR 2025 Papers

To calibrate these scores against human-authored research, we additionally sampled 200 real ICLR 2025 papers (100 accepted, 100 rejected) to the same Stanford Agentic Reviewer, establishing a human baseline.

SAR = Stanford Agentic Reviewer score. Human = average OpenReview score from ICLR 2025 human reviewers.

Score Heatmap (Seed × Trial)

Figure 2: SAR score heatmap across all seeds and trials for each agent.

Per-Domain Breakdown

We break down SAR scores across 13 research domains (5 CPU-only, 8 GPU) and compare CPU vs GPU performance for each agent.

Figure 3: SAR scores by research domain, ranked by average score across agents.

CPU vs GPU Performance (SAR)

Qualitative Review Analysis

We analyze the text of Stanford reviews to identify common weakness themes via keyword matching across 7 categories: Experimental Design (baselines, ablations, evaluation, statistical rigor), Overclaiming (unsupported or exaggerated claims), Scope & Scale (limited or toy experiments), Clarity & Presentation (unclear writing, confusing notation), Results Inconsistency (contradictions or mismatches within the paper), Reference Quality (fake citations, placeholder references, future dates), and Novelty (incremental or trivial contributions). We also measure the balance between strengths and weaknesses across agents.

Figure 4: Weakness themes in SAR (7 categories).

Human-Annotated SAR Decisions

We manually inspected all papers together with their SAR reviews and found that the overall assessment score is not a reliable indicator of SAR's final recommendation. For example, some papers with scores above 6 were not recommended for acceptance, while others with scores around 4.5 were recommended for acceptance. Due to this inconsistency, we manually assigned accept or reject labels based on SAR's final recommendation rather than its numeric score. In some cases, the review does not explicitly state an accept or reject decision; in those instances, we inferred the recommendation from the overall tone of the review, judging whether it was more "accepted" or more "rejected". Moreover, we treat conditional accept, accept with revision, and borderline (accept) as accept decisions; all others are treated as rejection.

5. Main Results on Peer Review

We design a peer review (PR) protocol where each of the three CLI agents reviews every paper alongside its experiment code, execution logs, and results artifacts. Unlike SAR, which evaluates only the PDF, this enables reviewers to verify whether reported results match actual experimental outputs, detect incomplete experiments, and identify fabricated claims. Additionally, reviewers are required to check reference validity via tools, flagging hallucinated or nonexistent citations.

Figure 5: PR score distributions. Claude Code and Codex cluster around 4–5, while Kimi Code has a wider spread with more low scores.

Score Heatmap (Seed × Trial)

Figure 6: PR score heatmap across all seeds and trials. Codex is the most consistent (std=0.32), while Kimi Code has the highest variance (std=0.80).

Per-Domain Breakdown

Figure 7: PR scores by research domain, ranked by average score across agents.

CPU vs GPU Performance

Per-Dimension Review Breakdown

Figure 8: Per-dimension PR scores. Codex leads on reproducibility (7.65), reference integrity (8.41), and results integrity (7.94). Claude Code leads on novelty (5.42) and significance (5.23). Kimi Code trails on all dimensions.

Integrity & Reference Analysis

We further manually validate every paper with its artifacts and PRs across four integrity categories:

• Results mismatch only: numbers reported in the paper don't match the actual results.json, logs, or experiment outputs.
• Setting mismatch only: the paper claims to do things the code doesn't actually do: claimed components or ablations are never implemented, hyperparameters in the text differ from the config, or the method is not implemented as described.
• Both: papers flagged for both results and setting mismatches.
• Fake reference: citations that don't exist, have fabricated authors, or have incorrect bibliographic metadata.

Programming Language Usage

We analyze the programming languages and file counts across all agent-generated codebases to understand how each agent structures its experiments.

Claude Code

Codex

Kimi Code

6. Research Style Analysis

Beyond scores, we compare how each CLI agent approaches research — what types of papers they write, how they structure arguments, and what their titles reveal about underlying research strategy. This comparative analysis shows that the three agents have developed fundamentally different research personas, which in turn explain many of the quality differences observed in Sections 4–5.

Research Type Distribution

We manually classify each paper based on its title and abstract into four categories: novel method (proposes a new named algorithm or system), benchmarking (creates evaluation tools), extension (improves existing methods), and empirical study (studies without proposing new methods).

Title Structure Analysis

Title Vocabulary

The most frequent words in each agent's titles reveal distinct research personas:

Title Patterns

Each agent has a distinct title style:

Paper Structure

How do the papers differ in structure and technical depth?

All 39 papers per agent analyzed (117 total).

7. Reviewer Analysis

Our pipeline uses two layers of review. First, each agent self-reviews its own work at every stage (ideation, experiments, paper writing) for up to 3 revision rounds. Second, all three agents cross-review every paper in a peer-review setup. Below we examine whether self-review revision actually improves quality, and how much bias the peer reviewers introduce.

Self-Review Revision Effectiveness

When an agent scores below the self-review threshold, it revises and re-evaluates. The table below tracks whether scores improve, stay the same, or decline after revision.

Peer Review Bias

Each paper is reviewed by all three agents. The score distributions reveal large systematic differences between reviewers.

Figure 9: How each reviewer scores papers from each agent (all 117 papers). Codex is the strictest reviewer (μ=1.9 for Kimi papers), Kimi Code is the most lenient (μ=6.2 for Claude papers).

8. Time Analysis

Figure 11: Time breakdown per pipeline stage. Experiments dominate for all agents. Claude Code spends the most time overall (13.7h avg), Kimi Code is fastest (4.1h).

Figure 12: Wall time distribution. Claude Code has a long tail with some runs exceeding 30+ hours due to experiment self-review revision loops.

9. H100 GPU Scaling Experiments

Do agents produce better research with more powerful hardware? We re-run all 8 GPU seeds on NVIDIA H100 (80GB) GPUs — nearly double the VRAM of the A6000 (48GB) used in our main experiments. All other settings remain identical. Codex runs 3 independent trials per seed (24 total).

Overall Comparison: A6000 vs H100

Per-Seed Breakdown (Peer Review, H100)

Codex H100: 3 trials per seed, scores averaged. A6000 scores from main experiments (Section 5).

10. Case Studies: Paper–Artifact Divergence

In this section, we present six illustrative case studies that support the conclusions of our manual inspection. Each case highlights a distinct failure pattern we observed across the 117 agent-generated papers, spans all three agents, and embeds the actual paper PDF so readers can inspect the evidence directly.

Case 1 · Claude Code

When Do Causal Discovery Algorithms Disagree? Diagnosing Assumption Violations via Per-Edge Profiling

Claude Code can occasionally produce a paper with a real insight, but weak evidence still prevents it from being convincing.

The paper offers a genuinely useful observation: distributional diagnostics can be detected more reliably, while structural diagnostics remain close to chance level. This kind of asymmetry is a meaningful takeaway and shows that agent-generated papers can still surface nontrivial empirical insights. However, the paper ultimately remains unconvincing because the evidence is weak. The results are not strong, the experimental settings appear chaotic in the artifacts, and the paper sometimes highlights its own method even when it is not the best or second-best. In addition, some key notions are not clearly grounded, which further weakens the paper's faithfulness.

Open in new tab · Download PDF

Case 2 · Claude Code

The Algebra of Compiler Passes: An Empirical Study of Idempotency, Commutativity, and Convergence in LLVM Optimization Pipelines

Claude Code may overclaim or present unsupported results when experiments are weak.

The paper claims evaluation on 87 benchmarks, but the artifact only supports a 20-benchmark subset. It also contains reference errors and relies heavily on synthetic programs. As a result, the paper overstates both the scale and the practical value of its findings. This case supports our observation that when experiments fail to produce sufficiently strong evidence, agents may compensate by inflating claims or presenting unsupported results. It also shows why artifact-aware review is essential: the mismatch is not obvious from the paper alone, but becomes clear once the code and outputs are inspected.

Open in new tab · Download PDF

Case 3 · Codex

Do Corruption-Family Text Residuals Help Zero-Shot CLIP? A Controlled Baseline Study

Codex reduces fabrication partly by running much narrower experiments.

Rather than producing large or ambitious evaluations, Codex tends to run controlled but very limited experiments. Here the study uses only a single frozen CLIP backbone on CIFAR-10, making the empirical scope too narrow to support broad conclusions. The idea is also close to prior prompt-based and unlabeled adaptation methods, so the novelty is modest. This case supports our claim that Codex's lower fabrication rate comes in part from being more conservative experimentally. However, that conservatism comes at a cost: the evidence is too limited, which leads to weaker papers overall.

Open in new tab · Download PDF

Case 4 · Kimi Code

DU-VPT: Decomposed Uncertainty-Guided Visual Prompt Tuning for Test-Time Adaptation

Kimi Code fabricates experimental results directly rather than actually running the experiments.

The artifact contains hard-coded benchmark statistics, and the reported per-run metrics are generated by sampling around these constants rather than by real model outputs. The published results mirror those prewritten target values almost exactly. Moreover, several analyses claimed in the paper, including forgetting analysis and shift-type diagnosis accuracy, have no implementation or logs in the artifact. This case supports our conclusion that Kimi Code often appears to fabricate results directly rather than obtaining them through actual experiments.

Open in new tab · Download PDF

Case 5 · Kimi Code

UniSched: A Critical Analysis of Simulation-Based Evaluation for CXL-Aware CPU Scheduling

Even when Kimi Code produces code, the method and implementation often do not match.

The reported results and settings do not align with the artifact, and the code itself contains clear problems, including implementation bugs in PMU-based task classification. The result files also show behaviors inconsistent with the paper's claims, such as nonzero migration counts where the method's story would suggest otherwise. Unlike Case 4, where the main issue is direct fabrication, this case shows that even when code exists, the implemented system often fails to correspond to the method described in the paper. It therefore supports our claim that Kimi's failures are not limited to fake numbers, but also include deeper mismatches between method, code, and evaluation.

Open in new tab · Download PDF

Case 6 · Claude Code

Characterizing Operator Interaction Effects in Data Cleaning Pipelines

A common failure is missing relevant baselines even when the paper substantially overlaps with prior work.

This case illustrates a common failure across agent-generated papers: missing the most relevant prior baseline even when the proposed study substantially overlaps with it. Here, the paper is highly similar to ShapleyPipe, yet it does not cite or compare against that work. As a result, the evaluation is incomplete at its core: without the most relevant baseline, the paper cannot establish either novelty or empirical advantage convincingly. This case therefore supports our broader observation that many agent-generated papers compare mainly against older or easier baselines while overlooking the most important recent or closely related methods.

Open in new tab · Download PDF