1. Introduction

We believe this paradigm—real shells, real filesystems, real build and test tooling, and execution-verified rewards—provides the right substrate for evaluating LLM coding agents. We build on the Harbor task format because it offers a standardized, reproducible way to place agents in executable engineering environments where correctness can be verified by running code rather than judging surface-level outputs. On top of this substrate, we construct three datasets along orthogonal capability dimensions, together covering the full spectrum of real-world coding work:

The three datasets use different task formats, but the admission criteria are shared:

• Quality: Every task is grounded in real-world engineering scenarios, oracle-verified (reward = 1.0), and reviewed by domain engineering experts for technical accuracy and test fairness.
• Diversity: DeepTerminalBench tasks span 20+ domains and 12 workload types; multi-turn tasks span 4 engineering activities and 3 interaction styles; version upgrade tasks span 17 repositories in 5 programming languages — ensuring no critical skill is left untested.
• Scalability: Task designs draw from continuously growing real-world sources (GitHub, Stack Overflow, CVE databases, RFCs, production postmortems, CS textbooks, and open-source release cycles). New domains, languages, and difficulty tiers can be added as the engineering landscape evolves.

2. Three Datasets at a Glance

This table fixes the comparison axes used throughout the article: evaluation axis, task unit, starting state, scoring rule, and difficulty source.

3. Shared Foundation: The Harbor Task Spec

All three datasets use the Harbor task specification introduced by Terminal-Bench 2.0. This makes the benchmark runnable by Harbor-compatible scaffolds such as Claude Code, OpenHands, Codex CLI, and Terminus-2, while keeping the new contributions concentrated in task semantics: multi-turn state for EvoCodeBench and version-upgrade structure for RoadmapBench.

A stock single-shot Harbor task contains five components:

Each dataset extends this stock format in a different way. The next section walks through the three variants.

4. Task Format: Three Variants

4.1 DeepTerminalBench: Flat Structure

DeepTerminalBench uses the stock Harbor task layout unchanged:

The execution model is simple: Harbor builds the Docker environment, delivers instruction.md to the agent, the agent produces code, and tests/test.sh checks correctness. One instruction, one execution, one verification.

4.2 Multi-Turn (EvoCodeBench): Rounds with Cumulative Tests

For multi-turn tasks, we introduce round directories. Each round carries its own instruction, solution, and tests, while the environment and top-level metadata remain shared:

Three things make this work. First, solutions are incremental — each round's solve.sh applies only that round's delta. Second, tests are cumulative — round N's tests re-verify everything still valid from rounds 1 through N, so regressions caused by later changes are caught immediately. Third, tests verify behavioral contracts rather than implementation details: instructions describe what the system should do, tests check the system's external behavior, and round N's tests cannot assume the agent chose the same code structure as the reference solution in round N-1. Together these principles allow different agents to reach equivalent behavioral goals through divergent implementation paths without breaking the evaluation.

4.3 Version Upgrade (RoadmapBench): Repository Snapshot + Roadmap Phases

RoadmapBench keeps the single-shot execution model but reshapes what each component contains. The environment is a real-world repository snapshot pinned at an earlier release (with commits and tags after that point stripped to prevent information leakage), and the instruction is a multi-phase roadmap describing behaviors introduced in the target version:

Like EvoCodeBench's principles, the roadmap specifies WHAT to build but never HOW — no algorithm names, implementation steps, or code snippets. Tests are organized as phase-level files (test_01_.py, test_02_.py, …) adapted from the official upstream test suite, and a runner computes a weighted reward across phases.

4.4 Side-by-Side Structural Comparison

5. Sources & Coverage

5.1 A Shared Pool of Real-World Problems

All three datasets draw from the same kinds of real-world engineering artifacts — the scenarios that practicing engineers actually encounter. Grounding tasks in these sources keeps the evaluation tied to observable engineering work and gives the datasets a renewable source of future tasks.

These sources are continuously growing. New CVEs are published daily, new open-source projects introduce novel architectures, new RFCs define new protocols, and every release cycle yields fresh version pairs — all three datasets can expand without exhausting authentic task sources.

5.2 Three Complementary Taxonomies

Each dataset uses the two axes that expose its main source of score variation. DeepTerminalBench varies engineering domain and workload type; EvoCodeBench varies interaction style and engineering activity; RoadmapBench varies programming language and source repository.

DeepTerminalBench: Domain × Workload Type

Tasks are organized along domain (what area of engineering or science) and workload type (what kind of work the agent must perform). The cross-product covers over 20 domains crossed with 12 workload types. The 50 curated tasks sample broadly rather than exhaustively covering every combination; the taxonomy reserves additional domains for future expansion, so adding one new domain immediately creates 12 new domain-workload combinations.

Domains span: Code and Build, Repo Engineering, Testing and Quality, Debugging and Observability, Data Engineering, Database and Storage, Systems and Networking, Cloud and DevOps, Security, ML/AI and MLOps, Scientific and Numerical Computing, Automation and Productivity, Compiler and Language Tooling, File Format and Protocol, Distributed Systems and Concurrency, API and Schema Design, Data Structures and Algorithms, Simulation and Modeling, CLI and Interactive Tools, Workflow and Rule Engine — with Embedded/IoT, Container Orchestration, Cryptography, and Network Protocol Engineering reserved for expansion.

Workload types include: Greenfield Implementation, Brownfield Modification, Bug Localization and Fix, Migration and Upgrade, Integration and E2E Wiring, Performance Optimization, Reliability and Recovery, Security Patch and Hardening, Reproducibility and Verification, Forensics and Analysis, Automation Scripting, and Explanation and Reporting.

EvoCodeBench: Interaction Style × Engineering Activity

Multi-turn tasks vary along interaction style (how information flows across rounds) and engineering activity (what type of work each round involves).

The 3 interaction styles crossed with 4 engineering activities yield 12 task-type combinations. Allocation across EvoCodeBench:

RoadmapBench: Programming Language × Repository Domain

RoadmapBench's axes reflect a reality: software engineers work across languages, and library evolution patterns recur in every ecosystem. The current release covers 17 repositories across 5 languages.

Why Three Different Taxonomies

Each taxonomy answers a different diagnostic question. DeepTerminalBench's Domain×Workload table shows which kinds of engineering work a model handles in a single dense session. EvoCodeBench's Interaction×Activity table shows which forms of requirement change break multi-turn reliability. RoadmapBench's Language×Repository table shows where version-upgrade ability depends on ecosystem-specific APIs, build systems, and conventions.

6. Dataset Statistics

6.1 Side-by-Side Task Statistics

6.2 Difficulty Calibration: Three Methodologies

The three datasets calibrate difficulty in different ways because difficulty means different things across the three settings.

6.3 Agent Behavior Under Evaluation

DeepTerminalBench: Opus 4.6 Across 200 Rollouts (50 tasks × 4 attempts)

DeepTerminalBench takeaways:

• 44% of tasks trigger at least one timeout across 4 attempts, indicating that many tasks push agents to their execution budget limits.
• ~83% of tasks include pre-existing environment files that the agent must read and work with before writing anything new.
• Lower Pass@4 corresponds to longer attempts — Pass@4 = 0.25 tasks average 31 minutes per attempt, while Pass@4 = 0.75 tasks average 15 minutes.

EvoCodeBench: Multi-Turn Results Across 13 Models

EvoCodeBench takeaways:

• Persistent execution scores lower than isolated-round evaluation. Claude-Opus-4.6 reaches 78.9 SR but 44.0 MT@4; GLM-5.1 drops from 63.9 SR to 36.2 MT@4; Kimi-K2.6 drops from 59.0 SR to 31.9 MT@4.
• End-to-end completion remains limited. Claude-Opus-4.7 has the highest full-task completion at 42.3, followed by GPT-5.5-High at 38.5 and Claude-Opus-4.6 at 34.6.
• The common failure symptom is concrete, not generic. In 957 high-confidence annotations from executed failed round fragments, 89.1% are missed active-round obligations, 6.5% are environment/tooling failures, and 3.4% are regressions. Correction rounds raise the regression share to 11.2%; conflict-labeled rounds raise conflict mishandling to 14.9%.
• Persistent state is the central stressor. The same round instruction receives higher scores when earlier rounds have already been applied by the reference solution than when the model must live with its own earlier edits.

RoadmapBench: Phase-Weighted Scoring Produces Graded Separation

The leaderboard results above demonstrate that multi-phase weighted scoring naturally separates model capabilities: the top model (Claude-Opus-4.6) achieves 32.2% resolved and 0.627 completion score,. Crucially, even models with low resolve rates achieve meaningful completion scores — confirming that phase-level partial credit exposes capability gradients invisible to binary pass/fail evaluation.

6.4 Comparison with Related Benchmarks

Oracle patches in RoadmapBench are roughly two orders of magnitude larger than those in bug-fix benchmarks, confirming that version-upgrade tasks require sustained multi-step engineering rather than single-function edits.

7. Verification & Scoring

7.1 Shared Principle: Execution-Based Verification

All three datasets emphasize dynamic, execution-based testing over static output checking. The majority of verification compiles, executes, and observes program behavior at runtime. Static output checks supplement execution-based verification but do not replace it. This layering prevents "teaching to the test" — solutions that produce correct output but use incorrect or insecure implementations will fail behavioral or attack-based verification.

For DeepTerminalBench, the median task employs three verification strategies simultaneously. The prevalence of each strategy across the 50 tasks:

EvoCodeBench and RoadmapBench tests follow the same execution-first philosophy but apply it at the round/phase level rather than task level.

7.2 Three Scoring Schemes

Because the three datasets have different task units (single session vs. multiple rounds vs. multiple phases), each uses a different scoring formula. All three share one principle: the scoring unit is a binary pass at the lowest level, and the task score aggregates these.

8. Quality Assurance

8.1 Shared Gates: Structure + Oracle + Expert + Calibration

All three datasets share a four-gate quality assurance pipeline before any task is admitted:

1. Static Structure Validation — The task package is well-formed: Dockerfile builds, test harness conforms to conventions, metadata is valid, instruction provides sufficient context. Automated checks guarantee structural integrity.
2. Oracle Verification — The reference solution is deployed into the task's Docker container and the full test suite runs end-to-end. Only tasks where the oracle achieves reward = 1.0 (RoadmapBench: also baseline < 1.0) are retained; the oracle run is the solvability evidence for admission.
3. Agent-Assisted Expert Review — Domain engineering experts review task instructions, test suites, and oracle solutions for technical accuracy, specification completeness, and test fairness. Experts verify that instructions contain sufficient information for the task to be solvable without hidden assumptions, that test assertions are reasonable and not overly brittle, and that the oracle represents sound engineering practice. Automated checks ensure structural integrity; expert review checks whether the task is a valid engineering evaluation item.
4. Multi-Model Pass@k Calibration — Multiple LLM agents attempt each task (k=4 trials). Tasks passed by every model on every attempt or failed by every model on every attempt are flagged and either adjusted or excluded.

Beyond these shared gates, each dataset has one additional mechanism tailored to its setting.

8.2 Dataset-Specific QA Mechanisms

The T-type/M-type methodology is most formally developed in RoadmapBench, but the same spirit applies to DeepTerminalBench and EvoCodeBench quality control.

9. Infrastructure: Extending Harbor

Harbor was originally designed for single-shot tasks: one instruction, one agent execution, one verification. DeepTerminalBench uses that engine unchanged. EvoCodeBench adds persistent sessions, round boundaries, and resume support; RoadmapBench adds pinned repository images and phase-level scoring. The implementation differences explain what each benchmark setting costs to run reproducibly.

9.1 Baseline: Harbor's Single-Shot Engine

The baseline execution model is simple: Harbor builds the Docker environment, delivers instruction.md to the agent, runs the agent inside the container, then runs tests/test.sh to produce a reward. DeepTerminalBench uses this directly — every DeepTerminalBench task is a stock Harbor task, runnable by any Harbor-compatible scaffolding without modification.

9.2 EvoCodeBench Extensions: Persistent Multi-Round Execution

Supporting multi-turn evaluation required rethinking several layers of Harbor. The overarching goal: make the evaluation environment behave like a real multi-turn coding session. When a user works with a coding agent across multiple turns, the agent remembers the conversation, the file system accumulates changes, installed packages persist, and running services stay alive. None of this resets between turns.

Persistent Environment

The core design decision is one Docker container and one agent session for the entire multi-turn task. The container is created once at the start. The agent is initialized once. Neither is torn down or rebuilt between rounds. All state accumulates naturally: files written in round 1 are visible in round 5, packages installed in round 2 remain available in round 4, the agent's own conversation history carries over throughout. Restarting the environment between rounds would measure a different setting: isolated repeated tasks rather than one persistent coding session.

Round Boundary Protocol

Between every two consecutive rounds, the framework executes a round boundary protocol:

1. The agent completes its work for the current round.
2. The framework runs the current round's cumulative verification script.
3. A binary reward (pass/fail) is recorded.
4. If the round passes, an optional state snapshot is saved, and the next round begins.
5. If the round fails, the entire task terminates immediately (fail-stop).

Fail-stop is deliberate: since later rounds depend on correct earlier implementations, allowing an agent to continue after a failure would produce meaningless results. From the agent's perspective, nothing visible happens at round boundaries — it simply receives the next instruction and continues working. Verification and checkpointing happen transparently.

State Management and Partial Evaluation

Building a multi-turn dataset requires running thousands of trials during development. Running every task from round 1 every time would be prohibitively expensive. We need the ability to start from the middle, which means placing the environment into the exact state it would be in after round N-1 without actually executing rounds 1 through N-1.

Two mechanisms address this:

• Reference solution fast-forward. The framework executes early rounds using the task's reference solution to bring the environment to a known-good state, then hands off to the target agent at the desired round. Fast-forwarded rounds are not verified and not scored; they serve purely to prepare the workspace.
• Snapshot-based resume. At each round boundary, the framework saves a full snapshot of the container state. A later run can restore from any saved snapshot, skip all prior rounds entirely, and begin from that checkpoint. Historical round results are preserved and carried forward into the new run's scoring.

These enable (1) difficulty calibration — iterate on a specific round's design without re-running the chain; (2) partial evaluation — test an agent on rounds 3–5 only; (3) cross-agent comparison — run the reference solution through all rounds, then resume with different agents from the same checkpoint.

9.3 RoadmapBench Extensions: Pinned Images and Phase Orchestration

RoadmapBench extends Harbor more modestly, along two axes:

• Pinned baseline Docker image. The Docker image built at Gate 1 verification (where the baseline fails and oracle passes) is saved and reused for all subsequent evaluations, ensuring reproducibility across runs even as upstream dependencies drift.
• Phase-level test orchestration + weighted reward. The test runner executes phase-level test files (test_01_.py, test_02_.py, ...) independently, records per-phase pass/fail, and computes the weighted sum as the task reward. Partial-progress scoring requires the runner to continue past phase failures rather than short-circuit, unlike EvoCodeBench's fail-stop model.

Beyond these, RoadmapBench reuses Harbor's stock execution engine without modification — a single instruction, a single agent session, a single verification pass.

9.4 Shared Limitations

• Snapshot restoration is not perfect. EvoCodeBench snapshots capture the container's file system but not all runtime state. If an agent's work depends on running background processes, in-memory caches, or live network connections from earlier rounds, these will not survive a save-and-restore cycle.
• Tests must be deterministic and non-interactive. All verification scripts must run without human input and produce consistent results. Tests depending on external network state, wall-clock time, or interactive prompts are not supported.
• Snapshot storage cost. For long EvoCodeBench tasks, full container snapshots across many rounds can accumulate significant storage. By default, snapshots are saved only for rounds that pass verification, reducing storage for failed trials.
• No mid-round recovery. If an agent crashes mid-round in EvoCodeBench, that round must be re-executed from the previous snapshot.

10. Task Examples

Below are two representative tasks per dataset, chosen to cover different task structures within each benchmark. For space, we summarize each example rather than reproducing the full instruction, focusing on the design pattern and the difficulty mechanism.

10.1 DeepTerminalBench

10.2 EvoCodeBench

Representative Real Task: Deterministic Data Pipeline CLI

This example is a released EvoCodeBench task, deterministic-data-pipeline-go. It asks the agent to build and then extend a Go CLI called dpipe for deterministic CSV processing, reproducibility metadata, validation, profiling, lineage, quality checks, snapshots, and changelogs. The task is representative because it combines all three multi-turn stressors in one session: cumulative feature growth, corrections to earlier behavior, and explicit conflicts with prior output formats.

The key property is cumulative verification. A correct round 15 implementation cannot merely satisfy the latest hash-format change; it must also preserve deterministic output, earlier checksum semantics where still valid, profile/validate/quality/schema behavior, audit logging, lineage metadata, and snapshot/changelog contracts introduced across the previous 14 rounds. This is the EvoCodeBench setting in concrete form: later requests are small, but each one is evaluated against the whole evolving tool.

10.3 RoadmapBench

Example 1: PyTorch Geometric 2.4 → 2.5 (Python)

Example 2: Fiber v2.49 → v2.50 (Go)

11. Evaluation Insights

11.1 DeepTerminalBench: Per-Task Pass/Fail Map

11.2 DeepTerminalBench: 200-Rollout Deep Dive

We examined Claude Opus 4.6 across 200 rollouts (50 tasks × 4 attempts). Since tasks were selected using Opus Pass@4 calibration, this is not a model leaderboard; it is a task-difficulty analysis using one high-scoring agent as the probe.

Across 200 rollouts, 89 succeeded and 111 failed. Comparing these two populations gives three measurable contrasts:

• Failed attempts run longer. 17.7 minutes vs. 17.0 minutes for successes — failing agents spend time on unproductive exploration.
• 23% of failures end in timeout. These are not wrong answers produced quickly; they are cases where the agent ran out of budget while still actively working — tasks genuinely require sustained multi-step reasoning.
• Lower Pass@4 tasks take longer. Pass@4 = 0.25 tasks average 31 minutes per attempt; Pass@4 = 0.75 tasks average 15 minutes.
• Multi-file bug diagnosis, cross-component integration, deep domain knowledge are common among the 22 tasks in the Pass@4 = 0.25 band — understanding WAL semantics, K8s scheduling priorities, or C memory conventions inferred from context rather than stated.

11.3 EvoCodeBench: Single-Round Skill Does Not Become Multi-Turn Reliability

EvoCodeBench measures a pattern absent from single-shot benchmarks: agents can solve many individual rounds when each round starts from a clean reference state, but fail when they must carry forward their own implementation. On 26 multi-turn tasks and 227 evaluated rounds, Claude-Opus-4.7 leads MT@4 at 54.0, GPT-5.5-High follows at 52.4, and Claude-Opus-4.6 reaches 44.0. These are the only three models above 40 MT@4.

The SR-to-MT@4 gap quantifies the cost of carrying state across rounds. Claude-Opus-4.6 reaches 78.9 single-round pass rate but 44.0 MT@4. GLM-5.1 reaches 63.9 SR but 36.2 MT@4, Kimi-K2.6 reaches 59.0 SR but 31.9 MT@4, and DeepSeek-V4-Pro reaches 56.4 SR but 30.6 MT@4. The drop is evidence that isolated instruction following is insufficient; the model must preserve a correct evolving workspace while requirements accumulate, change, and sometimes conflict.

Round-level results show how quickly reliability decays. Averaged across model-round observations, pass rates fall from 46.7 at round 1 to 26.9 at round 3, 21.3 at round 5, and 17.2 at round 7. Under fail-stop scoring, these drops quantify the cost of long-horizon state management: one regression can terminate the rest of the task.

Failure annotations sharpen the interpretation. The largest category is not broad conversational forgetting: among high-confidence labels, most failures are concrete current-round specification misses under cumulative tests. Regression and stale-contract errors are minority modes overall, but they become more visible on correction and conflict rounds, where the agent must revise behavior without breaking still-active obligations.

11.4 RoadmapBench: Cross-Model Separation

RoadmapBench results reveal clear separation across model capabilities. Claude-Opus-4.6 leads with 32.2% resolved (OpenHands) and 31.3% (Terminus 2), Failure modes differ systematically by capability level:

• Top-tier models (Claude-Opus-4.6, GPT-5.4) complete core phases reliably but miss subtle edge cases — error behavior when constraints conflict, API contract details not explicitly spelled out, or integration between newly introduced modules.
• Mid-tier models (Gemini-3.1-Pro, DeepSeek-V4-Pro, GLM-5.1) pass the first 1–2 phases but fail to integrate later phases that depend on earlier architectural decisions.
• Lower-tier models fail at the initial architectural setup — placing new code in the wrong modules, missing export registrations, or breaking existing APIs before implementing new ones.

The Completion Score captures this gradient: even models with low resolve rates achieve meaningful partial credit (e.g., DeepSeek-V4-Pro: 18.3% resolved but 0.486 completion), confirming that phase-weighted scoring exposes capability differences invisible to binary evaluation.

11.5 Capability vs. Consistency: A Cross-Dataset Finding

12. Looking Forward: Toward Adaptive Evaluation

The remaining evaluation constraint is path divergence: two correct agents may reach the same behavior through different code structures, and tests must accept both without becoming too loose. EvoCodeBench handles this with behavioral-contract tests, RoadmapBench handles it with attribution-driven task fixes, and DeepTerminalBench uses Pass@k calibration to filter brittle tasks. These mechanisms reduce false failures, but they still rely on human experts to anticipate valid implementation paths.

The research direction is an adaptive evaluation loop that generates later instructions and tests from the agent's actual intermediate state:

13. Summary

• Three directions of terminal-based evaluation: We expand Terminal-Bench along three orthogonal directions — depth, iteration, and evolution — all grounded in containerized terminal environments that exercise the same tools, constraints, and feedback loops as human engineers. Task designs draw from continuously growing real-world sources (GitHub, CVEs, RFCs, production postmortems, CS textbooks, and open-source release cycles), enabling the datasets to expand as the engineering landscape evolves.
• DeepTerminalBench — depth: 50 curated tasks selected from a larger pool using Claude Opus 4.6 Pass@4 calibration, each a complex engineering task in a rich pre-configured environment (median 10 files, up to 50+). Verification is primarily dynamic and execution-based — programs are compiled, run, and their runtime behavior validated, including adversarial security payloads and performance gates. The main leaderboard reports Claude-Opus-4.7 at 34.0 Pass@1, with GPT-5.5-high, DeepSeek-V4-Pro, Kimi-K2.6 and Qwen-3.6-Plus all clustered at 32.0; the separate 200-rollout analysis uses four attempts per task to analyze task-difficulty variation and failure patterns.
• EvoCodeBench — multi-turn iteration: 26 multi-turn tasks comprising 227 evaluated rounds capture the core stressors of iterative agent work: cumulative state management, requirement evolution, specification conflicts, and backward compatibility. Tasks are organized along two orthogonal axes — interaction style (explorative, contractual, document-driven) and engineering activity (construction, spec evolution, review-driven improvement, migration) — yielding 12 distinct task-type combinations. Evaluation across 13 models reports long-horizon score decay: Claude-Opus-4.7 leads at 54.0 MT@4, GPT-5.5-High follows at 52.4, Claude-Opus-4.6 remains at 44.0, and all other models are at or below 36.2. All tests verify behavioral contracts rather than implementation details, directly addressing path divergence in multi-turn evaluation.
• RoadmapBench — version upgrade evolution: 115 tasks across 17 repositories in 5 programming languages (Python, TypeScript, C++, Go, Rust) target long-horizon feature implementation across real version releases. Each task gives the agent a repository pinned at an earlier release and a roadmap-style instruction; the agent must implement behaviors introduced in the target version without access to the new source code, test files, or oracle patch. Multi-phase weighted scoring captures partial progress (median 5 phases, max 12), and oracle patches are roughly two orders of magnitude larger than bug-fix benchmarks (~3,700 LOC median). Attribution-driven quality control separates task defects from model-side failures, so remaining failures are attributable to model behavior rather than dataset defects.
• Shared infrastructure: All datasets are built on the Harbor evaluation framework. For multi-turn tasks, Harbor is extended with persistent environments, round boundary protocols, state snapshots, and partial evaluation support. For version upgrade tasks, Harbor manages pinned Docker environments, phase-level test orchestration, and weighted reward computation — enabling reproducible, scalable evaluation across all three dataset dimensions.