TL;DR Models got good at math and code first because problems there have a definite right answer, and engineers and mathematicians spent decades building the tests and proof checkers that can confirm it at the scale RL needs. Most professional work has no single right answer and no public record of how it gets done, so RL has much less to train against. Computer use gets agents into the software where that work happens, and from there the problem becomes building the data and verifiers for it.
Models can code well and still struggle with ordinary office work. We think that's mostly a verification problem, and it's the problem our computer-use data and RL environment work is built around. This piece covers why the gap exists and what it would take to close it.
Why math and code went first
Models got good at math and code first because answers in those fields can be easily verified. That’s because engineers and mathematicians spent decades building test suites, compilers and proof checkers for their own work, and those same tools can now grade a model's output. DeepSeek-R1 shows the payoff, with rule-based rewards on final answers driving large gains on math and competitive coding. Software automates what you can verify, and in these two fields people had already made things verifiable.
Financial, administrative and HR tasks, the long-horizon office work that fills most of a professional's day, are much harder to verify. No sales presentation is the single correct one, and writing a code-based check for what it should contain is hard, partly because what makes it good is difficult to write down in advance, even for the people who do it best. A professional knows what feels right and what could be improved, but that knowledge lives in their head rather than in a test suite. That's why the capability profile looks so lopsided, with models now narrowly superintelligent at math and still below PhD level in most other fields. Almost every real job has parts only a person can check, and in practice "verifiable" has come to mean verifiable by software.
The further right a task sits, the more the reward has to come from a person or from the market, and the slower the RL loop runs. Computer use pulls some office tasks from the third tier into the second by making their side effects checkable.
This is also where taste comes in. Ideally you'd encode the taste and experience of a well-versed professional, and short of logging everything they do, that's difficult. So the research question for professional domains becomes how to verify a long task that pulls from several sources, goes through revisions, and ends in a deck, a document or a decision with no clean check. Better models don't make that question go away, because execution is scaling much faster than the capacity to verify it, which stays tied to human attention, and a model that generates better doesn't automatically get better at judging its own answers. Solving the verification side, even partially, is what makes RL possible in those domains, and the capability gains tend to follow.
Coding had GitHub, and office work has nothing like it
Coding also came with a public record of process. A GitHub issue, its review thread and the final fix together show what the problem was, how people argued about it, and what the accepted fix looked like. That record is what SWE-bench was built from, pairing real issues with the pull requests that resolved them and the tests that pinned each fix down. Because the work happens in one editor, it's also easy to mine for which files someone opened, how they narrowed down the bug, and which tests they wrote and ran. That gave labs a baseline for how engineers actually work, and RL could build on it because the final check already existed. In coding, the codebase holds most of the context and the output is quick to verify, while most knowledge work has neither.
Professional work is spread across dozens of enterprise applications, it's proprietary, and a lot of it contains personal information. Getting a baseline for how people actually do the job is hard. In August, Google paid $10 million in a bankruptcy auction for Spirit Airlines' internal records, which include hundreds of millions of Teams messages and employee emails along with the spreadsheets and calendars that ran the airline. Shuttered startups are selling their old Slack archives and Jira tickets, and Meta started recording its own employees' clicks and keystrokes on work apps. Fast forward to today, most of the well-established companies are licensing their historical data to frontier labs for training frontier models.
Synthetic data is too clean
Creating the data synthetically has a problem of its own, which is that the result comes out too clean. Synthetic data is bad at producing the edge cases, mistakes and noise of a real business, and you'd have to hard-code a lot to get a file system that holds proposal, proposalfinal and proposalfinalfinalv2 side by side. Nobody would design a company drive that way, but that's how people actually work. The OSWorld 2.0 team ran into the same thing when they generated tasks with LLMs and got proposals back with unrealistic inputs and shallow workflows. METR keeps its time-horizon tasks self-contained and automatically scorable on purpose, which makes them cleaner than most economically valuable work, and agents score lower on the messier ones. An agent has to operate inside the mess, and data without it can't teach that.
Labs started close to home
There's a simpler reason coding came first, which is that labs solved the problems they had. Researchers want coding agents because they write code, and many of them come from math and science, so those fields were the natural next targets. There's also the vibe check. Structured evals exist, but a researcher who wants a quick read on a model will also just try it and judge the output, which works for code or math and doesn't work for a medical or legal output unless they have that background. So those domains waited.
The computer is where the work is
Almost all white-collar work happens on a computer, spread across several applications. And with remote calls, even meetings happen there. Models already know how to do a lot of this work, and what often holds them back is access to the tools and data it depends on. Computer use is one way around that. An agent that can operate a screen the way a person does, logged into the same accounts, could in principle work in any application on the machine, including ones with no API, without someone first building a separate API or MCP connection for each.
Clicking through an interface is slower and less efficient than calling an API or running a terminal command, so the best setup uses both. The agent calls an API when one exists and falls back to the interface when it doesn't, rather than stopping to ask for help. Newer software will increasingly be built with agents in mind, but many industries will take years to get there, and agents working in them in the meantime will have to use the same interfaces people use.
The gap is widest for desktop applications, where APIs are often limited or locked down entirely, especially in places like hospitals and government agencies that restrict access for security reasons. The strongest recent systems already combine the two approaches. UI-TARS-2 trains in an environment that offers a file system and a terminal alongside the screen, and UltraCUA lets the agent switch between clicking and calling code, which cuts the number of steps and the errors that carry over from one step to the next.
Some work is also visual by nature. An agent can't finish a video edit or a 3D model without seeing the rendered result, and computer use gives it that ability.
How computer use data grew up
The first job was teaching models what an interface is. They came from text, then learned tools and then images, mostly images paired with captions, where screenshots were underrepresented and leaned toward the web. Some of what an agent has to understand on a screen is big, like which image it's currently editing, and some is tiny, like the exact button that changes a gradient. The first SFT datasets covered that layer, teaching models what's on the screen and how to move around it.
Short tasks came next, things like opening a window or changing a setting, usually fifteen to thirty steps long, which is where the original OSWorld sits. Even AgentNet, the largest open desktop trajectory dataset, with 22.5K human-recorded tasks across more than 140 applications and 190 websites, averages under 20 steps per task. Those atomic tasks are the foundation, since a model that can break a long task into small pieces can chain them together.
Long horizons break in new ways
Long tasks bring their own problems, and that's where the attention is now. The agent moves back and forth between apps, gathers information from several sources and does repetitive work, and because screenshots are token-heavy, it loses track of what it collected earlier.
Computers also do unexpected things. When a pop-up appears or an app takes too long to load, the agent often keeps acting as if the screen were the one it expected. The failure we see most often is looping, where the agent tries something, fails, decides to try the same thing again and keeps going until the budget runs out. In one large study of browser agents, repetitive clicking, back-navigation and scrolling showed up in 71% of trajectories. A plausible mechanism is that models get more likely to make a mistake once their context already contains their own earlier mistakes, so one failed click makes the next one likelier. On OSWorld 2.0, agents spend under 7% of their budget checking and repairing their own work.
Benchmarks and datasets have moved to match. OSWorld 2.0 tasks take skilled humans a median of about 1.6 hours, and Claude Opus 4.7 needed an average of 318 tool calls per task, against roughly 30 on the original. The best configuration at release finished 20.6% of tasks outright, and three months later Anthropic reports [41.7% for Claude Fable 5.1](https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude Fable 5.1 & Claude Mythos 5.1 System Card.pdf) on the August task release, with 77.9% of checkpoints passed. APEX-Agents, which drops agents into long-horizon consulting, banking and legal work, opened at 24.0%, while frontier agents sit around 80% on the original OSWorld. The best agents today can do a lot, but they still can't work an eight-hour day unless you set them up for it specifically. This long-horizon regime is also where open-source models trail closed ones the most and where they're working hardest to catch up.
The original OSWorld is close to saturated, but on each long-horizon benchmark the best model still fails more than half the time.
Inside the SuperAnnotate CUA dataset
Our dataset is built for the regime where agents fail today, which is long, multi-app work done by real professionals. Every task is a person from a specific background doing their own kind of work on their own computer, with every action logged. Each recording becomes a trajectory, a sequence of steps a model can learn from, and the easiest way to see what we're producing is to look at a single step.
What a single step contains
Every step in a trajectory carries four things.
- The task. What the person was working on when they took the step.
- The screenshot. The screen exactly as it looked before they acted.
- The action. What they did next, such as a click at a specific spot or a typed string.
- The reasoning. Why that action made sense given the task and the screen, generated after the fact from what the person did.
Note that every task is recorded as a video and then split into screenshots. Because the data is natively video, it supports more than screenshot-based training, and the original recordings can be used for approaches that learn from video directly.
The first three are what computer-use training data usually contains, and they're enough for a model to learn which action comes next. The reasoning adds the why, which gives the model richer information about each decision.
Where the reasoning comes from
Writing that explanation by hand for every step would take far too long, so we generate it from what the person actually did. We partly built on OpenCUA's pipeline, where a generator writes the reasoning behind each recorded action and a reflector checks it against the screen and the ground-truth action. In their experiments, that reflective reasoning kept performance improving as they added more data.
The dataset at a glance
Task source. Professionals across 100+ backgrounds recording real workflows on their own computers
Per-task package. Task description, initial files, final files, action log, computer state
Screen Recording. The original video of each task from start to finish, which itself can be used as training data
Step format. Screenshot paired with the action taken, one pair per step
Reasoning. Synthetic thought per step, generated from what the person actually did, built on the OpenCUA approach and modified to hold up on long trajectories
Trajectory length. Average 390, median 350, longest max steps 4033
Applications. 130+ across productivity, media, accounting, education, development, design, CAD and GIS
RL-ready subset. Tasks filtered by complexity, resettable app environment and verifiability, with an auto-generated setup script and a drafted verifier
Trajectory length
When we started, the reference point was OSWorld, with tasks of around 30 steps, so we kept our trajectories shorter than we'd planned. Then OSWorld 2.0 put hundreds of steps into a single task, and length became the point. Ours now average around 400, and the longest is close to 4033.
Why the app coverage is broad
We cover 100+ applications and add more almost every day. Productivity tools such as project management, diagrams, documents, spreadsheets and presentations are the obvious ones, media makes up a large share with audio, image and video editing and 3D modeling, and the rest spans accounting, education, development, design, CAD and GIS. The GIS tasks were a good test of the idea, because every surveying company and a long list of other businesses live in that software, no public dataset had touched it, and the tasks turned out to be exactly the visually dense, long-horizon work that agents struggle with.
The visual side is where this data earns its place. A frontier model can produce a solid presentation and still put one piece of text over another or run a label across a divider, which a person sees in half a second. Recordings where a person looks at the rendered result and fixes it are how a model learns to see it too.
That’s the kind of work we want more of, so we’re expanding our app list toward software where the visual side of the task matters and an agent can’t fall back on an API.
What comes with each task
Every recording arrives as a package. Each one includes:
- Initial files. Files the person started the task with.
- Final files. What they produced by the end.
- Action log. What they did along the way.
- Computer state. The state of the machine during the task.
That package is what lets us turn a recording into an RL task, which is where most of our engineering goes and what the next section covers.
Turning SFT recordings into RL tasks
A recording that fails the filters is still good SFT data. One that passes gets a setup script for free, plus a verifier that has to survive a golden run, a known-bad run, a human and an agent before we trust it.
Not every recording can become an RL task, and picking which ones can is part of the pipeline. We filter on complexity, on whether the apps can be spawned inside an RL environment, and on whether the final answer can be verified. Resettable state is the hard constraint, because while you can build a realistic version of the task setup for most apps, you can't spin up a fresh AWS account hundreds of times, so anything like that stays in the SFT pile no matter how good the task is.
How verifiers get hacked
A task in OSWorld 2.0 asks the agent to register for a set of classes for the coming semester, which is exactly the kind of thing a student would hand to an agent. What the checker actually rewards is whether the calendar contains events matching a few course codes, so an agent can earn full reward by working out the codes and adding one calendar event at the wrong time and place, without registering for anything. From the model's point of view nothing about this is odd, since it was told what earns reward and found a cheaper way to get there. Optimize the same agent for token efficiency and it will take the shortcut every time, leaving a student who shows up on the wrong day and was never enrolled.
It wasn’t designed to be exploitable. Someone wrote a reasonable check for a proxy of the outcome, and the proxy turned out to be easy to fake. Once you look, the pattern turns up everywhere. A public issue on the original OSWorld lists evaluators that return full reward without verifying the task was done, including a folder-rename task that passes if the agent just runs mkdir. The OSWorld 2.0 paper now runs dedicated reward-hacking audits, and its own example is an evaluator that only checked waypoints, which would have scored a driving route on a task that asked for the shortest walking route. The August update changed 14 tasks specifically to block reward hacking and another 31 to make evaluation more reliable. Audits from the past year, across very different benchmarks, keep landing on the same finding.
OSWorld to OSWorld-Verified. Tasks and evaluators after release. 400+ person-hours of checks before release and still about 300 issues after; roughly two months of a 10-person team to fix them
OSWorld 2.0, August 2026 update. Reward hacking across the task set. 14 tasks changed to block hacks, 31 modified for reliability
SWE-bench Verified. 138 audited model failures. 59.4% traced to flawed tests or task design; OpenAI stopped reporting the benchmark
Five terminal-agent benchmarks. 1,968 tasks. 323 tasks (16%) hackable by frontier models from the task description alone
RLVR math verifiers. Equivalent rewrites of gold answers. Self-validation between 53.8% and 95.2% depending on the verifier, most failures on whitespace and punctuation
The OSWorld-Verified team's own summary was that reliable rewards took far more human effort than they expected, and the problem goes beyond benchmark maintenance. METR has caught frontier models rewriting tests and scoring code to post impossible scores while being able to say, when asked, that this wasn't what the user wanted, and on one task a model reward-hacked in every single run. If the cleanest domains are still arguing with the grader, a hand-written checker for a two-hour cross-app workflow deserves suspicion by default.
The design tension runs both ways. A verifier that checks only the final answer lets the agent delete unrelated files along the way and skip two required steps without penalty, which is why OSWorld 2.0 scores side effects as their own category. A verifier that's too strict encodes one person's path and punishes the agent for finding a better one, even though finding better paths is what RL is for. That side is measurable too. When experts re-graded 1,302 web-agent trajectories for AgentRewardBench, the rule-based checkers that common benchmarks use rejected many valid runs, so reported success came in lower than an expert would put it. OpenCUA's offline benchmark handles the same problem by listing several valid actions for each step, and OSWorld 2.0 spreads the reward across about 27 weighted checkpoints per task instead of one pass/fail. We treat reward design as its own problem, deciding for each task whether necessary intermediate actions should earn partial credit, whether a final-state reward is the right shape, or whether it needs a combination. The aim is a verifier lenient enough that the agent can explore and strict enough that it can't cheat.
Drafting a verifier, then trying to break it
We're building toward drafting verifiers automatically and then trying to break them, and the current process has six steps.
- Prompt a model with the trajectory, the final output, and what we've learned from writing OSWorld-style verifiers. It produces a first draft.
- Run the golden human trajectory through it. It should return 1.
- Run a known-bad path through it. It should return 0.
- Let an agent iterate on the draft until both checks pass.
- Have a human try to break it.
- Let an agent play against it and see whether it can score without doing the task.
The first draft is almost always too lenient, and the last step, an agent attacking the checker, finds the course-registration class of bug more reliably than reading the code does. The terminal-benchmark audit above landed on a similar loop, with one agent trying to pass without solving the task, another patching the verifier, and a third confirming legitimate solutions still pass.
What a reward costs
Even with a solid verifier, you still need to know whether the task is worth training on. A task the target agent fails on every rollout is fine for SFT but useless for RL, because there's nothing to learn from, and the same goes for one it passes every time. The useful band is in the middle, where the agent sometimes succeeds.
Finding that out is expensive, and the cost compounds at every stage. A single semi-long rollout with a capable model costs around $10 and can take up to 30 minutes of environment time. One rollout says very little about whether a task sits in the useful band, so you need 8 to 16 of them before the pass rate means anything, which puts each task at $80 to $160 and several hours of environment time before a single training step. Those rollouts also have to come from the model you actually plan to train, which in practice means running against the lab’s own endpoint. Training then repeats the same expense at a much larger scale, since RL needs many rewards spread across many GPUs and each one comes from a long rollout like these. ByteDance’s UI-TARS-2 team ended up building a dedicated sandbox platform just to run rollouts at scale and still calls scalable long-horizon GUI learning an open problem.
Benchmarks are proxies, and you need several
Every benchmark is a proxy for a capability, and whether an agent can work on a computer can be tested at many levels. OSWorld gives the high-level view of whether a task got done. Its verifiers deserve scrutiny, but it's still the main benchmark for navigation and memory across apps, and it doesn't test professional work or expertise. ScreenSpot-Pro sits at the other end and measures UI grounding alone, on dense high-resolution screens from 23 professional applications, where the best model at release found the right element 18.9% of the time. GDPval tests whether a model can produce a professional artifact from a set of files, which is valuable, though the model never touches software to get there, the evaluation is one-shot by OpenAI's own description, and OpenAI's automated grader still isn't reliable enough to replace the expert graders. τ-bench adds simulated users, and its pass^k metric asks whether the agent can do the task again rather than just once.
The combination is useful, because it tells you where to work. A model that reasons correctly about the next step and then clicks ten pixels off the button every time needs grounding data, and this isn't hypothetical, since better grounding from Jedi, a 4-million-example grounding set lifted a general foundation model from 23% to 51% on OSWorld. A model that hits every button but doesn't know what to do needs reasoning data instead. No single benchmark covers end-to-end professional tasks today, though platforms like Berkeley's CUA-Lite, which puts more than a dozen computer-use benchmarks behind one interface, make this kind of cross-reading cheaper.
The benchmark we'd want would seed a computer with a company's baseline data, allow both API and UI actions, check how well the work was done as well as whether it got done, and include simulated colleagues who message you mid-task or interrupt with something else. TheAgentCompany is the closest thing to that so far, a self-hosted software startup running real GitLab, file-sharing, project-tracking and chat services with LLM-backed coworkers, across 175 tasks in engineering, project management, finance, HR and admin. What stuck with us is that agents did better on the engineering tasks than on administrative ones a person would find easier, and struggled most with complex UIs and with talking to colleagues. OSWorld 2.0's simulated user and mid-task notifications push in the same direction. Building the full version with high fidelity is hard, and it's roughly where the field is heading.
What closing that gap looks like
Closing the gap means doing for professional work what RL already did for code, with long-horizon tasks and verifiers that hold up, and for the near future we think that's the path. Coding shows what the end of that road looks like. It isn't solved, but gains now arrive in small increments at large effort, from 95 to 96 to 96.5 on a benchmark, and those increments are where most of the practical improvement shows up. Today a presentation takes many rounds of fixes, and the hope is that it soon takes two, the way coding already does.
Small gains, long tasks
There's a reason small gains go a long way on long work. If each step of a task succeeds independently with probability p, the longest task a model finishes half the time is ln 0.5 divided by ln p, which comes to about 14 steps at 95% per step, 69 at 99% and 138 at 99.5%. Sinha et al. show the same effect formally, with diminishing returns on step accuracy compounding into large gains in how long a task a model can finish.
Going from 95% to 99% per step takes the 50% horizon from about 14 steps to about 69. The OSWorld 2.0 tasks discussed earlier take frontier agents hundreds of steps.
METR measures the same gap from the other side. In its spring assessment, public frontier models had a 50% time horizon of roughly 12 hours but an 80% horizon of about 1.5 hours, and METR cautions that some tasks need 98%+ success before they're worth handing off, a reminder that being capable and being reliable are measured by different numbers.
A model that can sometimes finish a very long task is a different system from one you'd leave alone with a much shorter task, and professional work needs the second kind.
The last gap is initiative. Today one engineer with agents does the work of four and still spends the whole day guiding and managing them, and the list of reasons left to hire a person increasingly ends with the drive to manage agents twelve hours a day. You can't tell a coding agent "you're John now" and expect it to take the requests, come up with the ideas and ship. That loop isn't closed in software and is even further from closed everywhere else, so the next step is an agent you can hire for a job with some input and nobody managing it hour to hour.
Where we think this goes
We think the next phase of agent training looks less like a dataset and more like a workplace. Coding got its training ground almost by accident, because decades of repositories, tests and code reviews were already there for RL to push against. Professional work never had that, so someone has to build it, in the form of companies an agent can join, with real context, colleagues who send requests and change their minds, and work that gets judged on quality as well as completion. It's also the only route we see to the agent you can hire for a job instead of supervising it hour by hour, since an agent can only learn to own work in an environment where there is work to own.
That's what our computer-use work is building toward, starting from recordings of real professionals doing long tasks and moving toward environments a lab can train in, with verifiers that hold up when an agent tries to cheat them.
The open question for the field is how much professional judgment can become a reliable reward without cleaning out the mess that makes the work professional.