Your Computer-Use Agent Needs a Task Model, Not Just a Screen Recording

A screen recording of someone doing office work looks like training data. In production, it is something more sensitive: a partial map of how the business actually runs.

That distinction matters as computer-use agents move from controlled demos into browsers, spreadsheets, admin portals, customer-service consoles, and internal back-office systems. If an agent can watch a person complete a task, the tempting next step is to ask the model to copy the clicks. But copying the clicks is not the same as understanding the job. It does not tell the organization which goal was being pursued, which steps were incidental, which branches depended on policy, which fields contained private information, or which parts of the workflow require human approval.

A new arXiv paper, Inducing Task Models from Computer-Use Traces, points toward a better layer for this problem. The authors describe naturalistic computer-use traces — screenshots plus mouse and keyboard actions — as a resource for deriving symbolic, auditable, and reusable models of everyday work. Their system, Task Model Induction, tries to discover latent tasks inside unconstrained traces, disentangle concurrent activity, and produce two linked outputs: a hierarchical objective model and a procedure model for the control flow that organized the work.

That is a more useful target than another pile of videos. A task model says, in effect: here is what the worker was trying to accomplish, here are the subgoals, here is the order that mattered, here are the branches that appeared, and here is the procedure that can be reviewed before anyone turns it into an automated skill.

The reported results are early but notable. In the paper abstract, TMI recovers interleaved tasks with 0.974 agreement against ground-truth groupings on controlled human and agent trajectories, reconstructs 74.9% of observed execution steps, and improves held-out task accuracy by 30.0% when skills derived from its task models are reused. Those numbers should not be read as a universal production guarantee. They should be read as a signal that the unit of reuse for computer-use agents may need to shift from “trace” to “model.”

For enterprise teams, that shift is practical. A raw trace is hard to govern. It may contain credentials, customer details, one-off workarounds, irrelevant browser tabs, personal habits, and policy exceptions that should never become reusable automation. A task model can become a review object. Security can ask whether it includes sensitive fields. Operations can ask whether the steps match the current standard operating procedure. Compliance can ask where approval gates belong. Engineering can ask which parts are deterministic enough for tools and which parts should remain model-mediated.

The adjacent research on agent skill transfer adds an important warning. Break It Down, Pass It On studies how LLM agents induce and reuse skills, and its abstract says induced skills can transfer unreliably or even hurt performance. The authors report that task-level skills often reduce performance below a no-memory baseline, while subtask-level skills perform better on average, and that text skills transfer better than code skills. They also propose a utility score based on specificity and abstractness as a diagnostic before execution.

That finding is exactly the caution teams need. “The agent watched Jane process a refund, so now it has a refund skill” is too coarse. Jane may have handled a special case. The account may have had a risk flag. The relevant step may have been the policy lookup, not the button sequence. A production agent memory layer needs to break work into reusable subtasks, preserve the rationale in human-readable form, and score whether the induced skill is likely to transfer before using it on a live case.

The same lesson appears from a different angle in MidTool, which argues that general tool use benefits from dedicated mid-training rather than being left entirely to post-training. Its construction pipeline includes real-world tool APIs, MCP skills, and document-grounded workflows to teach affordance recognition, argument grounding, workflow composition, and recovery from incomplete information. In other words, reliable tool use is not just a prompting trick. It is a training, data, and workflow-design problem.

Put these pieces together and a production pattern emerges. First, capture traces only inside explicit, approved boundaries. Second, redact or exclude sensitive content before model processing wherever possible. Third, segment the trace into candidate tasks and subtasks. Fourth, induce a task model that separates objective decomposition from procedural control flow. Fifth, send that model through human review before packaging anything as a skill. Sixth, attach diagnostics: transfer utility, source trace lineage, version, owner, expiration date, and the conditions under which the skill is allowed to run. Finally, monitor drift when the UI, policy, or downstream system changes.

That architecture may sound heavier than “record and replay,” but it is the difference between automation and operational knowledge management. A replay bot knows where the mouse went last Tuesday. A task-model registry knows what the business believes the process is, who approved that belief, what evidence it came from, and when it should be rechecked.

This also fits the simpler-agent guidance Anthropic has emphasized: start with clear, composable patterns and transparent traces before adding unnecessary complexity. The irony is that a task-model layer can make agent systems both more capable and simpler to operate. Instead of asking a general agent to infer the whole job from scratch every time, the organization gives it a reviewed map of the work. Instead of trusting a black-box memory, the team maintains a living library of procedures.

The hard part is not generating the first model. The hard part is keeping the model honest. Computer-use agents will be most valuable in messy workflows where documentation is stale, exceptions are common, and the UI changes faster than the process manual. That is exactly where teams must resist turning every successful trace into durable memory. Some traces are examples. Some are bugs. Some are policy violations. Some are useful only after being decomposed into smaller, safer pieces.

The practical takeaway is straightforward: if your roadmap includes computer-use agents, build the task-model layer before you scale the agent. Decide what counts as an approved trace, how sensitive data is filtered, who can promote a model into a reusable skill, how transfer is tested, and how obsolete procedures expire. Treat task models as governed artifacts, not model exhaust.

The future of computer-use automation will not be won by the team with the largest folder of screen recordings. It will be won by the team that can turn observed work into auditable, reusable, and maintainable process knowledge — and knows when not to reuse it.