Inducing Task Models from Computer-Use Traces
Yucheng Jiang, Zora Zhiruo Wang, Ruishi Chen, Diyi Yang
arXiv:2608.20319v1Every day, organizations accumulate huge streams of computer-use traces: screenshots, clicks, keystrokes, and app switches. But those raw traces are hard to audit or reuse, because they mix multiple goals at once and only show low-level actions. This paper tackles that problem with Task Model Induction, a method for turning messy activity logs into structured task models. The key idea is to first discover the hidden tasks inside an interleaved trace, then build two kinds of representation for each task: a hierarchical model of the goal decomposition, and a procedure model of the control flow that actually carried out the work. That gives you something more like a reusable playbook than a summary. The authors show that their approach can recover interleaved tasks very accurately and reconstruct most observed steps, and that the resulting skills improve downstream task performance. That matters because as computer-use agents move into real workplaces, we need models that are not just effective, but also interpretable, auditable, and transferable.
Also spotted that day
Previous daily papers