RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources

Resource2Skill, developed by Microsoft Research with collaborators at UC Santa Cruz and Shanghai Jiao Tong University, converts human-created tutorial videos, code repositories, articles, and reference artifacts into executable procedural knowledge for software agents. Its central artifact is a hierarchical multimodal Skill Wiki: each entry combines structured instructions, visual examples, adaptable or executable code, metadata, and provenance. At runtime, MetaBrowse uses BM25 to narrow the taxonomy, then a language model selects complementary skills that execute through MCP tools; the same distillation operator can search for missing capabilities online. Across Web, Excel, Reaper, PowerPoint, Blender, CAD, and Unreal Engine 5, Resource2Skill raises average artifact quality by +11.9 percentage points over agents without skills and wins in 26 of 28 model-domain aggregate cells, outperforming Anthropic’s Claude Code and OpenAI’s Codex harness baselines. On novel tasks, an offline pool of 891 skills augmented with 100 online skills improves performance by +21.6 percentage points, from 41.2% to 62.8%. Ablations show that the full multimodal wiki and MetaBrowse selection outperform text-only, embedding, and no-skill alternatives; the hierarchy-then-language-model selector reaches 68.9%, versus 57.3% without skills. A blinded human study also favors the skill-enabled system in 85.5% of non-tied comparisons, while failure cases reveal unresolved parameter binding and overly literal skill composition as remaining limitations.