Skill Recorder

A desktop recorder that turns a performed task into a reusable agent skill. On macOS and Windows 11 (x64 and ARM64) it captures low-frame-rate screen video, app and window switches, browser URLs, clipboard previews, an optional session-owned terminal with full command output, and optional spoken narration transcribed on the machine by Whisper. Nothing leaves the computer until the user presses Analyze, which sends the timeline to the bundled GitHub Copilot CLI to reconstruct one overall intent and an ordered list of steps. From an approved analysis it builds a Skill — a SKILL.md procedure rendered from a reviewable plan — or a scheduled Automation, for Microsoft Scout, Microsoft 365 Copilot Cowork or a generic skill-capable agent, with Copilot Studio still disabled. A secret-and-PII detection pass runs on device before anything is sent, and skill creation prefers the target agent’s native tools, such as the gh CLI, over replaying the recorded clicks. It ships as a source release: an installer builds the exact tagged commit on the machine.

Who built itA Microsoft repository rather than a personal one, released under Microsoft’s MIT license and CLA process. Its 164 commits come from two people: Giorgio Ughini, 78, and Adi Leibowitz, 81 spread over three email addresses and two linked accounts; a colleague added two and Dependabot two. 120 commits carry a co-author trailer and 100 of those say only “Copilot App”, so most of the history was written with Microsoft’s own coding agent in the loop.

1. 01

   Ten weeks, thirteen releases, and work that lands on a release branch

   The repository was created on 2026-07-29 — its first commit is dated 2026-07-24 — and has taken 164 commits: 107 in July, 34 in August and 23 in September, spread over thirteen releases, from v0.1.0, v0.2.0 and v0.2.1 published one after another at 19:50 on the same evening to v0.7.0 on 2026-09-21. Three of them landed on a single day, 2026-09-16 — v0.6.0 at 08:49, v0.6.1 at 12:06, v0.6.2 at 13:27 — which is the cadence of a team that publishes a patch as soon as a fix is verified instead of batching work. The unusual part is where a change lands: .github/copilot-instructions.md tells coding agents that pull requests must target the active release/<semver> branch and not main, because main trails the release branch and only advances when a version is cut and merged. That file still names release/0.3.0 as current, which is what a document edited once per release looks like when nobody revisits it. RELEASING.md then spells out the mechanics: a release pull request bumps package.json and package-lock.json without creating a tag, the annotated tag is created only after the resulting main commit passes every workflow, and the release notes must carry the full commit SHA plus the SHA-256 of install.ps1 and install.sh. Around all of it sit roughly 4,168 stars, 440 forks and 41 open issues.

2. 02

   Two names in the history, and a trailer that reads Copilot App

   The contributor list has six entries, but the work is really two people: Giorgio Ughini, 78 commits across two email addresses, and Adi Leibowitz, 81 commits across three addresses and two GitHub accounts (adilei and adilei-powerapps). A colleague, Dan Fiedler, contributed the commit that pinned the GitHub Actions to full-length SHAs, Ramakrishnan Raman added two, and Dependabot two. What makes the attribution worth reading is the trailer census: 120 of the 164 commits carry a co-author line, and 100 of those say only “Copilot App”, with 17 for adilei, two for the dependency bot and one for Ughini himself. So the repository declares, per commit, that most of this history was written with Microsoft’s own coding agent in the loop. Outside contributions arrive and then wait: two pull requests adding Simplified Chinese localization (#72 and #94, different authors) are both still open, as are the skill-runtime eval proposal (#70, open since 2026-08-24, where a maintainer asks “so, what is this exactly doing?”), an audio stop-flush fix (#93) and a set of Windows launch helpers (#67). Each attracts the same CLA bot comment, and the author of #68 writes that the Windows and Non-Windows workflow runs sit at action_required until a maintainer approves them.

3. 03

   A plan first, a file second

   The Skill Builder is a two-phase conversation, and its brief says so in a heading: “Two phases — never skip the plan.” The agent reads the approved analysis, calls propose_plan with its generalization, the fixed values it intends to hard-code and the ordered steps, then stops; the user answers in natural language, the plan is revised, one proposal per turn. submit_skill, the call that writes the artifact, follows only after a message says the plan is approved. The plan is typed rather than prose: common/skill.ts keeps a Zod schema in which every step is a calculation (reads, derives, decides) or an action (submits, sends, creates, deletes), and a literal that is the same on every run becomes a value with an id, a label and the exact string, referenced from step text as {{id}}. Rendering is deliberately literal: renderSkillMarkdown writes YAML frontmatter with the kebab-case name, a description emitted through JSON.stringify because that yields a valid YAML double-quoted scalar and a colon or comma therefore cannot break the file, an optional allowed-tools list, then the body with every token substituted. Release 0.7.0 added the other half of that bargain: an optional read-only Review SKILL.md renders the complete file without writing anything, serves those bytes to Add or Export without another model turn, and is discarded when the plan is edited.

4. 04

   The regression that produced an eval suite

   evals/ exists because the interesting failures are not in the recorder. The describer harness materializes a synthetic session (session.json and events.jsonl) into a temporary sessions root, runs the real pipeline and the real Describer, and scores the result against a rubric, with no video and none of the flakiness of live capture — fifteen to twenty-five seconds a scenario. Scoring is deterministic and model-free: the intent must name the right subject, the step count must sit in a range, the expected applications must appear, the key actions must form an ordered subsequence, and recorder bracketing, permission dialogs and tracking-parameter hops must not become steps; a forbidden hit fails outright, otherwise eighty per cent of checks pass. An optional --judge flag adds a Copilot grader rating faithfulness from zero to five, off by default. The builder harness was created after a real regression its README states plainly: when generalizing GitHub work, the builder preferred driving the browser with Playwright over the gh CLI, even though Scout runs on the user’s own machine where gh is installed and authenticated. Ten scenarios now pin the native capability, two of them the gh-versus-browser case directly, and that suite drove the fix in scout-catalogue.ts. A third harness scores the shape of the proposed skill plan, a fourth guards the on-device redaction pipeline, and that one’s real-image OCR variant self-skips with exit code zero when fonts, weights or network are missing.

5. 05

   One npm test, three workflows and a 79 KB compliance script

   Tests run on Node’s own runner with --experimental-transform-types plus a small hook that maps extensionless imports to .ts and swaps the single electron import for a headless stub, so the suite exercises real application source without a bundler. The npm test line names 43 files one by one, and the counts quoted in pull requests climb: 176 tests on 2026-08-23, 225 on 2026-09-15, 237 the next day and 267 on 2026-09-21. Continuous integration is three workflows. The Windows one runs on windows-latest and windows-11-arm, first proving install.ps1 rejects a mutable source reference, then performing a real commit-pinned source installation with the launch suppressed, checking lockfile portability, generating the license inventory, running the tests, building, packaging a native installer and verifying its architecture. Non-Windows does the same on macOS and Ubuntu, where macOS additionally prepares the complete corresponding-source bundle and asserts the platform’s libvips licence text is present. Portable preview builds run only when dispatched by hand, because the comment above them says an ordinary merge should not spend ten to fifteen minutes packaging. Around that sits a compliance apparatus larger than most of the application: scripts/compliance.mjs is 79 KB with a 39 KB test, package.json carries exact-version allowScripts approvals, and RELEASING.md requires the terms of the Copilot SDK, Electron, sharp-libvips, ONNX Runtime and Tesseract to be re-read before any of them moves.

6. 06

   Pinning the CLI, and rewriting the install instructions after a 404

   Two fixes show how much of this project is spent making somebody else’s tool predictable. The first pinned @github/copilot to 1.0.78 and runs it with --no-auto-update login --web-flow, because the previously bundled 1.0.71 rejects --web-flow and an auto-updated CLI in the cache made browser sign-in depend on whatever version a machine happened to hold. Its regression test is unusual: it drives the bundled CLI’s OAuth URL, PKCE challenge and loopback callback with a fake browser that denies authorization, so it never contacts GitHub and never signs anything in. The second is a Windows-only recovery path. On a managed machine where the public registry is blocked and the approved mirror needs no interactive sign-in, npm ci is retried exactly once against the approved registry — only when there is no explicit npm configuration and local device metadata supplies a Microsoft Entra tenant hint, and only for that child process, never by saving npm settings, weakening TLS, skipping integrity checks or changing script approvals. The documentation got the same treatment from the other direction: after users copied a README command containing <40-character-release-commit> unchanged and hit a 404, two pull requests replaced the copyable template commands with a pointer to the release page and three plain-language steps, on the reasoning that the people installing it are not necessarily developers.