ui screenshot: switch to Windows.Graphics.Capture; add --focus flag#522
Open
nmetulev wants to merge 6 commits into
Open
ui screenshot: switch to Windows.Graphics.Capture; add --focus flag#522nmetulev wants to merge 6 commits into
nmetulev wants to merge 6 commits into
Conversation
Default capture path now uses Windows.Graphics.Capture (WGC), reading the actual DWM-composited window surface. This preserves rounded corners with true alpha, removes the black borders previously caused by including the DWM shadow/resize padding inside GetWindowRect, and works while the window is occluded. PrintWindow is retained as a fallback when WGC is unsupported or fails. Adds --focus to bring the target window to the foreground before capture, useful for matching what the user is currently looking at without switching to --capture-screen semantics. --capture-screen now implies --focus and the foregrounding is done exactly once. Implementation: - New WgcCapture helper using free-threaded Direct3D11CaptureFramePool, B8G8R8A8 surface, copy-to-staging + Map for AOT-safe BGRA32 readback. COM interop via [GeneratedComInterface] / [LibraryImport] to comply with DisableRuntimeMarshalling and full trimming. - Element-crop origin now derived from DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS) so element bounding rectangles align with the WGC frame's visible-only size. - Cancellation (Ctrl+C) propagates through the WGC catch instead of falling back to PrintWindow. - WgcCapture is self-contained (its own IsBlankCapture); no internal coupling back into UiAutomationService. - PrintWindow blank-frame retry extracted to CaptureFromWindowWithBlankRetry shared by the WGC fallback and the no-WGC path. - Added D3D11/DXGI/DWM bindings to NativeMethods.txt and rooted WGC ABI types in TrimRoots.xml. - E2E UI test script exercises --focus. - Regenerated docs/cli-schema.json, ui-automation skill SKILL.md, and src/winapp-npm/src/winapp-commands.ts. Updated docs/ui-automation.md, docs/fragments/skills/winapp-cli/ui-automation.md, and .github/plugin/agents/winapp.agent.md to describe the new precedence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates winapp ui screenshot to use Windows.Graphics.Capture (WGC) by default (with PrintWindow fallback), and introduces a new --focus flag to optionally foreground the target window prior to capture. It also refreshes generated schema/wrappers and updates docs + E2E coverage to reflect the new behavior.
Changes:
- Implement WGC-based per-window capture with blank-frame handling, composited-surface alignment, and PrintWindow fallback.
- Add
--focus(and make--capture-screenimply focus) across CLI options/handlers and documentation. - Regenerate CLI schema + npm command wrappers; extend E2E script to validate
screenshot --focus.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/winapp-npm/src/winapp-commands.ts | Regenerated TS wrappers; adds ui screenshot --focus and new run fields. |
| src/winapp-CLI/WinApp.Cli/TrimRoots.xml | Roots WinRT ABI types to keep WGC working under trimming/AOT. |
| src/winapp-CLI/WinApp.Cli/Services/WgcCapture.cs | New helper implementing WGC capture and D3D11 readback. |
| src/winapp-CLI/WinApp.Cli/Services/UiAutomationService.Screenshot.cs | Switch default screenshot path to WGC; add focus/foregrounding logic; align crop origin. |
| src/winapp-CLI/WinApp.Cli/Services/IUiAutomationService.cs | Update screenshot API to include focus parameter. |
| src/winapp-CLI/WinApp.Cli/NativeMethods.txt | Add CsWin32 bindings needed for D3D11/DXGI/DWM interop. |
| src/winapp-CLI/WinApp.Cli/Commands/UiScreenshotCommand.cs | Plumb --focus option through the screenshot command. |
| src/winapp-CLI/WinApp.Cli/Commands/SharedUiOptions.cs | Define --focus and update --capture-screen description/semantics. |
| src/winapp-CLI/WinApp.Cli.Tests/UiSessionServiceTests.cs | Update test stub to match new screenshot method signature. |
| src/winapp-CLI/WinApp.Cli.Tests/FakeUiServices.cs | Update fake UI service to match new screenshot method signature. |
| scripts/test-e2e-winui-ui.ps1 | Add E2E coverage for ui screenshot --focus and PNG validation. |
| docs/ui-automation.md | Document WGC default behavior and new --focus usage. |
| docs/fragments/skills/winapp-cli/ui-automation.md | Update skill fragment with --focus guidance and new semantics. |
| docs/cli-schema.json | Regenerate schema for new screenshot options and updated descriptions. |
| .github/plugin/skills/winapp-cli/ui-automation/SKILL.md | Update plugin skill doc to include --focus and revised capture guidance. |
| .github/plugin/agents/winapp.agent.md | Update agent guidance to describe WGC default and --focus / --capture-screen. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Build Metrics ReportBinary Sizes
Test Results✅ 952 passed, 1 skipped out of 953 tests in 476.9s (+1.5s vs. baseline) Test Coverage❌ 16.4% line coverage, 34.2% branch coverage · CLI Startup Time40ms median (x64, Updated 2026-05-07 21:22:50 UTC · commit |
… constant) - Dispose ID3D11Device, ID3D11DeviceContext, capturedTexture, and stagingTexture in WgcCapture so repeated screenshots don't leak GPU/COM resources. - Replace blocking Thread.Sleep in async ScreenshotAsync foregrounding step with await Task.Delay(.., ct) so cancellation is honored promptly. - Replace D3D11CreateDevice magic SDKVersion: 7 with a named D3D11_SDK_VERSION constant. Skipped: TS wrapper appArgs feedback (pre-existing command-generator issue, not introduced by this PR; tracked separately). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
Addressed Copilot review feedback in a8a9689:
Not addressed: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Default
winapp ui screenshotcapture path now uses Windows.Graphics.Capture (WGC), reading the actual DWM-composited window surface. This:GetWindowRectincluding the DWM shadow / resize padding.PrintWindow is retained as a fallback when WGC is unsupported or fails.
--capture-screenstill does screen-DC BitBlt for popup overlays not owned by the target.Current behavior:
New behavior:
New flag:
--focusBrings the target window to the foreground before capture, useful when you want the screenshot to match what the user is currently looking at without switching to
--capture-screensemantics.--capture-screennow implies--focusand the foregrounding step happens exactly once.Implementation notes
WgcCapturehelper using a free-threadedDirect3D11CaptureFramePool, B8G8R8A8 surface, copy-to-staging + Map for AOT-safe BGRA32 readback.[GeneratedComInterface]/[LibraryImport]to comply withDisableRuntimeMarshallingand full trimming.DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS)so element bounding rectangles align with the WGC frame's visible-only size.WgcCaptureis self-contained (its ownIsBlankCapture); no internal coupling back intoUiAutomationService.CaptureFromWindowWithBlankRetryshared by the WGC fallback and the no-WGC path.FromAbi/ConvertToManagedsofinallyonly releases on error paths.NativeMethods.txt; rooted WGC ABI types inTrimRoots.xml.CI
scripts/test-e2e-winui-ui.ps1now exercisesscreenshot --focusand asserts the PNG. The existinge2e-test-uijob onwindows-latestcovers the new default WGC path end-to-end.Docs / generated
docs/ui-automation.md,docs/fragments/skills/winapp-cli/ui-automation.md,.github/plugin/agents/winapp.agent.mdto mention--focusand describe the new precedence.docs/cli-schema.json, theui-automationSKILL.md, andsrc/winapp-npm/src/winapp-commands.ts.