[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-05-04

[github-actions[bot]]

Summary

• Date of test: 2026-05-04
• Pages visited:
  • `(localhost/redacted) (Home)
  • `(localhost/redacted) (Quick Start)
  • `(localhost/redacted) (CLI Commands)
• Overall impression: The home page makes a great first impression with a clear value proposition and good navigation. The Quick Start guide is well-structured with clear steps, though a few areas of assumed knowledge and terminology confusion could trip up a complete beginner.

---

🟢 What Worked Well

• Home page hero is clear — "Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions" immediately tells me what this tool does.
• Quick Start CTA is prominent — The "Quick Start with CLI" button is immediately visible above the fold.
• Step-by-step numbered structure — The Quick Start guide uses clear numbered steps (Step 1–4), making it easy to track progress.
• Prerequisites list is explicit — Lists GitHub CLI, AI account, and repository requirements up front — no hunting around.
• Video included — An embedded video in Quick Start is great for visual learners.
• Alternative installation options — The TIP callouts for curl installer and gh auth login workarounds are genuinely helpful.
• CLI Commands page has a "Most Common Commands" table — Excellent quick-reference at the top before diving into detail.

---

🔴 Critical Issues

1. "Frontmatter" is used without definition

In Step 4 (Customize), the Quick Start says:

"If you changed the frontmatter (the configuration block between the --- markers at the top of the file)..."

The parenthetical helps, but frontmatter is jargon that a complete beginner to markdown-based tooling won't know. The link to /gh-aw/reference/frontmatter/ is buried in the NOTE callout — a new user might not see it.

Suggestion: Introduce "frontmatter" with a brief inline explanation the first time it appears, or add a "Key concepts" callout box near the top of the Quick Start.

2. COPILOT_GITHUB_TOKEN vs GITHUB_TOKEN distinction is confusing

In Step 2, the wizard sets up:

"COPILOT_GITHUB_TOKEN (a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN)"

A beginner will not understand why there are two different tokens, why the default GITHUB_TOKEN isn't enough, or how to create a COPILOT_GITHUB_TOKEN. The link to /gh-aw/reference/auth/ is present but the in-line explanation is thin.

Suggestion: Add a short "Why two tokens?" explainer box, or a numbered sub-step saying "To create a COPILOT_GITHUB_TOKEN, go to GitHub Settings → Developer Settings → Personal Access Tokens → ..."

3. No mention of what happens if GitHub Actions is disabled

The prerequisites say "GitHub Actions enabled — Check in Settings → Actions" but don't say what to do if it's disabled or why it might be disabled (e.g., org policy). A first-timer who hits this wall will be stuck.

---

🟡 Confusing Areas

4. "lock file" concept unexplained

In Step 4, the Quick Start references a .lock.yml file:

"Adds the workflow file (.md) and its compiled lock file (.lock.yml) to .github/workflows/."

Why does a markdown workflow need a "compiled lock file"? This is never explained in the Quick Start. A new user might wonder if they need to edit both files, or only the .md.

Suggestion: Add one sentence: "The .lock.yml is auto-generated by gh aw compile — you never edit it directly."

5. add-wizard command format is cryptic

gh aw add-wizard githubnext/agentics/daily-repo-status

The <owner>/<repo>/<workflow-name> format is explained afterward, but the githubnext/agentics repository is a separate repo from the user's own repo. A beginner might think they're referencing something in their own repository.

Suggestion: Add a brief note: "This fetches a pre-built workflow template from the githubnext/agentics public repository — you don't need access to it."

6. CLI Commands page is overwhelming in length

The CLI Commands page is 39KB+ of content. While comprehensive, the "Most Common Commands" table at the top is helpful, but there's no visual separation or callout for beginners vs advanced users. Someone just getting started has no idea which sections they can skip.

Suggestion: Add a "Start here" callout highlighting just init, add-wizard, compile, and run as the 4 commands beginners need, and mark the rest as "advanced."

7. Early development warning is buried

The home page has this important note:

"i Note: GitHub Agentic Workflows is in early development and may change significantly. Using agentic workflows requires careful attention to security considerations..."

This is rendered as a blockquote after several paragraphs of feature descriptions. A new user might not see it until after they've already started following the guide.

Suggestion: Move this disclaimer to near the top of the home page hero section, or add a prominent "Status: Early Preview" badge.

---

📋 Recommendations (Prioritized)

Quick Wins

1. Add a "Key Concepts" glossary box at the top of Quick Start defining: workflow, frontmatter, lock file, engine. Even 4 bullet definitions would unblock many beginners.
2. Explain the two-token pattern with a visual diagram or callout in Step 2 — this is the single most confusing prerequisite.
3. Add one sentence explaining lock files at first mention.

Medium-Term

4. Add a "Beginner path" callout on the CLI Commands page pointing to just the 4 essential commands.
5. Raise visibility of the "early development" disclaimer on the home page.
6. Add a "What is frontmatter?" expandable section in the frontmatter reference and link to it prominently.

Longer-Term

7. Add a troubleshooting section to Quick Start for the top 3 most common failure modes (auth issues, Actions disabled, wrong token).
8. Add an animated diagram showing the flow: markdown → gh aw compile → lock file → GitHub Actions → AI agent → safe outputs.

---

Screenshots

📎 [home.png] — Home page (hero, navigation, key features)
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/1f03139e4177e369a840deb22b9be6f9b1c7e091e6ed982b101e01cb2e40cee1.png?raw=true

📎 [quick-start.png] — Quick Start guide (prerequisites, Step 1–2 visible)
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/f5e474bcc0443ed34e3e5835f96074d41e30af612624bcc202e67def8f0298f5.png?raw=true

📎 [cli-commands.png] — CLI Commands page (top of page, most common commands table)
https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/742a3864141fc617e1f56eb0dca6e6dafb382bd1d026d76aa51c18635cd8d261.png?raw=true

---

References: §25301690479

Warning

Firewall blocked 5 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

Generated by Documentation Noob Tester · ● 2.3M · ◷

• expires on May 5, 2026, 5:02 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-05-05T05:02:16.946Z.

Closed by Workflow