Test Twin Forge live, rewrite build pages, fix citations and fictitious names - #2
Merged
Conversation
TESTED END TO END on the tenant. It works - triggers on the phrase, builds a 5-step plan, pulls sent mail + calendar via Work IQ, computes revealed behavior, asks behavioral questions one at a time. THREE BUGS FOUND AND FIXED: 1. .zip upload SILENTLY FAILS - dialog closes, no skill, no error. Only .md works. Documented everywhere. 2. references/ companion files do NOT travel with a .md upload. The agent reported 'the reference files dont exist'. Fixed by making SKILL.md fully self-contained - schemas and question bank inlined, references/ folder deleted. 3. Path B persona pack is not in the Cowork workspace, so it must be attached to the session. Documented. Also: re-uploading a deleted skill creates 'twin-forge-1' - the OneDrive file survives deletion. Added tested screenshot + coach playbook 'Tooling traps' table.
…-Imagineer-Hack into scenario-1-digital-twin
Read the old page cold as a participant: six required round-trips to other pages, repo paths (/Allfiles/...) in body text that 404 on the site, jargon in the first ten words (altitude, assembly map, snap these blocks, Step 0), and the three files introduced before any reason to care. Changes: - Zero required clicks. Everything needed is inline; bricks demoted to optional 'Full guide' links underneath each step. - Download buttons sit inline at the step that needs them, not on a separate Downloads page. - Opens with the problem in two sentences, then what you walk out with, then the files - not the file list first. - Removed every repo path from participant-facing text. - Steps numbered from 1. 'Assembly map / snap these blocks' -> 'Install Twin Forge', 'Make it shorter', 'Give it your voice', 'Give it judgment'. - Filenames as plain words (soul / voice / revealed) in the intro table. - Values-vs-rules comparison table replaces the abstract explanation. - Added a 'Stuck?' table covering the 4 failure modes found in live testing plus the 2 most likely content mistakes.
All 8 papers verified live against the arXiv API before linking - title, authors and abstract checked, not just that the URL resolves. Tracked down the one citation that was previously unverifiable: the 23.5x style-matching claim is Jemama & Kumar, 'How Well Do LLMs Imitate Human Writing Style?' (arXiv:2509.24930). The abstract states it verbatim: 'few-shot prompting yields up to 23.5x higher style-matching accuracy than zero-shot'. Removed the HyPerAlign reference that could not be confirmed. - NEW docs/resources/research.md - one canonical page explaining why the scenario is designed this way, with every source linked and the honest caveats stated (style transfers but unpredictability doesn't; the file learns but the model doesn't) - Inline links added in base build page, coach playbook, interview script, advanced-guardrail brick - Interview script now cites the 74% baseline it references - Wired into nav under Resources
arXiv is a preprint server - no peer review. I had been presenting eight preprints as equivalent evidence and hanging specific numbers on them. Checked every one against the arXiv API for a declared venue: only ONE states peer review (Lost in the Middle -> TACL). REMOVED every statistic sourced to a preprint: - the 23.5x style-matching figure (IEEE UEMCON preprint) - the 74/82/83/86% persona-accuracy table (no declared venue) - 99.9% style agreement, perplexity 29.5 vs 15.2 - 'scored 60% to 87%' presented as if it were measurement The underlying advice survives without them and reads better - 'showing beats describing' needs no statistic. KEPT, properly labelled: - Lost in the Middle - peer reviewed, Transactions of the ACL - Constitutional AI - now linked to Anthropic's own research site rather than the arXiv preprint Everything else moved to a 'Further reading' table explicitly marked NOT peer reviewed, with no claims attached. Research page renamed 'Where This Comes From' and restructured into what we stand behind / what we observed ourselves / further reading, plus an honest-limits section.
All invented names replaced with entries from the approved people list, and all companies/domains with entries from the approved fictitious company list. 22 files updated. People: Marcus Webb -> Avery Washington (gender neutral) Priya Raghavan -> Victoria Burke Dan Kowalski -> Graham Barnes Tessa Lin -> Maya Robinson Jordan Beck -> Caleb Miller Amara Osei -> Amara White Ravi Chandra -> Miguel Reyes Helen Park -> Serena Davis Companies: Northwind Traders kept - already on the approved list Contoso kept - already on the approved list Fenwick & Co -> First Up Consultants (firstupconsultants.com) Acme Industrial -> Tailwind Traders Litware -> Lakeshore Retail Emails now use first-name-only usernames per the guidance (avery@northwindtraders.com, not avery.washington@...). The invented fenwickco.example domain is gone. Product codename Atlas -> Lantern. Atlas collides with real shipping products, and the guidance covers products as well as companies. Avery Washington is gender neutral on the approved list, so the persona's pronouns are now they/them. Graham and Miguel stay he/him - both are listed male. Also switched the build anecdote to they/them so it doesn't identify anyone. Verified after the rename: all 96 calendar rows parse and all four planted contradictions still compute (23.5h launch vs 26.8h status; 12 of 12 customer calls owned by direct reports; strategy block declined/tentative/declined/declined; organizer on 20 of 96).
…participant nav Advanced build page rewritten to match the Base page shape: - opens with the problem, and the second problem specific to this track (judgment is an argument, not one voice - which is the honest reason to build agents instead of a skill) - download cards inline, no trip to a separate page - 'Assembly map / snap these blocks' -> plain steps: Split the work, Write the spec, Ground it, Write the agents, Expose it as a server, Add the guardrail, The twist, Let the critic patch it - every clock-eater promoted from a footnote to a warning where it will actually bite, including the Agent-mode check BEFORE anything else - values-vs-tiebreakers comparison table - 'Stuck?' table with 7 real failure modes - honest limit stated: the file learns, the model doesn't Also: - consistent naming everywhere: Scenario 1 - The Digital Twin, with the same emoji in nav and PathPicker - Facilitators removed from top nav. Participants shouldn't be reading role cards and the coach playbook mid-build. Still reachable from Resources and its own sidebar.
The IA was four layers deep before anyone reached a step they could
follow: Scenarios -> Levels -> Building Blocks -> Build page. Building
Blocks in particular was stale, left over from before the build pages
became self-contained.
- Top nav is now Home / How the Hack Works / Start Building / Submit /
Resources. 'Start Building' goes DIRECTLY to a build page.
- Levels, Building Blocks and Scenarios removed from top nav. Pages kept
so nothing 404s; old level pages now point at the build pages.
- Building Blocks renamed 'Reference guides' and moved under Resources,
with a note saying you probably don't need them.
- Brick content INLINED into the build pages as collapsible ::: details
blocks, at the step where it's relevant. Zero navigation - the detail
opens in place and the page stays scannable when closed.
Cowork: how skills work / getting good samples / schedules+triggers
Code: agent file anatomy / building the server / the guardrail
- All 'Full guide' round-trip links removed from build pages.
- External download links now open in a new tab.
- Levels page collapsed from four pages to one comparison table with
direct Start links.
- Home cards point at real destinations instead of the old maze.
Previous commit hid the guides under Resources and told people they probably didn't need them. That was wrong - they're good, detailed, and especially useful on the Base path. - 'Guides' is back in the top nav as its own item - Every build page has a callout near the top pointing at them - Each expandable block ends with a direct link to its full guide, opening in a NEW TAB so nobody loses their place mid-build - Guides index rewritten: no more 'you probably don't need these'. Now a 'use it when' table so people can tell at a glance which one answers their question - Renamed 'Reference guides' -> 'Step-by-step guides', which is what they actually are Net effect: quick version expands in place, full version is one visible click away in a new tab, and both are obvious from the build page.
The intro claimed AI 'knows everything about your company and nothing about you'. That is false - Copilot has memory and Work IQ has a personalization layer. Replaced with the claim that actually holds: Copilot personalizes by INFERENCE. You cannot inspect what it concluded, you cannot correct it directly, and it does not follow you between tools. What it does not have is what you have explicitly decided. That is defensible and it is still the reason to do the scenario. Also cut: - 'It can't write YOUR email' / 'the last mile is you' - metaphor - 'the productivity gain evaporates' - metaphor - 'the quirks are the whole signal' -> names the actual specifics - 'steal shamelessly', 'this feels wrong', 'the moment that lands', 'the best demo isn't the most complex one' - 'a sharp Cowork build beats a rushed code one' - unfounded comparison - 'villain's complaint' section headings - 'A value tells an agent nothing' -> 'A value cannot be applied. A rule can.' REMOVED the 'Where This Comes From' page and every link to it. how-it-works updated to match the flattened navigation.
Participants were being sent to github.com to click Raw and Save As, or to clone the whole Team Week repo. Both are wrong for a timed hack. - scripts/pack-downloads.mjs zips Allfiles into docs/public/downloads/ and runs automatically before docs:dev and docs:build, so downloads can never drift from source. No dependencies - uses Compress-Archive on Windows, zip elsewhere, both present in CI. - Download cards now point at same-origin files with the HTML download attribute, so they save directly instead of navigating. Twin Forge saves as SKILL.md, ready to upload into Cowork as-is. - git clone replaced with: download the zip, unzip, open in VS Code. Plus an optional one-liner (PowerShell and curl) that fetches, unzips and launches VS Code in one go. - NEW digital-twin.code-workspace in the starter - opens with agent mode enabled, parent-repo customizations on, pycache hidden, and the Copilot + Python extensions recommended. - docs/public/downloads/ gitignored since it's generated. Verified all three files return HTTP 200 and the zips contain a proper top-level folder.
The harness existed in the starter and the run of show scheduled 'Seal
your answers' at 0:10 and 'Reveal' at 1:50 - but neither build page
mentioned it. Participants following the build page would never have
known to run it, and the files looked like dead code.
Both tracks now open with it and close with it:
🟣 Code Step 1 python test/take_test.py
Step 10 python test/compare.py
🟢 Cowork Step 1 download The Twin Test, answer, set aside
Step 11 ask the twin the same 15, fill in the table
- NEW Allfiles/twin-test/twin-test.md - a printable version for Base,
since that track has no repo and no Python. Same 15 dilemmas, with a
fill-in table and the end-of-session prompt included.
- Packaged as a download alongside the others.
- Starter README now leads with the harness instead of burying it in a
quickstart code block, and explains what a mismatch means.
- AGENTS.md describes what test/ is for, so Copilot can explain it.
- Both demo checklists gained: 'one dilemma your twin got wrong, and
the rule you'd add because of it'.
Framing is consistent everywhere: no score, no leaderboard. Mismatches
are the output, and each one names the soul.md field to fix - which is
exactly what compare.py already printed.
The page still said 'TODO - the rubric' and was linked in the sidebar, which contradicts the rest of the site. Replaced with a note explaining the reveal mechanic instead, and unlinked it from the nav.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1. Five commits: a live-tenant test of Twin Forge, a rewrite of both build pages for first-time readers, a citation cleanup, and CELA-approved fictitious names.
Tested Twin Forge on a live tenant — it works, and it had three bugs
Uploaded it, triggered it, and ran the interview as a participant. It builds a 5-step plan, pulls sent mail and calendar through Work IQ, computes revealed behavior, and asks behavioral questions one at a time. Three things were broken:
.zipupload fails silently — dialog closes, no skill, no error. Only.mdworks.references/files don't travel with a.mdupload. The agent reported "the reference files don't exist." Fixed by makingSKILL.mdfully self-contained.All three are now documented on the build page and in a new coach-playbook "Tooling traps" table.
Rewrote both build pages for someone who's never seen this
Read the old pages cold: six required round-trips to other pages, repo paths in body text that 404 on the site, and jargon in the first ten words ("altitude", "assembly map", "snap these blocks", "Step 0").
Only peer-reviewed or lab-published citations
arXiv is a preprint server. I'd been presenting eight preprints as equivalent evidence with hard numbers attached. Checked every one for a declared venue — only one states peer review.
Removed: the 23.5× style-matching figure, the 74/82/83/86% persona-accuracy table, and a "60% to 87%" anecdote written as if it were measurement. The advice reads better without them.
Kept, labelled: Lost in the Middle (peer reviewed, TACL) and Constitutional AI (relinked to Anthropic's own research site). Everything else moved to a "Further reading" table marked not peer reviewed, with no claims attached.
New page: Where This Comes From — what we stand behind, what we observed ourselves, further reading, and the honest limits.
CELA-approved fictitious names
Every invented name replaced from the approved lists.
Northwind TradersandContosowere already approved and stayed; the rest weren't.Atlas→Lantern— Atlas collides with real shipping productsVerified after the rename: all 96 calendar rows parse and all four planted contradictions still compute.
Also
Facilitator kit removed from the top nav — participants shouldn't be reading role cards mid-build. Still reachable from Resources.
npm run docs:buildpasses. Python compiles. Swept clean for personal data, credentials, and old names.