I build AI systems and write about where AI belongs in them.

The through-line across this work is a single question: which parts of a workflow should a model touch, and which parts should stay deterministic? The answer is usually "fewer than you'd think, with a gate on the way out." Detection logic that needs to be auditable stays rule-based. Generation gets a validation layer before a human ever sees it. Judgment stays with the person, and the model helps them apply it consistently.

Everything here is public so it can be copied, argued with, or torn apart. If something is useful, take it.

One exception: the RFP scanner discussed throughout is client work and the implementation stays private. It appears here as a case study covering the architecture and the reasoning behind each decision, which is the part worth sharing anyway. Happy to walk through any of it in more detail.

Read Decision 4 in the RFP scanner case study.

Hybrid BM25 plus dense embeddings, fused with Reciprocal Rank Fusion. Worth reading if you're deciding between keyword and semantic search: the answer is usually both, because they fail in different directions. Keyword search catches exact terms of art that embeddings blur together. Semantic search catches analogous cases that share no vocabulary. RRF combines the rankings without needing the two scoring scales to be comparable, which is the practical reason to prefer it over weighted score blending.

Also covers incremental re-embedding with state tracking, so you're not re-paying for a full corpus rebuild every run.

Start with ai-validator, which is public and installable. Decision 5 of the RFP scanner case study covers what it looks like wired into a production pipeline.

The core idea: a model's output is a draft, not a result. Between generation and delivery there should be a gate that strips narration, flags placeholders and self-references, detects claims about actions the model couldn't have taken, and scores confidence. Then route on the score rather than shipping everything.

The three-way routing (pass, review-required, reject) matters more than the detection itself. Binary pass/fail forces you to choose between blocking too much and shipping garbage.

Read the RFP scanner case study, particularly Decision 1 and the "Where AI is and is not used" table.

That system monitors 417 public entities and deliberately uses no model for detection and scoring. Every hit traces to specific keywords with specific weights. This was a choice, not a limitation: opportunity detection needs to be auditable, and a false negative is much easier to diagnose when the logic is inspectable. The model is confined to the one task where it genuinely outperforms a rule, which is synthesizing a first draft from analogous past work.

The general pattern: put the model where the task is genuinely generative and the output is reviewable, and keep it out of anything you'll need to explain to a skeptical stakeholder later.

Read GOVERNANCE.md from the RFP scanner.

Most governance material is written for enterprises with compliance teams. This is the version for a system one person maintains: which data goes to which third-party processor and why, retention, credential handling, what the audit trail captures and what it misses, and a three-tier change control model where the tiers are drawn by blast radius rather than by code complexity.

The tier that matters most is the middle one. Changing a keyword weight is not a code change, it is a business decision about what the firm pursues, and it should not be made unilaterally by whoever happens to maintain the repo.

It also classifies the model risk explicitly (assistive, human-approved, non-decisioning) and names the three changes that would break that classification. And it ends with six things it does not cover yet, because a governance document that claims full coverage is not one.

Start with pm-skills.

Most published agent skills are written for engineers: TDD, refactoring, debugging. This is an attempt at the other half, the ambiguous calls that have no test suite. Should this message go out at all? What can actually be cut from this scope? Is this worth escalating?

Split into decision skills (take a situation, return a judgment) and artifact skills (take inputs, produce a document). The decision skills are the interesting half and the harder thing to write well.

Each one is plain markdown, so it works as a Claude Skill, a Cursor rule, a Codex instruction, or just a doc you read before a meeting. No tool lock-in.

Start with ccar-f-prep.

Preparation material for Anthropic's Claude Certified Architect – Foundations exam: 104 scenario questions and 151 offline test cases across nine implementation drills. Every drill ships as NotImplementedError with the requirements in the docstring, and the tests grade what you write.

The exam itself is gated to the Claude Partner Network, so most people reading this can't sit it. The drills are still worth the time, because the thing being tested is not exam trivia — it's the set of mistakes that break agent systems in production. Terminating a loop by pattern-matching the model's prose instead of branching on stop_reason. Enforcing a refund ceiling in the system prompt and getting 94% compliance on an irreversible financial action. Returning "Operation failed" for every error, so the agent retries a business-rule rejection eleven times and gives up instantly on a 503. Passing findings between subagents as flowing prose and losing every citation on the way through.

The design decision I'd defend: context isolation is mechanical, not asserted. A mocked subagent is a function of its prompt string and nothing else, no ambient state, no back channel. So a coordinator that forgets to pass prior findings forward doesn't fail a style check; the subagent genuinely cannot see them and produces a report covering nothing. Same failure you'd get in production, same diagnosis.

16 of the exam's 30 objectives are fully machine-graded, 2 partly. The remaining 12 are written decision drills with an answer key, because there's no artefact to lint for "plan mode was the right call here."

The questions are original, written against the exam guide's published task statements.

Start with ai-pm-system.

Skills, ritual workflows, and a knowledge and memory layer for product management work. Built on Cursor. Useful mostly as a worked example of what a personal AI operating layer looks like when it's structured rather than ad hoc.

Things here that are meant to be lifted directly:

The two harnesses are the part I'd expect to travel furthest outside their original context. mock_claude.py scripts model turns with real stop_reason values, so you can assert on loop control flow. mock_task.py scripts subagents, so you can assert on orchestration shape: how many turns the spawns took, which scope each agent received, what was actually in the synthesis prompt. Single files, no dependencies, no network.

Kept here on purpose. A guide that only shows the finished version isn't much of a guide.

• Runtime validation and evaluation are two different disciplines. ai-validator catches bad output as it happens, which is the right tool for protecting a reviewer's time. Knowing whether a change to prompts or retrieval actually improved the system is a separate problem that needs offline eval against a golden set. Building the first one taught me they're not substitutes, and that's the next thing I'm adding.
• Integration surface area is an ongoing operating cost, not a one-time build. Nine adapters against portals that can change their HTML without notice means the scrapers deserve monitoring proportional to how quietly they can fail. Designing for the day an integration breaks matters as much as designing for the day it works.
• A human in the loop and a loop that learns are different things. Reviewers accept, edit, or discard drafts, and that signal is the highest-quality training data the system produces. Closing that feedback path back into retrieval and prompting is the largest remaining opportunity in the design.
• Tests that check how you wrote something are brittle. Tests that make the wrong design not work are not. Writing the drills in ccar-f-prep is where this got concrete. The first instinct is to assert on structure: did you branch on the right field, did you call the right method. That breaks on any reasonable alternative implementation and teaches nothing. The version that works is to build the constraint into the environment, so the wrong approach produces a wrong answer for the same reason it would in production. Same instinct as preferring a deterministic gate to a prompt instruction — make the invalid path impossible rather than discouraged.

Collected from building the things above. Offered as opinions, not rules.

1. Put the model where the task is generative, not where it's decisive. Drafting, summarizing, and synthesizing are good fits. Scoring, routing, and gating usually shouldn't be a model call if a rule will do, because you'll have to defend the output.
2. Auditability beats accuracy for anything a stakeholder will question. A rule-based system that's 85% accurate and fully explainable often beats a model at 92% that no one can interrogate.
3. Every generative step needs a gate before a human sees it. Not a review step, a programmatic gate. Reviewers stop reading carefully after the fourth clean draft.
4. Retrieval quality is the ceiling on output quality. Prompt tuning cannot fix a bad top-k. Spend the time on retrieval first.
5. Design for the skeptical user, not the enthusiastic one. Watermarks, disclosure, and explicit "verify this" instructions are adoption features, not compliance overhead.
6. Anything that must hold every time belongs in code, not in the prompt. Prompting produces probabilistic compliance. At any non-zero failure rate on an irreversible action — money moving, data deleted, a message sent — the remainder is the whole problem. A prerequisite gate makes the invalid sequence impossible rather than merely unlikely.

A roadmap, in rough priority order. Most of it comes from a gap analysis I ran against what AI product roles are actually asking for, and the honest finding was that a good share of the work is writing down decisions I already made rather than building new things.

Recently shipped

• Unit economics for the RFP scanner. Done. About $0.074 per generated draft and roughly $11.57 a year against 234 analyst hours displaced, with the model published alongside it. The number matters less than what it implies: at this volume inference is a rounding error and integration maintenance is the real cost line, which is the opposite of what the token math alone suggests.
• A governance document. Done, and published here. Data flows to third-party processors, retention, credential handling, tiered change control, and an explicit model risk classification. The most useful part turned out to be the gap list at the end.
• Prep material for the Claude architect certification. Published here. 104 original scenario questions weighted to the published exam blueprint, plus 151 offline tests across nine implementation drills. The reusable part is the two mock harnesses, which let you test an agentic loop or a coordinator and its subagents with no API key and no network.

Near term

• Decisions currently living only in my head. Why RAG over fine-tuning on the proposal corpus. Why nine custom adapters over a procurement data vendor. Why Voyage over other embedding providers. Chunking strategy, and how the context budget is split between opportunity detail, retrieved sections, and biographies. Each is a paragraph, and for anyone evaluating the architecture the reasoning matters more than the result.

Building

• An MCP server over the adapter layer. The nine adapters already normalize REST, OData, RSS, SSR JSON, HTML scraping, and headless browser sources behind one interface, which is close to the problem the Model Context Protocol standardizes. Exposing them as tools (search_opportunities, get_agenda_text, retrieve_similar_proposals) would let an agent work against 417 jurisdictions through a standard surface instead of a fixed pipeline.
• An agentic loop, where it earns its place. Right now the pipeline is fixed: scan, score, draft, validate, send. A version where an agent decides which portals to query, evaluates relevance, and chooses whether to draft would be a genuine test of principle #1 above. I'm curious whether agency actually improves outcomes here or just adds variance to something that works. Either answer is worth publishing.
• Offline evaluation. A golden set of opportunities with expected retrievals, regression tests that run when prompts or scoring weights change, and an A/B comparison of BM25 alone against semantic alone against the RRF hybrid. That last one would retroactively prove or disprove the hybrid architecture choice, which I currently defend on reasoning rather than evidence.
• Closing the review loop. Reviewer accept, edit, and discard signals are the highest-quality feedback the system produces, and none of it currently reaches retrieval or prompting.

In progress

• pm-skills. Filling in the worked examples and the "what to watch for" sections, which are the parts that can only come from having made the call. Expanding coverage across the product lifecycle: discovery, definition, ship coordination, post-launch, stakeholder work.
• ccar-f-prep. Four objectives are still covered only by ungraded exercises. The tractable one is a linter for tool-description quality, checking that a description declares its input format, gives an example query, and states its boundary against similar tools. Same class of check as the config linter already in there, and it's the failure mode behind most tool-selection problems.

Genuinely open to it, and specifically on these:

• Disagreement with the decision frameworks in pm-skills. If a framework there doesn't match how you've seen a strong PM actually make that call, that's the most useful thing you could tell me. Open an issue.
• Adapters for jurisdictions or platforms I don't cover. The pattern is documented and adding a city is usually a config entry plus a URL check.
• Anyone who has built offline eval for a RAG pipeline. This is the area where I'd learn the most from someone who has already done it.
• MCP server design review. Particularly on where to draw tool boundaries, which seems like the decision that's hardest to change later.
• Forks of pm-skills with your own material in the placeholders. The frameworks are portable; the experience in them shouldn't be mine.
• Arguments with the drills in ccar-f-prep. Several encode a judgment call rather than a fact: where the line sits between a genuine source conflict and a time series, when a subagent should recover locally versus propagate upward. If you'd have designed one differently, the reasoning is more interesting than the test.

Issues and pull requests both work. Or email me at ian@ihk.ca and skip the ceremony.

Terms used above, defined the way I mean them here. Some of these are contested, so this is my usage, not a standard.

Adapter - A small module that translates one external source into a shared internal format. The RFP scanner uses nine, covering REST APIs, OData, RSS, server-rendered JSON, HTML scraping, and a headless browser, all returning the same structure downstream.

Agent - A system where a model decides what to do next, including which tools to call and when to stop, rather than executing a sequence a developer wrote. The distinction that matters: a pipeline with a model call inside it is not an agent, because the control flow is fixed.

Agentic loop - The repeated cycle of a model observing state, choosing a tool, seeing the result, and deciding again. The thing that makes something an agent rather than a script.

BM25 - A ranking function for keyword search. Scores documents by term frequency, adjusted for how common the term is and how long the document is. Good at exact terms of art, blind to synonyms.

Chunking - Splitting source documents into retrievable pieces before embedding them. Chunk boundaries have more effect on retrieval quality than most people expect, since a chunk that splits an argument in half retrieves poorly no matter how good the embedding model is.

Confidence score - A number a validation layer assigns to generated output, used for routing rather than for display. In ai-validator it drives the pass, review, or reject decision.

Context window - The total amount of text a model can consider at once, including the prompt, retrieved material, and its own output. Treat it as a budget to allocate deliberately rather than a limit to bump into.

Deterministic - Same input, same output, every time, with no model involved. Used here as a design property worth choosing on purpose, mainly for anything that needs to be explained or audited later.

Embeddings (dense retrieval) - Numeric representations of text where similar meanings land near each other. Catches analogous cases that share no vocabulary. The complement to BM25, and the reason hybrid retrieval works.

Fine-tuning - Further training a model on your own data so the behavior is baked into the weights. Distinct from RAG, where the data is retrieved and supplied at request time instead. For a corpus that changes often and where you need to trace which source produced a claim, RAG is usually the better fit.

Golden set - A fixed set of inputs with known-good expected outputs, used to detect whether a change improved or degraded the system. The core of offline evaluation.

Hallucination - Model output that is fluent and false. The variety worth gating on is the model claiming to have taken an action it could not have taken, such as reviewing a file it was never given, because it signals fabricated sourcing rather than a simple factual slip.

Hub-and-spoke - A multi-agent topology where every message between specialists routes through a coordinator rather than peer to peer. Costs a hop; buys one place to handle errors, route information, and observe what happened. Direct chains are faster and considerably harder to debug when one agent's malformed output silently corrupts the next.

Human in the loop (HITL) - A design where a person reviews or approves output before it has effect. Worth distinguishing from a loop that learns, where the reviewer's decisions feed back into the system.

MCP (Model Context Protocol) - An open standard, introduced by Anthropic in late 2024, for connecting models and agents to external tools and data sources through a common interface rather than bespoke integrations. Now supported across multiple vendors.

Offline evaluation - Measuring whether a change to prompts, retrieval, or models made the system better, run against a fixed dataset before shipping. Different from runtime validation, which catches bad output as it happens. Both are needed and they answer different questions.

Orchestration - Coordinating multiple steps, models, or agents, including retries, branching, and state between calls. A weekly cron job is scheduling, not orchestration.

RAG (Retrieval Augmented Generation) - Fetching relevant material at request time and supplying it to the model as context, so answers are grounded in a specific corpus rather than in whatever the model absorbed during training.

RRF (Reciprocal Rank Fusion) - A method for merging two or more ranked lists using only positions, not scores. Its practical advantage is that the input rankings never have to be on comparable scales, which is exactly the problem when combining BM25 with embedding similarity.

Semantic layer - A curated, queryable representation of an organization's knowledge that sits between raw documents and whatever consumes them. The RFP scanner builds a small one over a firm's past proposals.

Skill - A markdown file describing how to approach a class of task, loaded by an AI tool when relevant. In pm-skills these are split into decision skills, which return a judgment, and artifact skills, which return a document.

stop_reason - The field in an API response saying why the model stopped generating: it wants a tool result, it finished its turn, it hit a token limit. The correct basis for agentic loop control flow, and the thing people reach past when they start parsing the model's prose for a closing phrase instead.

Subagent - A model instance spawned by a coordinator to handle one part of a task, running with its own isolated context. The consequence people miss: it inherits nothing. Anything it needs has to be in the prompt it was handed, which is where citations and prior findings quietly get lost.

Top-k - How many retrieved chunks get passed to the model. Tuning this is a tradeoff between recall and diluting the context with marginal material.

Not exhaustive. These are the things that actually changed how I build, plus a few I keep returning to. Links move, so search the title if one breaks.

Retrieval

• Cormack, Clarke, and Büttcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods (SIGIR 2009). Two pages, and the method underneath most modern hybrid search. Worth reading in the original because the idea is genuinely that simple.
• Manning, Raghavan, and Schütze, Introduction to Information Retrieval (Cambridge, 2008). Free online. Predates the current wave entirely, which is the point: BM25, scoring, and fusion are not new problems.

Agents and protocols

• The Model Context Protocol specification and docs at modelcontextprotocol.io, with the schema and SDKs at github.com/modelcontextprotocol. Read the security and trust section even if you only plan to consume MCP servers rather than build one.
• Anthropic's engineering writing on building effective agents. The useful argument is that most problems labeled as needing an agent are better served by a well-structured workflow, which is close to principle #1 above.
• The Claude Certified Architect – Foundations exam guide, from Anthropic's Partner Academy. Read as a document rather than as exam prep, it is a compact statement of what Anthropic considers competent agent architecture: thirty task statements across orchestration, tool design, context management, and reliability. The scenarios are more instructive than the format suggests.

Skills and AI-assisted engineering practice

• mattpocock/skills. The framing that agent coding failures map onto ordinary software engineering failures, misalignment, jargon gaps, missing feedback loops, and design entropy, rather than being novel AI problems.
• mattpocock/dictionary-of-ai-coding. Good companion to the above. Builds a small set of primitives, statelessness, context versus context window, parametric versus contextual knowledge, then uses them consistently to explain failure modes that otherwise look mysterious.
• obra/superpowers. A stricter take: an enforced multi-phase workflow rather than optional discipline. Worth reading as the opposite end of the control versus flexibility tradeoff even if you would never adopt it as written.
• anthropics/skills. The first-party reference for the SKILL.md format itself.

Evaluation

• Chip Huyen's writing on AI engineering and evaluation. The clearest treatment I have found of why evaluation is the hard part and why it stays hard.
• Hamel Husain's material on LLM evals, particularly on building an evaluation set from real failures rather than imagined ones. This is the reading behind the offline eval item in the roadmap above.

Design and architecture

• Ousterhout, A Philosophy of Software Design. Deep modules and simple interfaces. Increasingly relevant as generated code accelerates how fast a codebase can accumulate shallow abstractions.

Product management, 15 years, including Apple, Amazon, GoPro, Intuit, and McAfee. Currently a Master's student in Urban Planning at San Jose State, which is why several of these projects point at California municipal data.

Reach me at ian@ihk.ca.