Architecture Review Boards for GenAI

The Architecture Review Board exists to answer one question before a system reaches production: can we operate this safely? For deterministic software, the ARB has a mature playbook — data flows, failure modes, security boundaries, scalability, cost. That playbook has worked for decades because the systems under review share a property: given the same input, they produce the same output, and that output is knowable in advance.

Generative AI breaks that property. And because it breaks it quietly — the system still compiles, still passes its tests, still demos beautifully — many organizations let AI workloads route around the ARB entirely, or wave them through on a deterministic checklist that no longer fits. Both paths end in the same place: an ungoverned model making consequential decisions in production. The answer is neither to block AI nor to exempt it. It is to extend the board's mandate to cover what is genuinely new.

Three assumptions GenAI invalidates

1. Determinism. The traditional ARB assumes that if a system is correct in test, it is correct in production. A model's behavior is distributional: it can be right 99% of the time and confidently wrong on the case that matters. "It passed QA" is no longer a sufficient claim.

2. Stable behavior over time. Deterministic systems change only when their code changes. A model-backed system can change behavior when the provider updates the model underneath it — no code change, no deploy, no notification. The thing the board approved may not be the thing running next quarter.

3. Inspectable logic. An ARB can read a decision tree. It cannot read 70 billion weights. The reasoning behind any given output is not available by inspecting the artifact. Without deliberate design, the system is a black box to the very people accountable for it.

Extending the board's mandate

Each invalidated assumption maps to a new review dimension. A GenAI-ready ARB adds the following to its existing remit:

Grounding and evidence

The first question for any AI system entering the data plane: can it show its work? Every consequential output must be tied to retrieved evidence and carry its provenance — model, prompt version, confidence. A system that cannot produce an evidence trail for its judgments is not ready, regardless of how good its outputs look. (We treat this as a hard gate; see Grounding Beats Fluency.)

Human-in-the-loop placement

Not every decision needs a human; some cannot tolerate full automation. The board's job is to locate the line. High-confidence, low-stakes outputs may auto-confirm. Low-confidence or high-stakes outputs route to a review queue. The placement of that threshold — and the auditability of what crosses it — is an architectural decision the ARB must own, not an implementation detail left to the team.

Version pinning and drift

The board should require that model IDs, prompt templates, and embedding versions are pinned and versioned as code — not floating references that change under the system. Paired with this is a drift monitor: a continuous evaluation against a labeled golden corpus that detects when behavior shifts, whether from a provider update or a prompt change. Approval is granted to a pinned, measured configuration, not to "the model, whatever it becomes."

Evaluation as a release gate

Deterministic systems gate releases on tests. AI systems gate on evaluation metrics against a golden corpus — recall, precision, exact-match on structured fields, agreement on classifications. A prompt or model change that regresses these metrics beyond a configured tolerance does not ship. This turns quality from a subjective review-meeting debate into an objective, automatable gate.

A practical intake checklist

For any AI system arriving at review, the board should be able to get a clear answer to each of these:

• What decisions does the model make, and what is the blast radius of each being wrong?
• Is every consequential output grounded in retrievable evidence, with provenance persisted as data?
• Where is the human-in-the-loop boundary, and how is the auto-confirm rate monitored?
• Are the model, prompt, and embedding versions pinned and change-controlled?
• What golden corpus gates releases, and who owns it?
• How is prompt injection from untrusted content contained?
• What is the per-feature cost ceiling, and what enforces it?
• How does the system fail — loudly, with a typed error, or silently with a plausible answer?

Governance is the adoption accelerator

The instinct to route AI around governance comes from treating the ARB as a brake. For deterministic systems run by mature teams, sometimes it is. For GenAI, it is the opposite: the absence of governance is exactly what keeps these systems stuck in pilot, unable to clear risk and compliance sign-off. An ARB that knows how to review AI is the mechanism that gets AI into production safely — and keeps it there as the models underneath keep moving.