Academy

The rule layer: deterministic guardrails around a probabilistic model

A language model is probabilistic and sometimes errs. The way to make it reliable isn't to perfect the model; it's to build a deterministic layer that catches what slips through.

The Felesh team 4 min

A language model is fundamentally probabilistic: the same input yields different outputs, and errors are inevitable. The most common mistake when building reliable systems is trying to force the model to be flawless. A better approach accepts this probability and wraps the model in a deterministic layer to catch whatever slips through.

The division of labour: cognition versus rules

The foundational principle is separating two kinds of work. The model handles the cognitive work: understanding context, reasoning, generating language, and making decisions that are not rule-bound. The rule layer — the harness — handles the deterministic work: validation, error recovery, policy enforcement, and anything expressible as a strict rule. The model is free to generate; the harness ensures that the generation conforms to the rules.

A trigger, not just a prompt: where the rule belongs

The key architectural question is where each rule should live. These fall into three categories:

  • Hard policy belongs in the harness, not the prompt. For example, “any output that contradicts the given context must be rejected” — a deterministic check downstream does this, not a hope that the model will respect it itself.
  • Soft guidance belongs in the prompt, where the model decides. Instructions like “prefer fidelity over fluency” require balance. The model weighs the trade-off but is not mechanically constrained.
  • Format constraints are validated by the harness. The model generates structured output as a best effort; the harness validator verifies it and routes errors to a fallback path.
Where each rule livesThree rules, sorted by home. Refunds need an approved claim: a harness policy check, which blocks a reply that promises a refund with no claim on file. Prefer fidelity over fluency: prompt guidance, which the model weighs itself. The ETA is written in the order’s date format: a harness validator, which holds the reply back when the date is malformed.rulelives inrefunds need an approved claimharness: policyreply: “Refunded, sorry.” with no claim on fileprefer fidelity over fluencyprompt: guidancereply: says what the record says, in plain wordsETA in the order’s date formatharness: formatreply: “ETA: 16 Oct” or it is not sent
Each rule has one home: hard policy and format checks sit in the harness, soft guidance sits in the prompt.

Walk the data path

To design this layer effectively, run a decisive exercise: walk the data path of each call. Before invoking the model, what does the harness assemble? What exact input does the model receive? What is it permitted to invent, and what must be strictly extracted from context? After the model returns, what does the harness verify or mitigate? Tracing this path clarifies which responsibility belongs to the model and which to the rule layer — and that removes the most common design error.

One reply, checked against its recordOrder #5530 has a record that says in transit with an ETA of 16 Oct. The record flows to the model, which drafts a reply that the parcel was delivered on 15 Oct. The draft flows back to the harness, which checks status = delivered? against the record and fails. The reply that is sent is the fallback: in transit, ETA 16 Oct.“Where is my parcel?” · order #5530harnessrecord (order #5530):status: in transitETA: 16 Octstatus = delivered?record: in transitmodel: draftsreads only the record“Delivered 15 Oct”fallback reply: “in transit, ETA 16 Oct”
Walking one reply shows where the error is caught: the draft’s date fails against the record, and the harness sends a fallback instead.

The verifier as a guardrail

A powerful extension of the rule layer is the verifier model. When a single model cannot guarantee a critical constraint, a second model acts as a guardrail. The primary model generates; the secondary model evaluates both the input and the output, rejecting the result if a rule is violated. This is the same primary-and-verifier pattern, but here the verifier’s role is a policy guardrail: a gatekeeper that won’t let an undesirable output through the gate.

A verifier blocks an unapproved creditA customer asks for a credit for order #5530, which was late. The model drafts “I added a 15% credit.” The verifier stands between the draft and the send, and checks the draft against the rule that credits need an agent’s approval. The record shows no approval, so the draft is blocked, and the reply that is sent instead is passed to billing for confirmation.“My order #5530 was late. Can I have a credit?”model“I added a 15% credit.”verifierrule: credits need an agent’s approvalno approval on filefallback: “passed to billing for confirmation”
The verifier reads the question and the draft together, and blocks a draft that promises a credit no rule grants.

The key invariant

The core principle remains: the goal is not to force the model into flawless compliance. That is impossible. The goal is to allow the model to generate freely, while the harness assumes full responsibility for enforcing constraints. The rule layer catches exactly what slips through the model’s training. This is the structural difference between a system built on the hope of a perfect model, and one designed around the certainty that a model will err. The latter is always more stable.