Building a Five-Stage LLM Refusal Gate for Production

Picture a support queue on a Tuesday morning. An agent reads one ticket and drafts a refund promise. The ticket is missing the order state. Nobody notices until the customer replies.

That draft was a decision made with missing evidence. The decision owner was the support lead. The consequence landed on the customer. The point of reversibility was the send button.

So I stopped asking whether the model was trustworthy. I started asking where the refusal should live. A gate is cheaper than an apology. Here is the harness, in five stages, each with its own check.

Stage 1: Define the Stop Conditions

Rules in a wiki get read once. Rules in a file get executed. Put yours in stop_conditions.json.

[
  {
    "id": "refund-promise",
    "trigger": "draft contains a refund commitment",
    "required_evidence": ["customer_id", "order_state", "refund_policy_id"],
    "owner": "support-lead",
    "allowed_action": "draft_only",
    "recovery": "ask_customer_for_order_id"
  }
]

Four fields matter most: the trigger, the evidence, the owner, the recovery. If you cannot name one owner, you have a policy, not a rule.

Verification: python -m json.tool stop_conditions.json > /dev/null && echo "schema ok" should print schema ok and exit zero. If this file is empty, your gate is a rubber stamp.

Stage 2: Set Up the Harness Environment

I keep a scratch server for gate work. The point is a machine you can break without breaking production.

When testing high-performance workflows, utilizing a reliable LLM API aggregator like n1n.ai ensures that when your gate does permit a call, it hits robust endpoints with low latency.

ssh you@your-free-server
python3 -V                 # 3.9+ for the stdlib calls below
git clone <your-fork-url> stop-gate && cd stop-gate
python3 -m venv .venv && . .venv/bin/activate
export N1N_API_KEY="<from your console>"
export MODEL_BASE_URL="<from your console>"
export MODEL_ID="<pick one your console lists>"

I am not naming models here, because the list changes and my naming it would age badly. When routing traffic through aggregators like n1n.ai, you gain the flexibility to swap underlying models without rewriting your validation harness.

Verification: python -c "import os; print(bool(os.environ['MODEL_BASE_URL']))" prints True. One working transport call is enough for the whole harness.

Stage 3: Implement the Zero-Token Gate

The gate checks evidence before any model call. A refusal therefore costs zero tokens.

# gate.py — reference implementation, stdlib only
import json, os, urllib.request

STOP = json.load(open("stop_conditions.json"))

def transport(prompt, base, key, model):
    body = json.dumps({"model": model, "input": prompt}).encode()
    req = urllib.request.Request(
        base, data=body,
        headers={"Authorization": f"Bearer {key}",
                 "Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.load(r)

def coverage(case, rule):
    have = set(f for f in rule["required_evidence"] if case.get(f))
    return have, set(rule["required_evidence"]) - have

def decide(case, call_model=None):
    for rule in STOP:
        _, missing = coverage(case, rule)
        if missing:
            return {"decision": "REFUSE", "rule": rule["id"],
                    "missing": sorted(missing), "owner": rule["owner"],
                    "recovery": rule["recovery"], "model_called": False}
    text = call_model(case) if call_model else None
    return {"decision": "DRAFT", "rule": None, "missing": [],
            "output": text, "model_called": call_model is not None}

Stage 4: Prove the Refusal

Now the part people skip. Prove the refusal never reaches the model.

# test_gate.py
from gate import decide

def boom(_):
    raise AssertionError("model must not be called on a refusal")

case = {"customer_id": "c-88", "draft": "we will refund you"}
out = decide(case, call_model=boom)
assert out["decision"] == "REFUSE", out
assert out["missing"] == ["order_state", "refund_policy_id"], out
print("refusal gate ok:", out["owner"], out["recovery"])

Verification: python test_gate.py prints refusal gate ok: support-lead ask_customer_for_order_id. When a refusal still calls the model, you have not built a gate. You have built a comment.

Stage 5: Log and Analyze Blockers

A refusal is data, not silence. Log what was missing, who owns it, and how a human recovers.

def record(path, case_id, rule, missing, decision, owner, recovery, ts):
    line = {"case_id": case_id, "rule": rule, "missing": sorted(missing),
            "decision": decision, "owner": owner,
            "recovery": recovery, "ts": ts}
    with open(path, "a") as fh:
        fh.write(json.dumps(line) + "\n")

Verification: jq -r 'select(.decision=="REFUSE") | .missing[]' decisions.jsonl | sort | uniq -c | sort -rn groups the blockers by count:

 14 order_state
  9 refund_policy_id

Fourteen refusals, one field. Fix the intake form instead of the prompt. That is the whole argument for logging last.

Human Review Protocol

The card has four lines, in this order: what the agent wanted to do, which evidence is missing, who can unblock it, what approving anyway means. Focus order must match that reading order. No color-only signals. Plain words, no system names.

My review protocol is small and repeatable. Two reviewers read one card aloud, one of them with a screen reader. I count hesitations and time how long it takes each to name the owner and the recovery path.

• Success measure: every refusal card names exactly one owner and one recovery path within ten seconds.
• Stop measure: any approval that proceeds while a required field is empty. When that happens, halt the release and fix the rule. Never relax the fixture.

It does not prove the model is safe, only that one class of action waits for evidence.

When to Skip This Architecture

Skip this if a regulator requires a contractual model version pinned to a document. Skip it for sub-second interactive flows, because the gate writes a record. Skip it if your team cannot name a single decision owner. Anonymous refusals teach nobody anything.

So which missing piece of evidence should stop your approval this week? Write that one rule, run the fixture, and let the gate refuse for you.

If you want a dependable infrastructure layer to back your automated testing workflows, exploring options like n1n.ai can streamline your multi-model integrations.

Get a free API key at n1n.ai