This post is the pattern in full, with the actual code shapes we use. It is not clever. That is the point.

The shape of the thing

Three parts, always the same:

1. A classifier that scores each item and decides: act, or escalate.
2. A queue that holds escalated items with everything a reviewer needs on one screen.
3. A feedback loop — every human decision becomes a labelled example for the next tuning pass.

The threshold between “act” and “escalate” is a business decision dressed up as an engineering one. It comes straight from the cost of a mistake:

The classifier

The scoring call is deliberately dull. One function, one structured answer, no streaming, no agents:

from anthropic import Anthropic

client = Anthropic()

def classify(invoice_text: str, po_candidates: list[dict]) -> dict:
    """Match an invoice to a PO, or admit we can't."""
    response = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=512,
        system=SYSTEM_PROMPT,  # the rules a human would state, verbatim
        messages=[{
            "role": "user",
            "content": render_prompt(invoice_text, po_candidates),
        }],
    )
    result = parse_json(response.content[0].text)
    # result: {"po_number": "PO-4471", "confidence": 0.94, "reason": "..."}
    return result

Two details that matter more than the model choice:

• The system prompt is the rules from the scoping call, written down. When the client says “we never auto-post invoices over €10,000”, that sentence goes in verbatim.
• The model must be allowed to say I don’t know. A forced choice with no escape hatch is how confident nonsense ends up in your accounting system.

The queue

The queue is a table. Resist making it more than a table.

CREATE TABLE review_queue (
    id           bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    item_type    text        NOT NULL,   -- 'invoice', 'ticket', ...
    payload      jsonb       NOT NULL,   -- everything the reviewer sees
    suggestion   jsonb       NOT NULL,   -- what the model would have done
    confidence   numeric     NOT NULL,
    status       text        NOT NULL DEFAULT 'pending',
    decided_by   text,
    decided_at   timestamptz,
    created_at   timestamptz NOT NULL DEFAULT now()
);

The reviewer sees the item, the model’s suggestion, and its reasoning — and makes one of two moves: approve the suggestion or correct it. One keystroke each. If clearing the queue takes more than a few seconds per item, people stop clearing it, and the whole system quietly rots.

A review queue nobody clears is worse than no automation at all: the work still isn’t done, and now everyone believes it is.

The feedback loop

Every decision is a labelled example. Once a month, we look at where humans overrode the model and tune — usually the prompt, occasionally the threshold, rarely the model:

# tuning-run.yaml — inputs to a monthly review pass
window: 30d
export:
  - decisions_where: status = 'corrected'
  - decisions_where: confidence > 0.9 AND status = 'corrected'  # the scary ones
review:
  - prompt_rules      # do the written rules cover the misses?
  - threshold         # is 0.9 still right for this cost of error?

The metric we watch is not accuracy. It is escalation rate over time. A healthy system starts around 20–30% escalated and drifts down as the rules sharpen. If it drifts up, the process changed and nobody told the software — which is exactly what the queue is for catching.

Where this leaves you

Ninety percent automatic with a fast human queue beats one hundred percent automatic in every deployment we have done. The queue is not a compromise on the way to “full” automation — it is the finished state, and the reason the system is still trusted a year later.

If you’re weighing a process of your own, the five-question checklist is the place to start — question four is the one this whole post hangs off.

01 — Does it happen at least weekly?

Frequency is the whole economics of automation. A task done 200 times a week pays back a build in months. A quarterly report, almost never. Count the repetitions honestly — including the times someone does it wrong and redoes it.

02 — Can you write the rules down?

If an experienced person can explain how they decide — “if the invoice matches a PO, post it; if not, flag it” — software can do it, and AI handles the messy middle: reading PDFs, interpreting emails, extracting fields. If nobody can explain the decision, fix the process before you automate it.

03 — Is the data reachable?

Automation needs a way in and a way out. Email inboxes, spreadsheets, and most modern tools are easy. A desktop app from 2009 with no export is not impossible — but it moves the quote. Know where your data lives before the call; it’s the first thing we’ll ask.

04 — What does a mistake cost?

This decides how much human review stays in the loop. A mistagged support ticket costs minutes; a wrong payment costs real money. High-stakes steps keep a human approval; low-stakes steps run straight through. Most good systems are 90% automatic with a small review queue — like the flagged invoice in our own pipeline.

05 — Will anyone own it?

Software needs one person on your side who cares that it keeps running — who reviews the flagged items and tells us when the process changes. Without an owner, the best automation quietly dies in six months. Name that person before you build.

The short version. Weekly or more · rules you can state · reachable data · known cost of error · a named owner. Score four out of five and it’s probably worth a call.

If you scored your process and it passed, the next step is the same one every project here starts with: a 30-minute call where you walk us through it. We’ll tell you honestly if it’s a no — that’s half the calls, remember.