AI Agents in Marketing Ops, With Evaluation Sets and Review Queues

We ship an LLM step when it beats current handling on a labelled set of around 200 real examples held back from the prompt work. Close to a third of the use cases clients bring us never clear that bar, and those we hand back rather than build. The failure is rarely the model. It is that the task was never defined tightly enough for anyone, human or otherwise, to be consistently right about it. Where this earns its cost: Inbound RFQ triage, where an email with a drawing or a spec sheet attached has to become a structured enquiry carrying material, quantity, tolerance and a routing decision. CV and profile parsing into a candidate record where the schema is yours rather than a vendor's. Call and meeting notes turned into field updates on the correct object instead of a wall of text in a note. Product classification against your own taxonomy, which is the one thing generic tools cannot do, because they have never seen your taxonomy. How the confidence question is settled: Every classification returns a decision and a confidence score. Below the threshold we agree in writing, the item goes to a review queue with the model's reasoning visible and a one-click correction. Corrections append to the eval set, so the queue is also the mechanism that improves the system rather than a permanent tax. Above the threshold, the write happens automatically and is logged with model version and prompt hash, so a bad batch can be found and reversed by version instead of by hand. What we refuse: No autonomous outbound. Nothing we build sends an email or message to your customer or candidate without a person approving that specific message. No agent with write access to closed-won records or to anything feeding invoicing. No multi-agent architecture where one model checks another's work, because the second model is usually a way of avoiding an eval set and it pays more per run to be wrong more expensively. And if a regular expression and a lookup table reach ninety-six percent on the same eval set, we build the regular expression and tell you what it saved. What you receive: The labelled eval set as a CSV you own, with the scoring script. That file, more than the prompts, is the asset: it is what lets you change model or vendor next year without starting over. The workflows in n8n or Make with model calls, thresholds and review queue wired in. A cost-per-run figure measured on your real volume rather than estimated, including what the review queue costs in human minutes. And a decision record naming which use cases we tested, which we rejected, and the number that made us reject them. Who this is not for: Anyone wanting an AI SDR or automated personalised outbound at volume. We do not build it and will not subcontract it. Teams with nobody able to spend a few hours labelling examples; there is no substitute for that, and we cannot label your RFQs correctly on your behalf. And organisations needing a fixed accuracy guarantee written into the contract. We commit to a threshold and to not shipping below it, which is a different promise and the only one we can keep. The retainer exists because these systems drift. Input distributions change, a vendor deprecates a model, your taxonomy grows a branch. Monthly work is re-scoring against the eval set, acting on review-queue corrections, and retiring steps that no longer earn their run cost.