Spend and Performance Anomaly Alerting with Runbooks

An alert is worth building only if a named person would change something within an hour of reading it. On the accounts we have done this for, that comes to between four and nine rules. We have never shipped more than twelve, and we delete rules every quarter. How the thresholds are set: Baselines come from eight weeks of the same hour on the same weekday, because Tuesday 09:00 and Sunday 22:00 are different populations and a single daily average hides both. Deviation is scored against median absolute deviation rather than a mean and standard deviation, since one runaway day poisons a mean and then the rule stops firing exactly when it should. Every rule carries an absolute floor in currency alongside the statistical trigger, so nobody is woken at 02:00 because a campaign spending forty dollars a day doubled. The rules that matter: Tracking outage first: zero conversions for two consecutive business hours on a property whose eight-week median for that slot is well above zero. This is the failure that costs the most and gets caught the latest, usually by someone noticing a quiet week. Second, divergence between browser-side and server-side event counts past fifteen percent, which is how a broken tag or an expired access token announces itself before the conversion numbers move. Then budget pacing forecast to month end, spend velocity against the same hour last week, and platform-side blockers: disapprovals, feed errors, payment failures, account-level restrictions. For crypto and financial services accounts the restriction alerts fire more than the performance ones, and that is the right ratio. What an alert looks like: The message names the owner, states the observed value and the expected range, links to the runbook step for that specific rule, and lists the last three times it fired and what was done. An alert with no runbook is a notification, and notifications get muted. Routing is by severity into separate channels, with paging reserved for the two or three rules that are genuinely revenue-critical. Nothing goes to an email distribution list; a list with eleven people on it has nobody on it. The kill policy: Any rule that fires more than six times in a quarter without a recorded action gets deleted rather than retuned. We review the log with you every quarter and the default answer is removal. A channel people trust because it is quiet is worth more than every rule we could think of. What we refuse: No thresholds on percentage change alone; a fifty percent swing on a small base is noise and firing on it is how a team learns to ignore the channel. No alerts on metrics your team cannot act on, which rules out most share-of-voice and competitor-position monitoring. No anomaly detection sold as a model where a fixed threshold agreed with the media lead would do the same job and be explainable at three in the morning. Who this is not for: Accounts where nobody is on duty, since an alert with no responder is theatre. Teams wanting one system watching every metric they collect. Anyone looking for a monitoring product; this is configuration, statistics and runbooks built on your own warehouse and your existing Slack and paging tools, not software we license to you. Where it runs: On your data, in your cloud project, as scheduled queries or a small scheduler service, with the rule definitions in version control so a threshold change has an author and a date. If you later want it moved in-house entirely, everything needed to run it is already yours.