VWO, Optimizely, GrowthBook and Statsig Implementation

An experimentation tool is implemented when its exposure counts and your warehouse's user counts agree to within 1 percent. That is the acceptance test I work to and the only one worth writing into a statement of work. The installs I am called in to repair are typically 8 to 15 percent apart, which is enough to reverse the sign of a result on any test measuring an effect worth having. Choosing the tool: This matters less than vendors claim and more than teams assume, and it turns on who will actually run tests. GrowthBook and Statsig compute results against your own warehouse, so every number in the interface can be reproduced in SQL by somebody who does not trust it, which over time is the property that keeps a program alive in front of finance. VWO and Optimizely Web sell a visual editor, which is right for a marketing team with no standing engineering allocation and wrong for a React codebase, where a WYSIWYG variant survives until the next deploy touches the DOM it was drawn against. If you have already signed, I will implement what you signed and tell you once, on the first call, where it will hurt. What gets built: Identity resolution first, because everything downstream depends on it. A stable unit id that survives the transition from anonymous to logged in, with sticky bucketing so a user does not change arm at sign-in, which is the most common cause of a mismatch nobody can explain. Then exposure events fired at assignment with deduplication, so a component re-rendering four times does not report four users. Then the data path into BigQuery, Snowflake or ClickHouse and the metric layer in dbt, one definition per metric with a named owner, so that the tool and the board deck cannot quietly hold different definitions of revenue. Then the anti-flicker snippet, configured with a timeout you have consciously chosen, or removed entirely in favour of server-side assignment, which is separate work I will scope rather than smuggle in. Metric definitions: This is where the vertical shows. Gambling: net gaming revenue after bonus cost and chargebacks at day 30, with registrations demoted to a guardrail. Marketplaces: GMV net of cancellations at day 30, paired with a supply-side guardrail so a buyer-side win that consumes seller inventory is visible. Kids-and-family: conversion measured on the parent account, engagement on the child side as a guardrail only, and no child-level identifier written into the tool at all, which sometimes rules a vendor out and is worth discovering before signature rather than after. Quality gates: Forced-variation parameters so anybody can view any arm on demand. A QA pass run on staging and again on production before ramp, because the two environments differ in exactly the way that breaks bucketing. An SRM alarm posted into Slack on a chi-square threshold of p below 0.001 with the test id attached. A documented rollback that a person on call can execute at two in the morning without me. What I decline: A visual-editor-only setup for a team that ships React weekly. An implementation where I am not allowed to see the warehouse, since the acceptance test is unverifiable without it. And signing a readout on numbers that cannot be reproduced in SQL. Not suitable for: Teams looking for ongoing vendor support or a seat to run their tests. This ends when the acceptance test passes and your engineers can operate it. Handover is two sessions and a runbook: bucketing logic, event contract, metric definitions, alarm thresholds and the rollback.