The Experimentation Edge
Growthbook
0
The Experimentation Edge is a podcast about how product teams decide what to build and what not to build. Product, growth, and engineering leaders share how A/B testing, feature flags, and experimentation drive business outcomes, with examples from companies like DoorDash, Atlassian, and UPS. Hosted by Ashley Stirrup, CMO at GrowthBook, it is aimed at product managers, engineers, data scientists, and growth leaders at B2B tech companies. Episodes focus on experimentation culture, statistical rigor, and shipping with confidence, without marketing speak.
Episodes
-
Typeform on how to stop running experiments and start earning them 24.02.2026 31mIf 80% of A/B tests fail, how do you de-risk decisions that touch pricing, product, and brand? Aleksandra (Aleks) Bass, Chief Product & Technology Officer at Typeform, shares how her team “earns the right to A/B test” with medical-grade rigor—moving from literature reviews and user tests to simulated trials before exposing changes to customers. She details Typeform’s repositioning from “forms” to an AI engagement platform—and the pricing and packaging bet behind it: a 15% drop in new business count offset by a 32% increase in ASP and a 25% lift in annual attach. Aleks unpacks how Typeform AI a... -
How Moxie Pest Control boosted conversions 5% with data and coaching 05.03.2026 17mThink pest control isn’t a digital business? Think again. Raj Mehta, Vice President of Product and Technology at Moxie Pest Control, outlines how he turned a spreadsheet-run operation into a data-driven engine across 9,000+ daily calls. Raj shares how consolidating fragmented systems into a data lake unlocked automation—from shrinking lead routing from 20–25 minutes to under 30 seconds—to deploying AI-powered call intelligence that scores every sales and retention conversation against a playbook. He breaks down a practical roadmap for traditional businesses: build MVPs, pilot in one branch wit... -
Upwork on AI-Driven Ops at scale 11.03.2026 30mSummary What do you test rigorously—and what do you ship fast and fix forward—when every change could impact millions? Vinoj Kumar, Vice President of Engineering at Upwork, leads at the intersection of infrastructure and product, where feedback loops are longer and the blast radius is wider. He shares a pragmatic framework for experimentation—blast radius x reversibility—that sets testing rigor, plus how he measures success in product terms: faster search, resilient marketplace trust, and developer velocity. Vinoj explains why “high engagement” can mask low-quality experiences, how his team in... -
From chatbots to open-world agents at Microsoft: evals, go-live metrics, and copilot velocity 17.03.2026 28mAI is moving so fast that what “good” looked like a few months ago is already outdated. So how do you measure value, ship safely, and scale what works? Marco Casalaina, VP of Products, Core AI and AI Futurist at Microsoft, joins to unpack how his team builds and evaluates next‑gen AI—at hyperspeed. Marco leads the AI Futures team and previously led Azure OpenAI, Azure Cognitive Services, Responsible AI, and AI Studio; before Microsoft, he ran Salesforce Einstein. He explains why enterprise value is best measured by go‑lives and real usage, how Microsoft’s Foundry equips developers with agent‑s... -
Truist on shipping faster and more safely with AI and human-in-the-loop banking 02.04.2026 27mHow do you boost developer velocity in a highly regulated industry—without sacrificing safety or customer trust? Charles Williams, Senior Vice President and Software Engineering Director at Truist (formed from the BB&T and SunTrust merger), shares how his team elevates developer experience to ship faster and more reliably. Charles breaks down shifting quality “left” with automation, measuring success with both DORA metrics and developer sentiment, and why human-in-the-loop is non-negotiable for AI in finance. He details Truist’s governance model—steering committees, enterprise architecture, an... -
The 5 pixels that cost LinkedIn a million dollars a month 02.04.2026 37mSummaryHow do you build a culture where nothing ships without evidence—and leaders actually act on the data? Makram Mansour, Head of Marketplace at ID.me and former experimentation leader at LinkedIn and Intuit, shares the systems, mindsets, and guardrails behind “experimenting everywhere.” At LinkedIn, he helped support 10,000+ annual experiments with 2,000 weekly platform users, and he explains the hard-earned lessons (like a 5px UI tweak causing a million-dollar ad loss) that led to a “test before release” mandate. At Intuit, he operationalized “fail forward,” partnering with HR to rewrite... -
Fyxer's engineering playbook to go from $1M to $35M ARR 02.04.2026 30mSummaryHow do you drive hypergrowth without guessing? Kameron Tanseli, Head of Growth Engineering at Fyxer—an AI assistant for your email—breaks down the experimentation playbook that helped the company scale from $1M to $35M ARR, with sights set on $100–$150M. Kameron explains how startups should think about A/B testing differently: de-risk big bets, not just button colors. He shares a risk-based approach to when to run rigorous tests vs. ship-and-measure, why a 25% win rate is a sign you’re testing ambitiously, and how PLG features should be shipped first, then rapidly iterated to drive usag... -
Ancestry on AI Storytelling, Paywalls, and Decision Quality 14.04.2026 25mSummaryWhat happens when A/B testing stops being a tool and becomes your operating system? Suresh Teckchandani, VP of Product & Technology at Ancestry (formerly PayPal and eBay), shares how the team scaled experimentation from isolated tests to capability-building that drives roadmap and revenue. He details the “growth metering” and paywall experiments that unlocked a 5.3% lift in key engagement and improved conversions—then became platform features. Suresh explains Ancestry’s centralized experimentation platform with self-serve access for PMs and engineers, why “obvious” UX changes can be the... -
Inside Chess.com's Plan to Run 1,000 Experiments in a Single Year 21.04.2026 31mSummaryChess.com ran its first A/B test in 2023. Two years later, the team is on track to run 1,000 experiments in a single year—and they've already shipped 195 in Q1.In this episode, Ashley Stirrup sits down with Nafis Shaikh, Director of Product Management at Chess.com, to get inside the experimentation engine powering one of the world's most beloved gaming products.Nafis brings experience from Zynga and Prodigy and a refreshingly honest take on what changes when a product built on passion suddenly has to serve a 10-million-DAU user base that spans absolute beginners to rated FIDE players. H... -
How Experimentation Led to Annual Growth at Fanatics 07.05.2026 27mHow Experimentation Leads to Annual Growth Every Year at Fanatics -
How UPS generated half a billion from 80+ apps with A/B testing 11.05.2026 23mThis episode of The Experimentation Edge shows how UPS's A/B testing program drove $500M+ in incremental revenue across 80+ customer-facing applications. Dave Massey — head of the J.E.D.I. team (Journey Experience and Design Innovation) — walks through the first test that proved UX could move revenue, how he defended counterintuitive results to skeptical execs, and how a small experimentation team can override opinion with data at enterprise scale.SummaryDave Massey walked into UPS in 2016 and immediately got pulled into a meeting about AB testing tools. By the end of the day, he owned the pla... -
How DoorDash saved millions with one A/B test 12.05.2026 31mThis episode of The Experimentation Edge unpacks how DoorDash's experimentation platform runs 12,000+ A/B tests per year across 42 million monthly active users — and now powers merchant-led testing on menu pricing and promotions. Ilya Izrailevsky, Senior Engineering Manager leading the platform, explains how feature flags, marketplace experimentation, and CEO-level experiment reviews built a multi-million-dollar experimentation culture across consumers, dashers, and merchants.SummaryMost companies struggle to scale experimentation beyond engineering teams. DoorDash runs over 12,000 experiments... -
Atlassian on the talent product turnaround from A/B testing 13.05.2026 22mThis episode of The Experimentation Edge explores how A/B testing, feature flags, and user research transformed Atlassian's talent product after it failed with its first users. Andrew Willingham — 11 years at Amazon, now Head of Legal and People Products at Atlassian — shares how product experimentation works when you can't test at scale, why your customer and your user are not the same person, and how the metrics you choose decide which experiments you can even run.SummaryAndrew Willingham, Head of Legal and People Products at Atlassian, spent 11 years at Amazon before joining Atlassian a yea... -
The 2% close rate increase that turned Ford Credit's product teams into believers 02.06.2026 36mSummaryOn this edition of The Experimentation Edge, Ashley Stirrup talks with Geoffrey Bell, Experimentation Product Specialist at Ford Credit, about building an experimentation practice inside a captive auto lender. Geoffrey shares the losing test that earned his program credibility, the "experimentation piggy bank" he picked up at Microsoft, and the breakthrough of connecting online experiments to offline dealership receivables. The throughline: a program proves its worth not just by the wins it ships, but by the expensive mistakes it prevents and the revenue it can finally trace. It's for p... -
RingCentral's DART framework: The four metrics that actually measure AI agents 15.06.2026 32mSummaryRingCentral's Director of Product Management for AI Products, Mayank Agarwal, joins host Ashley Stirrup to dismantle the metrics most teams use to judge AI agents. Drawing on his background founding an AI-first quantitative trading firm and scaling Groupon's bookable marketplace, Mayank explains why accuracy and thumbs-up/down feedback both mislead, and introduces DART — a four-metric behavioral framework (decay, acceptance, relevance, task completion) ported from how he measured trading strategies. He also breaks down a Groupon flash-discount experiment that backfired and the scarcity... -
Signet Jeweler's "View All" page made more money by showing less 17.06.2026 27mSummaryCraig Kistler, VP of Experience Design, Personalization, and Experimentation at Signet Jewelers (the parent company of Kay, Jared, Zales, Peoples, and Banter), joins host Ashley Stirrup on The Experimentation Edge to unpack how a hybrid online-and-in-store jewelry retailer runs experimentation at scale. Craig shares the counterintuitive "view all" experiment where his team blocked the product grid, added friction on purpose, and grew revenue; why he optimizes for revenue per visitor instead of conversion rate; and how Signet deliberately traded a high-volume testing program for fewer, h... -
Squarespace killed its blank template and built something better 23.06.2026 22mSummaryWhat do you do when your big launch increases engagement and tanks conversion? On this episode of The Experimentation Edge, host Ashley Stirrup talks with Lina Blackman, Director of Product Analytics at Squarespace, about the blank template launch that flopped — and how its learnings became Blueprint, Squarespace's AI-guided website builder. Lina explains how her embedded analyst team runs 150–200 experiments a year for 3 million customers, the two questions she asks every time a test loses, why teams only need one or two big wins a quarter, how Squarespace calibrates statistical certai... -
Twitch on why false negatives kill product ideas 24.06.2026 28mSummaryHow do you make a high-stakes product decision when the safe choice is to never test it at all? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Arun Bodapati, director of data science at Twitch, about the discipline behind trustworthy experimentation. Drawing on his experience at Schwab, Uber, and Twitch, Arun explains why false negatives are the most dangerous result a team can produce, what hygiene to nail before you push play, and how Twitch used geo-fenced experiments and causal inference to finally settle a pricing question it had avoided for years. It's... -
Ship faster, measure better: experimentation tips from JPMorgan Chase 25.06.2026 26mSummaryHow do you know if the thing you just shipped actually worked? On this episode of The Experimentation Edge, host Ashley Stirrup, CMO of GrowthBook, sits down with Kevin Yang, Executive Director and Head of Experimentation at JPMorgan Chase, who has spent six years building experimentation across Chase's digital platforms. Kevin shares how his team turned experimentation into more than a billion dollars of estimated value, why the losing experiments matter more than the winners, and the simple chart exercise he uses to prove that a million-dollar change is invisible without a control gro... -
How Disney picks which experiments to run 29.06.2026 33mSummaryWhat does it look like to kill a multimillion dollar feature before anyone builds it? In this episode of The Experimentation Edge, host Ashley Stirrup talks with Crystal Ammari, a digital product optimization and experimentation strategy leader whose career spans Nike and The Walt Disney Company. Crystal shares the "dry test" that used a single fake button to measure demand for video chat (4 million users, 106 clicks), why she reframes experimentation as savings and gains rather than wins and losses, how a misconfigured tool, not bad methodology, made tests take six months, and how a st...
Popular in
The podcast also appears in the podcast charts of these countries.