Analytics &amp; Experimentation

Metric Definition

How to define a metric that is measurable, attributable, and resistant to gaming —the foundation of all analytics work.

Anomaly Detection

Statistical and rule-based methods to flag metrics that deviate beyond expected variance —so a 20% drop is investigated in minutes, not discovered in next month's report.

Correlation Analysis

Measuring how two metrics move together, and the critical discipline of not confusing correlation with causation —the most common analytics error.

Statistical Significance

Computing and interpreting p-values and confidence intervals so you know whether a 3% lift is real or noise —the gate between 'run another experiment' and 'ship it'.

Data Storytelling

Structuring an analysis as a narrative —question, finding, so-what, recommendation —so a stakeholder acts on it instead of filing it.

Test Design

How to design an experiment that isolates the effect of one change —hypothesis, metric, allocation, duration —before any traffic flows.

Sample Size Calculation

Computing the minimum sample size before a test starts —so you know whether the test is feasible and never run an underpowered test that cannot detect the effect you care about.

Variant Creation

Building the treatment variant and the assignment mechanism so users see a consistent experience and the experiment is isolated from other changes.

Guardrail Metric

Metrics that must not degrade during an experiment —the 'do no harm' checks that prevent a winning feature from shipping if it breaks something else.

Stop Condition

Pre-defined rules for when to stop a test —sample size reached, guardrail breached, or max duration —so stopping is not a judgment call made mid-test.

Funnel Definition

Defining a funnel as an ordered sequence of steps with time bounds —so the drop-off rate between steps is measurable and comparable.

Drop-off Analysis

Quantifying and diagnosing the users who exit between funnel steps —not just the rate, but the reason, so the drop-off is fixable.

Micro-Funnel

A funnel within a single page or interaction —the 3-step micro-journey (e.g., open modal, fill field, submit) that determines whether a macro-funnel step converts.

Conversion Rate

The fundamental funnel metric —the fraction of users who complete a step —and the four common definitions that produce different numbers for the same funnel.

Bottleneck Identification

Finding the single funnel step that, if fixed, would lift overall conversion the most —so effort goes where the leverage is, not where the drop is biggest.

Event Design

Designing the event taxonomy —names, properties, triggers —so the data is queryable, consistent, and durable across product changes.

Session Analysis

Grouping events into sessions and analyzing session-level behavior —length, depth, exit point —to understand engagement beyond single events.

Heat Map

Visualizing where users click, scroll, and hover on a page —to find the elements that attract attention and the ones that are ignored.

User Journey

Mapping the multi-session path users take from first touch to conversion —the non-linear, real-world flow that funnels approximate but cannot capture.

Cohort Analysis

Grouping users by a shared characteristic (signup date, acquisition source, plan) and tracking their behavior over time —to see retention, engagement, and revenue curves by cohort.

Conversion Definition

Defining what counts as a conversion —the boundary between interest and commitment —so every funnel, experiment, and revenue number rests on a shared, unambiguous event.

Attribution Model

Choosing and implementing an attribution model —last-touch, first-touch, linear, time-decay, data-driven —that fairly credits the channels and touchpoints that drove a conversion.

LTV Calculation

Computing customer lifetime value —the total revenue a user generates over their full relationship with the product —to set acquisition budget, prioritize segments, and measure business health.

CAC Payback

Measuring customer acquisition cost and the payback period —how much it costs to acquire a paying customer and how many months of revenue to recover that cost —the gating metric for sustainable growth.

Revenue Attribution

Attributing revenue to the products, features, and user actions that generated it —so product decisions are grounded in financial impact, not vanity metrics.

Dashboard Design

Designing dashboards that answer questions rather than display data —choosing layout, density, and interaction so a user can find the answer in 30 seconds, not 30 minutes.

KPI Selection

Choosing the 3-5 key performance indicators that define success for the team —the numbers that, if they move, mean the team is winning —and resisting the temptation to track 20.

Real-Time Monitoring

Setting up real-time monitoring for the metrics that need immediate response —error rates, latency, conversion drops —so the team detects and responds to incidents in minutes, not hours.

Executive View

Building the executive dashboard —the 5-7 numbers the CEO, CFO, and board review weekly —that tells the story of the business without requiring the executive to drill into operational detail.

Self-Serve Analytics

Giving the whole company access to query, explore, and visualize data without filing tickets —while maintaining data quality, governance, and cost control so self-serve does not become chaos.

Hypothesis Formation

Writing a testable hypothesis —'if we change X, then Y will increase by Z because of mechanism W' —before any code is written, so the experiment tests a prediction rather than exploring a change.

Randomization

Implementing correct randomization —stable assignment, balanced buckets, no carryover effects —so the treatment and control groups are comparable and the experiment measures causation, not correlation.

Guardrail Setup

Setting guardrail metrics that automatically kill an experiment if the treatment harms the business —so a winning primary metric never comes at the cost of a degrading retention, revenue, or latency.

Ship Decision

Making the ship decision after an experiment —interpreting the result, checking guardrails, deciding between ship/iterate/abandon, and documenting the learning regardless of the outcome.