Getting Started with ABS

ABS is a free, client-side A/B test sample-size calculator that runs entirely in your browser. It helps data scientists and product teams plan experiments by computing required sample size, test duration, and feasibility in seconds. No sign-up, no installation, no data sent to a server.

Quick Start

  1. Open https://abs.codecide.net in any modern browser.
  2. The calculator loads with sensible defaults for a typical proportion-based A/B test (baseline rate of 5%, relative MDE of 10%, 80% power, 5% significance).
  3. Results appear instantly in the right-hand panel. No button to press.

That is it. You can now adjust parameters to match your experiment.

Core Parameters

The left panel contains everything you need to describe your experiment. Here is what each section controls.

Metric Type

Choose what you are measuring:

Option Use case Test method used
Proportion Conversion rates, click-through rates z-test
Continuous Revenue per user, time on page, scores t-test
Count Events per user (rare events) chi-squared

The default is Proportion, which covers most web experiments.

Baseline Rate

Enter the current conversion rate for your control group as a decimal. For example, if your signup page converts at 5%, enter 0.05. This value must be between 0 and 1.

Minimum Detectable Effect (MDE)

Toggle between Relative and Absolute mode:

  • Relative: The MDE as a percentage of the baseline rate. An MDE of 10 at a 5% baseline means you want to detect an absolute lift of 0.5 percentage points (to 5.5%).
  • Absolute: The raw absolute difference you want to detect. An MDE of 0.005 is a 0.5 percentage point lift.

The slider and number input sync automatically. Drag to explore how sample size scales with the effect size.

Significance Level (Alpha)

The probability of a false positive. Common values:

Alpha Use case
0.01 High-stakes decisions
0.05 Standard product tests
0.10 Exploratory experiments

Statistical Power

The probability of detecting a true effect when one exists. Common values:

Power Use case
0.80 Standard (recommended)
0.90 Higher certainty
0.95 Critical decisions

Tail Type

  • Two-tailed: Detects differences in either direction (default).
  • One-tailed: Detects differences in one direction only. Requires half the sample size but should only be used when you are certain the treatment cannot perform worse.

Framework

Choose between Frequentist and Bayesian approaches:

  • Frequentist: Standard z-test, t-test, or chi-squared based on metric type.
  • Bayesian: Uses Beta-Binomial priors. When selected, additional fields appear for prior alpha and prior beta parameters (default to 1 each for a uniform prior).

Variants and Allocation

  • Number of variants: How many groups are in the experiment (including control). Must be at least 2.
  • Allocation ratios: The traffic split between variants. For example, [0.5, 0.5] means 50% to control and 50% to treatment. Ratios must sum to 1.0.

Unequal allocation (e.g., 90/10) increases total sample size. The Allocation Efficiency Cost field in the results shows how much total N increases above the equal-split baseline.

Multiple Comparison Correction

When testing more than 2 variants, you must correct for multiple comparisons:

Method Description
None No correction (not recommended for >2 variants)
Bonferroni Conservative; divides alpha by number of comparisons
Benjamini-Hochberg Controls false discovery rate; less conservative
Dunnett Optimized for comparisons against a single control

Traffic and Dropout

  • Daily eligible users: The number of users entering the experiment each day. This determines test duration.
  • Expected dropout rate: Attrition as a percentage (0-100). Total sample size is inflated by 1 / (1 - dropout_rate / 100) to compensate.
  • Minimum runtime: The shortest acceptable test duration in days. Altering this has no effect on the calculation; it serves as a reference.

Advanced Options

Expand the Advanced section to access:

  • CUPED correlation: Reduces required sample size when you have a pre-experiment covariate correlated with the outcome. Enter the correlation (0 to <1). Sample size is reduced by 1 - ρ². A correlation of 0.5 reduces N by 25%.
  • Sequential testing: Inflates sample size to account for multiple interim analyses. Enter the number of planned looks.
  • Continuity correction: Applies Yates' continuity correction to proportion-based sample size calculations.
  • Average revenue per user (ARPU): Enables a projected incremental revenue estimate in the results panel.

Reading the Results

Results update live as you change any parameter.

Sample Size

  • Sample Size per Variant: The number of experimental units needed in each group.
  • Total Sample Size: Sum across all variants, accounting for allocation ratios and dropout.

Duration and Feasibility

  • Days / Weeks: Estimated calendar time based on daily traffic.
  • Feasibility status:
Status Threshold Meaning
Feasible ≤ 28 days Test is practical
Slow 29–84 days Test will take 1–3 months
Impractical > 84 days Consider reducing MDE or running on more traffic

Effective MDE

Shows the MDE in both absolute and relative terms after all corrections (multiple comparisons, unequal allocation) are applied.

Revenue Projection

If you provide an ARPU value, the results include estimated incremental revenue from a successful test, computed as MDE_absolute × total_sample_size × ARPU.

Sensitivity and Power Curves

Below the main results, two interactive charts help you explore trade-offs:

  • Sensitivity curve: Shows how sample size changes as the MDE is scaled from 50% to 200% of its nominal value.
  • Power curve: Shows achieved power across a range of per-variant sample sizes.

Hover over points on either chart to see exact values.

Demo Datasets

If you are new to A/B test planning, click the Datasets dropdown near the top of the input panel. Pre-configured scenarios include:

  • High-volume ecommerce checkout
  • Low-traffic B2B SaaS signup
  • Mobile onboarding flow
  • Email campaign CTR
  • Multi-variant pricing page
  • Revenue-per-user (continuous) test

Selecting a dataset replaces your current inputs with realistic values and immediately recalculates.

Resetting

Click Reset to defaults at the bottom of the input panel to restore all parameters to their initial values: a two-tailed proportion test with 5% baseline, 10% relative MDE, 80% power, and 5% alpha.

Using the API

If you need to integrate sample-size calculations into your own tools, ABS provides two programmatic interfaces.

REST API

The API is hosted at https://abs.codecide.net/api. Full interactive documentation is available at https://abs.codecide.net/api/docs.

Three endpoints are available:

Method Endpoint Purpose
POST /api/calculate Full sample-size calculation
POST /api/validate Validate inputs without running a calculation
POST /api/inverse-mde Find the MDE for a fixed sample-size cap

All endpoints accept JSON and return a consistent response envelope:

{
  "success": true,
  "result": { /* endpoint-specific output */ },
  "metadata": {
    "processingTimeMs": 1.2,
    "inputCount": 20,
    "outputCount": 4
  }
}

Error responses follow this structure:

{
  "success": false,
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Inputs contain blocking validation errors.",
    "validationErrors": [
      {
        "field": "baselineRate",
        "message": "Baseline rate must be between 0 and 1 (exclusive).",
        "severity": "error"
      }
    ]
  }
}

Example: calculate sample size for a basic proportion test.

curl -X POST https://abs.codecide.net/api/calculate \
  -H "Content-Type: application/json" \
  -d '{
    "metricType": "proportion",
    "testMethod": "z-test",
    "framework": "frequentist",
    "tailType": "two-tailed",
    "baselineRate": 0.05,
    "mdeMode": "relative",
    "mdeRelative": 10,
    "alpha": 0.05,
    "power": 0.8,
    "numberOfVariants": 2,
    "allocationRatios": [0.5, 0.5],
    "correctionMethod": "none",
    "dailyTraffic": 10000,
    "dropoutRate": 0,
    "minimumRuntimeDays": 14,
    "continuityCorrection": false
  }'

MCP (Model Context Protocol)

AI assistants that support the Model Context Protocol can call ABS tools directly. The MCP endpoint is available at https://abs.codecide.net/mcp using streamable HTTP transport.

Three tools are exposed:

Tool name Purpose
plan_ab_test Full sample-size, duration, and feasibility
solve_min_detectable_effect Inverse MDE for a given fixed N
estimate_precision Expected confidence-interval half-width

The plan_ab_test tool includes sensible defaults. At minimum, provide baseline_rate, mde_relative, and daily_traffic. You can also reference named presets like signup or checkout via the preset parameter.