The pager blares at 7:03 a.m. on a Monday. The on-call engineer groans, checks the dashboard, and sees CPU spiking to 92%, well above the static 85% critical threshold. After a quick glance at the traffic graph, the cause is obvious: the weekend lull has ended, and the usual Monday-morning ramp-up is in full swing. The alert is a false positive, but it is not an isolated hiccup. Across the team, similar Monday-morning spikes fire dozens of times each week, eroding confidence in the monitoring system. Engineers begin to silence notifications, assuming another "expected" surge, and genuine anomalies start to slip through unnoticed. This pattern, alert fatigue born from treating routine seasonality as catastrophe, carries a measurable cost: missed incidents, longer mean-time-to-detect, and a drag on productivity as teams waste hours triaging noise. The problem is not just noisy alerts; it is that our tools are fundamentally misspecified.
The core issue is simple: static thresholds applied directly to a raw time series cannot separate expected patterns from true outliers. When the underlying signal contains daily, weekly, or seasonal cycles, those cycles inflate the variance that the threshold treats as anomalous. Consequently, the detection method is fundamentally misspecified; it flags features of the workload as failures. To build production-grade anomaly detection, we must first decompose the series, isolating the unpredictable component, and only then apply statistical thresholds. Decomposition is not an optional tweak. It is a prerequisite for any alerting system that hopes to earn trust.
The Fundamental Flaw of Static Thresholds
Static thresholds treat every deviation from the mean as suspicious, even when the deviation is baked into the workload. Consider a web service that handles user requests from 08:00 to 18:00 on weekdays and drops to a trickle on weekends. A 95th-percentile threshold on raw request count will fire dozens of times each weekday during the lunch-hour rush, yet those spikes are perfectly normal.
Statistically, the distribution of such a metric is multimodal: weekday peaks form one cluster, weekend lows another. A single-threshold model assumes a unimodal, symmetric distribution, so it is fundamentally misspecified. Empirical studies report false-positive rates climbing above 40% when raw thresholds confront strong weekly seasonality.
Why do teams still cling to static limits? The answer is convenience. Setting a hard cut-off requires no data preprocessing, no domain expertise, and no additional infrastructure. It is the path of least resistance, especially when monitoring tools expose a simple "alert if > X" field. The price of that convenience is hidden in the noise: engineers waste hours investigating predictable load patterns, and real incidents are buried under a mountain of alerts. The solution starts by understanding that not all variance is anomalous. Some of it is seasonality, some is trend, and only the remainder is truly unexpected. But before we can isolate that remainder, we must make a critical choice about how to model it.
Additive vs Multiplicative: The Critical First Decision
The first decision is choosing between additive and multiplicative decomposition. This choice determines whether the seasonal component is modeled as a fixed offset or a proportional factor. Getting it wrong contaminates the residuals, leaving traces of seasonality that drown out real signals. When we model correctly, however, we extract a clean residual stream where true anomalies stand out.
The additive model assumes
[ Y_t = T_t + S_t + R_t, ]
where the magnitude of the seasonal swing stays roughly constant regardless of the trend level. This fits metrics like server request counts that oscillate by a similar number of requests each hour, even as overall traffic grows.
The multiplicative model assumes
[ Y_t = T_t \times S_t \times R_t, ]
where the seasonal effect scales with the baseline. If the trend doubles, the seasonal peaks double as well. This is typical for revenue, user sign-ups, or any metric where growth amplifies both baseline and cyclic variation.
How do we decide? Visual inspection is a quick first step: plot the seasonal sub-series (for example, all Mondays) and see whether its spread widens with the trend. A more quantitative heuristic is the coefficient of variation (CV) of those sub-series; a CV above 0.2 often signals multiplicative behavior. For a formal approach, fit both additive and multiplicative STL models and compare their AIC or BIC scores. The lower value indicates the better trade-off between fit and complexity.
Getting this decision wrong contaminates the residuals. If you force an additive structure on data that truly multiplies, the seasonal swing remains partially in the residual series, inflating its variance and causing either excessive false alarms or missed anomalies. Conversely, applying a multiplicative filter to additive-type data can over-shrink the residuals, making genuine spikes appear insignificant. Model selection is not a theoretical nicety; it is the first operational step that dictates detection quality.
Choosing the correct form prevents seasonal leakage into the residuals. A practical rule of thumb: if the height of the seasonal peaks stays roughly the same as overall traffic grows, adopt the additive model. Server request counts on a stable fleet often fit this pattern. Conversely, if the peaks grow together with the baseline (think of monthly revenue that rises as the user base expands), use the multiplicative model. The holiday-season spike in e-commerce sales, for example, becomes larger each year as the platform scales.
Diagnostic steps:
- Visual inspection: plot the seasonal sub-series; look for widening bands as the trend rises.
- Coefficient of variation: compute the CV of each seasonal slice; a CV > 0.2 suggests multiplicative behavior.
- Model comparison: fit both additive and multiplicative STL versions and compare information criteria (AIC or BIC); the lower score wins.
Selecting the wrong form leaves a trace of seasonality in the residuals, which either inflates false alarms or masks real outliers. The decision therefore sits at the very top of the detection pipeline, shaping every downstream metric. When we model correctly, however, we extract a clean residual stream where true anomalies stand out.
Building Robust Thresholds with MAD
With clean residuals in hand, the detection question becomes straightforward: where do we draw the line? The operational rule is crystal clear: apply anomaly thresholds only to the residual component. The residuals represent the part of the signal that cannot be explained by trend or seasonality, so any statistically significant deviation is worthy of attention.
Imagine measuring CPU usage at 10 a.m. on a typical workday. The raw value might be 78%, and a naïve Z-score calculation could return 3.2 σ, prompting an alert. In reality, the 78% is exactly what the daily pattern predicts; the residual at that timestamp is near zero. A common misconception is that removing the trend alone ("detrending") is sufficient. In practice, seasonal cycles still dominate the variance, so thresholds applied to detrended data continue to trigger on expected peaks. Only by stripping both trend and seasonality do we obtain a clean residual distribution that justifies statistical bounds. Because residuals are expected to be centered around zero with relatively narrow spread, simple linear thresholds (Z-score bands, IQR fences, or robust alternatives) become meaningful. The detection step therefore shifts from "does this raw value look high?" to "does this residual look unusually large?"
Median Absolute Deviation (MAD) is often the production-default method for constructing thresholds on residual components. The formula is straightforward: threshold = median(residuals) ± k × MAD, where MAD = median(|residuals - median(residuals)|).
MAD's advantage over standard deviation stems from its robustness to outliers. Standard deviation squares deviations from the mean, meaning a single extreme residual point inflates the threshold width significantly. In anomaly detection systems, this creates a perverse dynamic: genuine anomalies widen the threshold, making it harder to detect subsequent anomalies. MAD uses median operations, which are immune to outlier magnitude. A residual of 10,000 contributes the same to MAD as a residual of 1,000; only the sign matters, not the magnitude.
For heavy-tailed residual distributions, this robustness is essential. When your baseline includes occasional genuine anomalies (which it will, even in healthy systems), MAD prevents those anomalies from corrupting the threshold calibration. Standard deviation would require you to exclude known anomalies from baseline calculation; MAD handles them automatically.
Practical calibration requires choosing the multiplier k. The range of 3 to 6 covers most production scenarios. Lower k values increase sensitivity (more alerts, more false positives). Higher k values reduce false positives (fewer alerts, more missed anomalies). The appropriate value depends on your metric's residual behavior and your operational tolerance for alert volume.
Calibration against historical baselines enables principled selection. Split your historical data into training and validation windows. For each candidate k value, compute thresholds on training residuals, apply to validation windows, and measure precision, recall, and F1-score against labeled anomaly events. Grid search across k values (for example, 2.5, 3.0, 3.5, up to 6.0) identifies the value optimizing your target metric. For most monitoring scenarios, F1-score optimization balances sensitivity and specificity effectively." data-type="Needs Verification">Calibration against historical baselines enables principled selection. Split your historical data into training and validation windows. For each candidate k value, compute thresholds on training residuals, apply to validation windows, and measure precision, recall, and F1-score against labeled anomaly events. Grid search across k values (for example, 2.5, 3.0, 3.5, up to 6.0) identifies the value optimizing your target metric. For most monitoring scenarios, F1-score optimization balances sensitivity and specificity effectively.
Streaming computation requires adaptation. Recomputing MAD from scratch over a sliding window is computationally expensive and introduces lag. Instead, maintain exponential moving statistics: update the median and MAD using exponential decay weights, achieving O(1) update complexity per observation. The implementation maintains running estimates that adapt to gradual shifts in the residual distribution without storing the full history. This approach keeps the threshold responsive while avoiding the overhead of batch recalculation.