Anomaly Detection
Anomaly detection answers a different question from a conventional alert:
- a threshold rule asks whether a known limit was crossed;
- an anomaly rule asks whether this value is unusual for this entity, given what obserae previously learned about it.
That distinction matters. Sending 500 MB may be routine for a backup server and
exceptional for a workstation. An anomaly rule does not remove thresholds
altogether: it replaces one global threshold in raw units with a threshold in
standardised deviation (k).
This guide takes you from a traffic hypothesis to a detector you can operate. By the end, you should be able to:
- design a useful numeric signal in NFQL;
- explain exactly what the detector learns for each entity;
- choose between EWMA, Median + MAD and Seasonal baselines;
- predict the effect of every tuning parameter;
- distinguish a statistical deviation from an alert worth investigating;
- diagnose the common causes of noisy, blind or permanently-learning rules.
Prerequisite. An anomaly query normally ends in
STATS … BY …: one numeric value per entity. If aggregation is new to you, read Aggregation in the NFQL guide first.
1. The mental model
An anomaly rule needs three things:
| Concept | Meaning | Example |
|---|---|---|
| Entity | The thing that gets its own baseline. | client_ip |
| Metric | The numeric value observed for that entity on each run. | out = SUM(client_to_server_bytes) |
| Cadence | How often a new observation is produced. | every 15 minutes |
For example:
FROM sessions | LAST 900
| WHERE server_ip == "internet4" OR server_ip == "internet6"
| STATS out = SUM(client_to_server_bytes) BY client_ip
With Metric = out and Group by = client_ip, obserae learns one
outbound-volume baseline per client. client_to_server_bytes is the inferred
client-to-server direction, so Internet downloads are not mistaken for uploads.
Observation, baseline, breach and alert are different
These terms are easy to conflate:
- An observation is the numeric value returned for one entity on one rule run.
- A baseline is the normal centre and spread learned from earlier observations of that entity.
- A raw breach occurs when the new value is more than
kspreads from that prior centre. - An alert is a raw breach that also passes direction, dead-band, persistence and cooldown controls.
The chart can therefore show an anomalous point that did not page anyone. That is expected: the chart shows statistical breaches; rings identify the alerts that were actually emitted.
Shadow first, live when validated
An anomaly rule has two notification modes:
- Shadow runs the real evaluator, learns the real per-entity baselines and records evaluations that would have fired, but creates no alert, incident or cooldown state. This is the safe default for rules created in the interface.
- Live uses the same model and decision path, then creates alerts and incidents when an eligible breach occurs.
Use the Lab’s single-entity explanation and Validate all entities first, then leave the rule in Shadow long enough to encounter representative quiet and busy periods. Switching between Shadow and Live does not itself erase learning; changing the signal or statistical parameters does, because the old baseline no longer describes the configured detector. Shadow evaluations intentionally show eligible breaches before live cooldown: treat their rate as an upper bound on notification load, not as a count of incidents that were sent.
What happens on every rule run
For each row returned by the query, obserae performs this sequence:
- Build the entity key from the configured Group by columns.
- Read the numeric Metric. A
NULLor non-numeric value is ignored. - Apply the configured transform, if any.
- Compare the value with the entity’s previous baseline.
- During warm-up, learn silently. Afterwards, compute a signed z-score.
- If
|z| > k, mark a two-sided raw breach and freeze the baseline. Otherwise, fold the observation into the baseline. - Apply the alert-emission controls: direction, dead-band, persistence, then cooldown.
- In Shadow mode, record the eligible decision but stop before alert and incident creation.
The comparison always uses the baseline from before the current observation. The detector never judges a value against a centre that already includes that same value.
The most important query rule: no result row means no observation, not an observation of zero. If a host produced no matching session, a grouped
COUNT(*) BY client_ipquery produces no row for it. Its baseline is left unchanged. A low-direction anomaly cannot detect a disappearance unless the query still returns that entity with an explicit zero. Use a Heartbeat rule when disappearance itself is the condition you need to detect.
A numerical example
Assume one host has a prior centre of 20 and a spread of 4. The next
observation is 35:
z = (35 - 20) / 4 = 3.75
With k = 3, this is a raw high-side breach because 3.75 > 3.
- Direction = High allows it; Direction = Low suppresses the alert.
- Persistence = 2 normally waits for a second consecutive breach.
- The baseline remains at its prior value because the point breached the two-sided statistical band, even if an emission control suppressed the alert.
- Cooldown can suppress another notification, but it does not make the point normal.
This separation keeps notification policy from contaminating what the detector learns.
2. Design the signal before tuning the algorithm
Most disappointing anomaly rules are built on a poor signal, not a poor value
of k. Start with an explicit behavioural hypothesis:
For each entity, measure one behaviour over one meaningful period; alert when it departs materially from its own history.
Examples:
| Hypothesis | Entity | Metric |
|---|---|---|
| A client is uploading unusually much data to the Internet. | client_ip | SUM(client_to_server_bytes) |
| A client is reaching unusually many Internet peers. | client_ip | COUNT_DISTINCT(server_ip) |
| A source is probing unusually many ports on one target. | client_ip, server_ip | COUNT_DISTINCT(server_port) |
Five properties of a good signal
- It has one clear meaning. Do not mix uploads, downloads and internal
traffic into a metric named
volumeif only exfiltration matters. - The entity is stable.
client_ipis often useful; grouping by an ephemeral Internet peer can create excessive cardinality and baselines with too little history. - One row represents one entity. The query should return the group-by columns and one numeric metric per entity.
- The scale is reasonably stable. Counts and byte volumes are usable, but normally need a transform because their spread grows with their level.
- The query returns enough observations. An entity seen once a day cannot warm up a ten-sample baseline in an hour.
Cadence and query window define the observation
The query window answers “how much traffic goes into one value?”; cadence answers “how often is that value measured?”. Treat them as part of the algorithm.
LAST 900with a 15-minute cadence produces adjacent 15-minute observations.LAST 3600with a 15-minute cadence produces heavily overlapping values. A persistence of3then means three overlapping measurements, not three independent hours.LAST 300with a 15-minute cadence leaves gaps and may miss activity.
The evaluator also applies the rule’s maximum lookback bound. An explicit
shorter LAST remains shorter; an over-long window is clamped. See
Alerting for cadence, lookback and cooldown configuration.
The built-in anomaly detectors deliberately align their LAST window with
their cadence.
Do not filter away the values the baseline must learn
This query is appropriate for an interactive hunt:
FROM sessions | LAST 900
| STATS peers = COUNT_DISTINCT(server_ip) BY client_ip
| HAVING peers > 20
It is a poor anomaly-rule source. Values from 0 to 20 never reach the
detector, so the baseline learns only the already-large tail. For an anomaly
rule, remove the HAVING and use the rule’s direction and dead-band controls to
decide what deserves an alert.
WHERE is different: it defines the population being measured. It is correct
to filter to Internet destinations for an exfiltration signal. Remember,
however, that an entity with no matching row is absent rather than zero.
Start with the smallest useful grouping key
Every distinct key owns independent state:
client_ip one baseline per client
client_ip, server_ip one baseline per client-target pair
client_ip, server_ip, server_port one baseline per client-target-service tuple
More dimensions make the detector more specific but divide the history into smaller series. They also increase memory and warm-up time. Use composite keys for behaviours whose meaning really depends on the pair or tuple, such as a vertical port scan.
Max keys bounds the number of result entities accepted by a rule. If one run
exceeds the cap, per-entity evaluation is skipped for that run and a throttled
meta-alert tells you to narrow the query or raise the cap. The rule editor
defaults to 50,000; Seasonal state is much larger per entity, so use a
deliberately lower cap when possible.
3. Use statistical operators to discover a signal
NFQL statistical operators help you inspect traffic before automating it. They
are aggregates used in STATS; they measure the rows inside each group and
return numeric values that can be explored with HAVING.
| Operator | What it measures | Practical use |
|---|---|---|
COUNT_DISTINCT(col) | Number of different values. | peers, ports, users, services |
ENTROPY(col) | Shannon entropy of a numeric value distribution, in bits. | how evenly traffic is spread over ports or numeric categories |
STDDEV(col) | Sample standard deviation around the mean. | stable versus variable activity |
MAD(col) | Raw median absolute deviation from the median. | robust spread during exploration |
MEDIAN(col) | Middle value. | robust typical value |
PERCENTILE(col, n) | Continuous percentile, where n is 1..99. | tails such as p95 volume |
SKEWNESS(col) | Sample asymmetry. | diagnosing a long left or right tail |
KURTOSIS(col) | Excess kurtosis. | diagnosing heavy tails and rare extremes |
All except COUNT_DISTINCT take a numeric column. In particular,
ENTROPY(server_ip) is invalid because an IP address is not numeric. Use
COUNT_DISTINCT(server_ip) for destination breadth, or entropy over a numeric
field such as server_port.
Distinct count and entropy answer different questions
FROM sessions | LAST 3600
| STATS distinct_ports = COUNT_DISTINCT(server_port),
port_entropy = ENTROPY(server_port) BY client_ip
| SORT distinct_ports DESC
COUNT_DISTINCT measures breadth. ENTROPY also considers frequency:
- one value only has entropy
0bits; nequally frequent values havelog2(n)bits;- many rare values plus one dominant value have lower entropy than the same values used evenly.
A client making 1,000 connections to port 443 and one to port 22 has two
distinct ports but entropy near zero. A source touching 200 ports once each has
both a high distinct count and high entropy. Use data from your environment to
choose a fixed entropy threshold; values such as 3 or 4 bits are not
universal security boundaries.
Standard deviation and MAD measure spread differently
STDDEV is sensitive to extremes because it is centred on the mean and squares
deviations. MAD is centred on the median and uses absolute deviations, so one
large outlier moves it much less.
FROM sessions | LAST 86400
| STATS n = COUNT(*) BY client_ip, BUCKET(opened_at, 300)
| STATS buckets = COUNT(*), average = AVG(n),
sd = STDDEV(n), mad = MAD(n) BY client_ip
| HAVING buckets >= 24
| SORT sd ASC
This can surface clients whose non-empty five-minute buckets are unusually regular. It is only a candidate-beaconing view: buckets with no matching row are absent, so this does not calculate true inter-arrival time or insert zero-count buckets.
Do not confuse two MADs. NFQL’s
MAD(col)returns the raw median absolute deviation for exploration. The Median + MAD anomaly baseline multiplies MAD by1.4826so that, on roughly normal data, its spread is comparable to a standard deviation.
STDDEV, SKEWNESS and KURTOSIS are undefined on very small samples. Do not
interpret their output until each group contains enough values; use a count in
the same STATS pipeline to enforce that requirement.
Statistical operators and anomaly rules compose naturally: first use the
operators to find a meaningful metric, then let an anomaly rule baseline that
metric per entity. The exploratory query may use HAVING; the automated
baseline query normally should not.
4. How the anomaly score is calculated
Let x be the raw observation and T the configured transform:
y = T(x)
z = (y - centre) / spread
raw breach when |z| > k
The centre and spread are stored in the transformed working space. The chart converts the centre and band back into raw units, which is why a transformed band can be asymmetric around the displayed centre.
The sign of z has operational meaning:
z > 0: above normal;z < 0: below normal;|z|: distance from normal in learned spreads.
k is a distance, not a guaranteed false-positive rate
With ideal independent Gaussian data, k = 3 recalls the familiar 3-sigma
range containing about 99.7% of observations. Network metrics are rarely ideal
Gaussian samples: counts are discrete, volumes are right-skewed, windows can
overlap, and traffic has daily structure.
Treat k = 3 as a starting distance:
- increase
kfor fewer, more extreme raw breaches; - decrease
kfor subtler deviations and more noise; - do not tune
kbefore fixing the metric’s shape, baseline method and cadence.
A flat baseline is a special case. If the learned spread is almost zero, even a
one-unit change is an enormous z-score. Raising k is usually the wrong fix;
use a spread floor or choose a deterministic threshold for a truly invariant
metric.
5. Choose the baseline method
All three methods are per entity, warm up silently, judge the current value against prior state and freeze on a raw two-sided breach. They differ in what they consider “normal”.
EWMA: a continuously adapting baseline
EWMA keeps a transformed mean, variance and sample count. For a non-breaching
observation y:
new_mean = old_mean + alpha * (y - old_mean)
new_variance = (1 - alpha) *
(old_variance + alpha * (y - old_mean)^2)
Use EWMA when the metric changes gradually and does not have a strong weekly schedule. It is the default and usually the first method to try.
Strengths: small fixed state, smooth adaptation, simple tuning.
Limits: it does not know the time of day. A large value that occurs during warm-up, or remains inside a broad band, can pull the baseline. A value that actually breaches is frozen and does not pull it.
Understand alpha as memory
alpha is the weight of the newest non-breaching sample. Its approximate
half-life in observed samples is:
half_life = ln(0.5) / ln(1 - alpha)
alpha | Approximate half-life | Behaviour |
|---|---|---|
0.05 | 13.5 samples | slow, stable adaptation |
0.10 | 6.6 samples | balanced starting point |
0.20 | 3.1 samples | fast adaptation |
0.30 | 1.9 samples | very short memory |
Multiply by cadence only when the entity produces a row on every run. With a
15-minute cadence, alpha = 0.1 has a half-life near 100 minutes for a
continuously present entity, but much longer for a sparse one.
Lower alpha resists slow drift but takes longer to accept a legitimate regime
change. Higher alpha follows a new regime faster but can learn a slow attack as
normal. Pair anomaly detection with an absolute threshold when a hard risk limit
must never be learned away.
Median + MAD: a robust sliding window
This method keeps the last N accepted transformed observations:
centre = median(window)
spread = 1.4826 * median(|value - centre|)
Use Median + MAD when the baseline may be contaminated by isolated large but non-breaching observations, especially during warm-up, and you want the centre to resist them.
Strengths: robust centre and spread; one extreme value has little influence on the next decision.
Limits: more state per entity; no awareness of time of day; a quiet or discrete series can make MAD collapse to zero unless a spread floor is set.
Window N ranges from 8 to 256 and defaults to 32:
- larger
N: steadier, more robust, slower to adapt; - smaller
N: more reactive, less stable; - warm-up samples cannot exceed
N.
The window covers N × cadence only for an entity observed on every run.
Median + MAD is not a general cure for alert noise. On right-skewed byte
volumes, its robust spread can be narrower than EWMA’s spread and generate more
breaches. On a legitimate recurring backup, freeze-on-breach prevents the burst
from ever entering the window, so it can alert forever. Use log1p for skewed
volumes and Seasonal for a real weekly rhythm.
Seasonal: one EWMA per hour and weekday
Seasonal keeps up to 168 independent EWMA baselines per entity:
7 weekdays x 24 hours = 168 time-of-week slots
An observation is compared only with previous observations from the same UTC weekday and UTC hour. A high Monday-morning value can be normal while the same value on Sunday night remains anomalous.
Use Seasonal when a legitimate daily or weekly pattern is the cause of false positives.
Strengths: models repeated time-of-week behaviour that EWMA and Median + MAD cannot represent.
Limits: every slot warms up independently, state is much larger, and a schedule tied to local civil time may shift relative to UTC around daylight saving changes.
Warm-up can take weeks. At a 15-minute cadence, a continuously present entity can contribute about four samples to a given hourly slot each week; a warm-up of 10 therefore needs roughly three visits to that week/hour. Sparse entities take longer. A Seasonal rule that shows no alerts but has low slot coverage is not yet proven quiet; it may simply be unprimed.
Method decision table
| Traffic shape | Start with | Main tuning control |
|---|---|---|
| Gradual change, no weekly rhythm | EWMA | alpha |
| Isolated baseline contamination or outliers | Median + MAD | Window N |
| Repeated hour-of-week pattern | Seasonal | alpha, plus enough history |
| Hard invariant or safety limit | Threshold rule | raw threshold |
Changing the baseline method from the Anomaly Detection page resets learned state and restarts warm-up. That reset is necessary because EWMA scalars, a MAD window and a Seasonal grid are not interchangeable.
6. Make the statistical model fit the metric
Choose the transform and spread floor before fine-tuning k.
Transform: correct the distribution’s shape
| Metric shape | Transform | Typical examples |
|---|---|---|
| Approximately symmetric with stable variance | None | ratios or already-normalised scores |
| Non-negative discrete count | sqrt | COUNT, COUNT_DISTINCT |
| Non-negative right-skewed magnitude | log1p | byte or packet sums |
The transform changes the space in which centre, variance, MAD and z-score are calculated. The chart inverts the result back to raw units.
sqrtreduces the level-dependent variance common in count data.log1p(x) = log(1 + x)handles zero and compresses a long volume tail.- Both transforms clamp negative input to zero. Do not use them for a signed metric whose negative values carry meaning.
The std.anomaly detectors already use sqrt for counts and log1p for byte
volume.
Spread floor: prevent a collapsed denominator
The Spread floor is configured in raw metric units. A floor of 1 for a
count says the detector should not trust its learned spread to be finer than one
count unit. obserae converts that floor into the estimator’s working space at
the current level.
Use it when a metric is discrete or nearly constant:
- count quantum
1→ start with a floor of1; - a measurement rounded to 10 MB → consider a floor matching that meaningful resolution;
0disables the floor.
The floor changes the statistical denominator and therefore the raw breach decision. It is different from a dead-band, which acts only after a breach.
7. Decide which breaches deserve an alert
Once the baseline is credible, emission controls turn statistical breaches into operationally useful alerts.
| Control | Default | Effect |
|---|---|---|
| Direction | Both | emit high breaches, low breaches or both |
| Dead-band | 0 off | require a minimum raw-unit deviation from the centre |
| Persistence | 1 off | require consecutive direction-matching breaches |
| Cooldown | rule setting | suppress repeated notifications for the same entity |
Direction
Use High for signals where only growth is suspicious: bytes out, peer count, port count. Use Low when a fall is meaningful and the query still returns a numeric row.
Direction filters alert emission, not baseline protection. A large low-side breach is still frozen even on a High-only rule.
A drop to zero is visible only if zero is returned. If the entity disappears from the grouped result, no anomaly decision runs. Use Heartbeat for that semantic.
Dead-band
A dead-band suppresses statistically large but operationally trivial changes.
For example, a count moving from 1 to 3 can be many learned spreads away but
still not warrant paging.
Unlike HAVING, dead-band does not remove the observation before the detector
sees it. The raw breach remains frozen and tracked.
Persistence
Persistence requires consecutive raw breaches in the selected direction. A normal point or a breach in the other direction resets the streak. Be careful with overlapping query windows: three consecutive values may contain much of the same underlying traffic.
obserae includes two safety behaviours:
- a material breach at least
2 × kspreads from the centre bypasses persistence and fires immediately; - a breach below the dead-band escalates after
10 × persistenceconsecutive breaches, preventing permanent suppression of a low-and-slow deviation.
The immediate bypass must also exceed the dead-band in raw units, so a collapsed spread cannot turn a trivial one-unit change into a “massive” alert.
Parameter quick reference
| Parameter | Default in the editor | Valid values | When increased |
|---|---|---|---|
k | 3 | > 0 and <= 50 | fewer, more extreme raw breaches |
alpha | 0.1 | strictly between 0 and 1 | faster adaptation, shorter memory |
Window N | 32 | 8..256 | steadier MAD baseline, slower adaptation |
| Warm-up | 10 | at least 2; no more than N for MAD | longer silent learning period |
| Spread floor | 0 off | raw value >= 0 | wider minimum statistical band |
| Dead-band | 0 off | raw value >= 0 | larger material deviation required |
| Persistence | 1 off | integer >= 1 in the editor | more consecutive breaches required |
Severity does not change the score or firing decision. It classifies an emitted alert for prioritisation and routing.
Freeze-on-breach and cooldown
Any primed two-sided raw breach freezes the baseline, even when direction, dead-band, persistence or cooldown prevents an alert. This has two consequences:
- notification policy never teaches the model that a deviation is normal;
- a genuine permanent regime change can continue breaching indefinitely.
After confirming a permanent change is legitimate, establish a fresh baseline
rather than increasing k until the old and new regimes fit one band.
8. Warm-up, missing data and baseline lifecycle
Warm-up counts observations, not elapsed time
The UI default is 10 samples; the built-in detectors use 10 or 12. During warm-up, values are learned and never alert.
For a continuously present entity:
approximate EWMA/MAD warm-up time = samples x cadence
For a sparse entity it takes longer because absent runs contribute nothing. For Seasonal, the sample requirement applies separately to each time-of-week slot.
Raise warm-up when initial data is unrepresentative. Do not raise it merely to silence a bad signal: that postpones the problem instead of fixing it.
Baselines survive normal restarts
Learned baselines are restored after restart, so a routine deployment does not normally send every entity back through cold start. Recent unsnapshotted learning can be lost after an abrupt crash, but the rule simply relearns it.
Per-entity state stays bounded by Max keys, the selected estimator’s fixed state size and optional retention. Any edit that changes the signal or its statistical decisions resets the rule’s learned state centrally, whether the edit comes from the GUI, the Lab or the API. Deleting a rule removes its learned state.
Treat structural signal changes as a new model
Changing the query, metric, group-by key, cadence, transform, method or any
statistical decision setting (k, direction, dead-band or persistence) starts a
new learning cycle. This is necessary because freeze-on-breach means even a
threshold change can alter which observations the old model absorbed. Editing
operational metadata such as the name, severity, cooldown or remediation keeps
the learned model.
For a structural change, prefer duplicating the rule and letting the new version warm up before retiring the old one. This preserves coverage and prevents old state from being interpreted under a new signal definition.
9. Validate with Anomaly rule preview
Analysis → Anomaly Lab opens Anomaly rule preview. It replays retained data through the same estimator used by the alert engine. Use it before routing a new rule to an output.
The guided view deliberately separates What is unusual? (sensitivity and
adaptation) from When should we notify? (direction and consecutive unusual
observations). Show expert controls reveals the exact estimator, k,
alpha, MAD window, transform, spread floor and emission gates. Both views edit
the same configuration; guided mode is not a separate simplified engine.
You can start from an existing anomaly rule, a saved query or ad-hoc NFQL. Pick
a representative entity, then inspect how method, k, alpha or N, warm-up,
transform, floor and emission controls change the band and fire count.
A disciplined tuning workflow
- Validate the signal in Investigation. Check units, direction, entity key
and expected row cardinality. Remove
HAVINGfrom the baseline query. - Use the engine cadence. The Lab defaults to the rule cadence when possible. A different replay bucket represents a different series.
- Test several entities. Include a quiet endpoint, a busy endpoint and a known unusual period. One “easy” host is not representative.
- Choose transform and floor. Use
sqrtplus floor1for counts;log1pfor byte volume; confirm the displayed band is plausible. - Choose the baseline method. Use the traffic shape, not the lowest alert count, to select EWMA, MAD or Seasonal.
- Set warm-up. Ensure the replay actually leaves warm-up and, for Seasonal, has meaningful slot coverage.
- Tune
k. Balance missed behaviour against the number of alerts your team can investigate. - Add emission controls last. Direction first, then dead-band for material impact, then persistence for isolated blips.
- Validate on a wider period. Include weekdays, weekends, maintenance and known incidents where possible.
The preview first spells the signal out, including its entity key and the fact that a missing query row is unknown, not zero. It reports the selected entity’s simulated detector emissions per observed wall-clock day, the separate data/model coverage, raw anomalies, notifications, peak z-score and how the estimator’s learned spread compares with the series. It does not report accuracy: without labelled incidents, missed attacks and false positives cannot be measured. Per-entity cooldown is applied later by the live evaluator, so routed alert volume can be lower. The Lab refuses to rank a configuration that stayed mostly in warm-up: zero alerts without coverage is not a successful detector.
The chart uses four separate outcomes:
- Unusual observation — the model judged the point outside its expected range. Alert policy does not rewrite this statistical fact.
- Suppressed by policy — direction, dead-band or persistence withheld the
notification. A persistence marker shows progress such as
1/3. - Simulated notification — the current preview settings would notify. An immediate severity bypass is named explicitly.
- Recorded notification — the historical alert journal says a live rule actually notified; this is evidence, not a reconstructed decision.
Every unusual observation also has a decision sentence with the observed value, exact expected bounds, policy reason, persistence progress and severity-bypass state. Hovering the chart shows the same server-computed z-score and policy decision. After any setting change, Effect of … compares unusual observations, policy suppressions, simulated notifications and immediate bypasses before and after. It is therefore expected for persistence to leave the unusual-observation count unchanged while reducing notifications.
The response labels this result selected_entity, simulated and
bucketed_historical_approximation; it never presents one entity as fleet-wide
alert volume.
After inspecting representative entities, select Validate all entities. This runs one bounded scan and reports whole-signal notification load, raw anomalies, data coverage, model coverage and the noisiest entities. If the scan cap is reached, the result is marked partial rather than silently extrapolated.
Understand replay limits
The Lab and Anomaly Detection charts reconstruct a series from retained traffic; obserae does not store every historical baseline decision.
- The Lab uses the rule cadence by default, but widens very small buckets on long horizons to keep replay bounded.
- The Anomaly Detection chart chooses a display bucket that fits the selected window. Its reconstructed breach markers may therefore differ from live decisions made at the actual cadence.
- Journalled alerts are authoritative; reconstructed points are explanatory.
- Replay works for queries ending in a plain
STATS <metric> = <aggregate>(…) BY <entity>. Complex pivot cascades, trailingHAVING/SORT, or sources without a time column may not be replayable.
Applying Lab settings writes them to an editable rule. Pack-owned rules are read-only: duplicate one before tuning it.
10. Diagnose a rule from its symptoms
| Symptom | Likely explanation | Corrective action |
|---|---|---|
| Rule never leaves warm-up | entity is absent on many runs, metric is NULL, cadence is slow, or Seasonal slots lack visits | inspect raw query rows; widen the population; wait for real coverage |
| Expected zero/drop is missed | the group disappeared, so no zero row was produced | use Heartbeat or redesign the query to emit explicit zeros |
Count metric fires on 1 -> 2 | discrete metric, collapsed spread | use sqrt and spread floor 1; add a raw dead-band if the change is immaterial |
| Byte-volume rule is noisy | right-skewed heavy tail judged in raw space | use log1p before raising k |
| Every morning or Monday fires | legitimate time-of-week rhythm | use Seasonal; remember slots are UTC and need weeks of coverage |
| Recurring backup still fires under MAD | frozen recurring burst never enters the robust window | use Seasonal if scheduled, or separate that traffic from the signal |
| Slow drift is learned away | EWMA adapts too quickly | lower alpha; pair with a fixed threshold for hard limits |
| Accepted permanent regime keeps firing | freeze-on-breach anchors the old baseline | warm up a fresh rule for the new regime |
| Many alerts remain during cooldown | cooldown is per entity; many different keys are firing | inspect grouping cardinality and the activity heatmap |
Meta-alert says max_keys exceeded | query returned more entities than the configured cap | narrow the population, simplify group-by or deliberately raise the cap |
| Chart points do not align with fire rings | reconstructed display bucket differs from live cadence | replay in Anomaly Lab at the engine cadence; trust journalled fires |
| Persistence changes no yellow points | yellow means statistically unusual; persistence is notification policy | read the before/after panel and the suppressed/simulated markers |
Raising k changes almost nothing | spread is nearly zero, or breaches are extremely far out | set a spread floor, transform the metric, or use a threshold rule |
When noise appears, debug in this order:
query meaning -> missing rows -> cadence/window -> transform/floor
-> baseline method -> warm-up -> k -> emission controls
Changing k first often hides the symptom without fixing the model.
11. Built-in anomaly detectors
A fresh install has no active anomaly rule. Install std.anomaly from
Rule Sets to add nine maintained detectors:
| Detector | Entity and metric | Default metric treatment |
|---|---|---|
| Adaptive exfiltration | upload bytes per client | log1p, EWMA |
| Egress fan-out | distinct Internet peers per client | sqrt, floor 1, EWMA |
| DNS exfiltration | DNS sessions per client | sqrt, floor 1, EWMA |
| Lateral spread | distinct internal peers per client | sqrt, floor 1, EWMA |
| Lateral admin surge | distinct internal admin peers per client | sqrt, floor 1, EWMA |
| Port scan (vertical) | distinct ports per client-target pair | sqrt, floor 1, EWMA |
| Host sweep (horizontal) | distinct targets per client-port pair | sqrt, floor 1, EWMA |
| Half-open surge | no-reply targets per client | sqrt, floor 1, EWMA |
| Auth brute force | admin/auth sessions per client-target pair | sqrt, floor 1, EWMA |
The pack uses inferred client_ip / server_ip roles and directional byte
columns, so canonical storage order is never confused with client/server
direction. Its windows match its 5- or 15-minute cadences, and its warm-up is
10 or 12 observations. The shipped rules leave Direction at its default,
Both; on a custom copy, High is often more operationally appropriate for
volume, peer and port counts if unusual drops do not matter to your use case.
Pack rules are read-only. You can enable or disable them, but duplicate a rule to adapt its query or parameters. The copy has its own baseline and should warm up before replacing the original.
Three starting recipes
Adaptive exfiltration
FROM sessions | LAST 900
| WHERE server_ip == "internet4" OR server_ip == "internet6"
| STATS out = SUM(client_to_server_bytes) BY client_ip
For a high-side custom detector, start with entity client_ip, metric out,
EWMA, log1p, k = 3, alpha = 0.1, warm-up 12, direction High,
cadence 15 minutes.
Lateral spread
FROM sessions | LAST 900
| WHERE server_ip == "internal4" OR server_ip == "internal6"
| STATS peers = COUNT_DISTINCT(server_ip) BY client_ip
For a high-side custom detector, start with entity client_ip, metric peers,
EWMA, sqrt, spread floor 1, k = 3, alpha = 0.1, warm-up 10,
direction High, cadence 15 minutes.
Vertical port scan
FROM sessions | LAST 300
| STATS ports = COUNT_DISTINCT(server_port) BY client_ip, server_ip
For a high-side custom detector, start with entity client_ip, server_ip, metric
ports, EWMA, sqrt, spread floor 1, k = 3, alpha = 0.1, warm-up 10,
direction High, cadence 5 minutes. Watch cardinality: every client-target
pair owns a baseline.
These are starting configurations, not universal truth. Validate them against your traffic and alert budget in the Anomaly Lab.
12. Operate the Anomaly Detection page
Analysis → Anomaly Detection is the operational view for statistical rules. Deterministic rules remain on the Rules page.
The overview separates rules that are active, still learning or off, and shows tracked entities and recent fires. Open a rule to inspect:
- recent fires, with the entity and observed value;
- each entity’s learned baseline and warm-up state;
- an activity heatmap of deviation by entity and time;
- an observed-versus-expected chart with centre, band, raw breaches and emitted alerts;
- a Seasonal 24 x 7 grid when that method is used.
Clicking a fire opens the case at the relevant time. The alert’s recorded value and matched rows are authoritative. The surrounding band may be reconstructed from retained traffic, so the page labels when it is showing reconstructed or current-baseline context.
Entity labels add cartography identity and, for public addresses, available country, ASN, threat-feed and cloud context. This enrichment helps prioritise an alert but does not alter its statistical score.
Switching the baseline method inline asks for confirmation, clears learned state and starts warm-up again. Pack-owned rules offer Duplicate instead of Edit.
13. Which detection type should you use?
| Need | Use |
|---|---|
| Explore and understand a traffic distribution | NFQL statistical operators |
| Enforce a known limit shared by everyone | Threshold rule |
| Detect unusual behaviour relative to each entity | Anomaly rule |
| Detect that an expected entity or signal disappeared | Heartbeat rule |
| Detect a hard limit and behavioural drift | Threshold and Anomaly rules together |
A mature workflow is:
hypothesis -> exploratory NFQL -> stable metric -> Anomaly Lab
-> silent warm-up -> limited alert routing -> production tuning
Anomaly detection is strongest when it complements deterministic controls. It can discover deviations you did not know how to threshold, while fixed rules still enforce invariants that must never become normal.
See also: