Daily Use

Anomaly Detection

Anomaly detection answers a different question from a conventional alert:

  • a threshold rule asks whether a known limit was crossed;
  • an anomaly rule asks whether this value is unusual for this entity, given what obserae previously learned about it.

That distinction matters. Sending 500 MB may be routine for a backup server and exceptional for a workstation. An anomaly rule does not remove thresholds altogether: it replaces one global threshold in raw units with a threshold in standardised deviation (k).

This guide takes you from a traffic hypothesis to a detector you can operate. By the end, you should be able to:

  1. design a useful numeric signal in NFQL;
  2. explain exactly what the detector learns for each entity;
  3. choose between EWMA, Median + MAD and Seasonal baselines;
  4. predict the effect of every tuning parameter;
  5. distinguish a statistical deviation from an alert worth investigating;
  6. diagnose the common causes of noisy, blind or permanently-learning rules.

Prerequisite. An anomaly query normally ends in STATS … BY …: one numeric value per entity. If aggregation is new to you, read Aggregation in the NFQL guide first.


1. The mental model

An anomaly rule needs three things:

ConceptMeaningExample
EntityThe thing that gets its own baseline.client_ip
MetricThe numeric value observed for that entity on each run.out = SUM(client_to_server_bytes)
CadenceHow often a new observation is produced.every 15 minutes

For example:

FROM sessions | LAST 900
  | WHERE server_ip == "internet4" OR server_ip == "internet6"
  | STATS out = SUM(client_to_server_bytes) BY client_ip

With Metric = out and Group by = client_ip, obserae learns one outbound-volume baseline per client. client_to_server_bytes is the inferred client-to-server direction, so Internet downloads are not mistaken for uploads.

Observation, baseline, breach and alert are different

These terms are easy to conflate:

  • An observation is the numeric value returned for one entity on one rule run.
  • A baseline is the normal centre and spread learned from earlier observations of that entity.
  • A raw breach occurs when the new value is more than k spreads from that prior centre.
  • An alert is a raw breach that also passes direction, dead-band, persistence and cooldown controls.

The chart can therefore show an anomalous point that did not page anyone. That is expected: the chart shows statistical breaches; rings identify the alerts that were actually emitted.

Shadow first, live when validated

An anomaly rule has two notification modes:

  • Shadow runs the real evaluator, learns the real per-entity baselines and records evaluations that would have fired, but creates no alert, incident or cooldown state. This is the safe default for rules created in the interface.
  • Live uses the same model and decision path, then creates alerts and incidents when an eligible breach occurs.

Use the Lab’s single-entity explanation and Validate all entities first, then leave the rule in Shadow long enough to encounter representative quiet and busy periods. Switching between Shadow and Live does not itself erase learning; changing the signal or statistical parameters does, because the old baseline no longer describes the configured detector. Shadow evaluations intentionally show eligible breaches before live cooldown: treat their rate as an upper bound on notification load, not as a count of incidents that were sent.

What happens on every rule run

For each row returned by the query, obserae performs this sequence:

  1. Build the entity key from the configured Group by columns.
  2. Read the numeric Metric. A NULL or non-numeric value is ignored.
  3. Apply the configured transform, if any.
  4. Compare the value with the entity’s previous baseline.
  5. During warm-up, learn silently. Afterwards, compute a signed z-score.
  6. If |z| > k, mark a two-sided raw breach and freeze the baseline. Otherwise, fold the observation into the baseline.
  7. Apply the alert-emission controls: direction, dead-band, persistence, then cooldown.
  8. In Shadow mode, record the eligible decision but stop before alert and incident creation.

The comparison always uses the baseline from before the current observation. The detector never judges a value against a centre that already includes that same value.

The most important query rule: no result row means no observation, not an observation of zero. If a host produced no matching session, a grouped COUNT(*) BY client_ip query produces no row for it. Its baseline is left unchanged. A low-direction anomaly cannot detect a disappearance unless the query still returns that entity with an explicit zero. Use a Heartbeat rule when disappearance itself is the condition you need to detect.

A numerical example

Assume one host has a prior centre of 20 and a spread of 4. The next observation is 35:

z = (35 - 20) / 4 = 3.75

With k = 3, this is a raw high-side breach because 3.75 > 3.

  • Direction = High allows it; Direction = Low suppresses the alert.
  • Persistence = 2 normally waits for a second consecutive breach.
  • The baseline remains at its prior value because the point breached the two-sided statistical band, even if an emission control suppressed the alert.
  • Cooldown can suppress another notification, but it does not make the point normal.

This separation keeps notification policy from contaminating what the detector learns.


2. Design the signal before tuning the algorithm

Most disappointing anomaly rules are built on a poor signal, not a poor value of k. Start with an explicit behavioural hypothesis:

For each entity, measure one behaviour over one meaningful period; alert when it departs materially from its own history.

Examples:

HypothesisEntityMetric
A client is uploading unusually much data to the Internet.client_ipSUM(client_to_server_bytes)
A client is reaching unusually many Internet peers.client_ipCOUNT_DISTINCT(server_ip)
A source is probing unusually many ports on one target.client_ip, server_ipCOUNT_DISTINCT(server_port)

Five properties of a good signal

  1. It has one clear meaning. Do not mix uploads, downloads and internal traffic into a metric named volume if only exfiltration matters.
  2. The entity is stable. client_ip is often useful; grouping by an ephemeral Internet peer can create excessive cardinality and baselines with too little history.
  3. One row represents one entity. The query should return the group-by columns and one numeric metric per entity.
  4. The scale is reasonably stable. Counts and byte volumes are usable, but normally need a transform because their spread grows with their level.
  5. The query returns enough observations. An entity seen once a day cannot warm up a ten-sample baseline in an hour.

Cadence and query window define the observation

The query window answers “how much traffic goes into one value?”; cadence answers “how often is that value measured?”. Treat them as part of the algorithm.

  • LAST 900 with a 15-minute cadence produces adjacent 15-minute observations.
  • LAST 3600 with a 15-minute cadence produces heavily overlapping values. A persistence of 3 then means three overlapping measurements, not three independent hours.
  • LAST 300 with a 15-minute cadence leaves gaps and may miss activity.

The evaluator also applies the rule’s maximum lookback bound. An explicit shorter LAST remains shorter; an over-long window is clamped. See Alerting for cadence, lookback and cooldown configuration.

The built-in anomaly detectors deliberately align their LAST window with their cadence.

Do not filter away the values the baseline must learn

This query is appropriate for an interactive hunt:

FROM sessions | LAST 900
  | STATS peers = COUNT_DISTINCT(server_ip) BY client_ip
  | HAVING peers > 20

It is a poor anomaly-rule source. Values from 0 to 20 never reach the detector, so the baseline learns only the already-large tail. For an anomaly rule, remove the HAVING and use the rule’s direction and dead-band controls to decide what deserves an alert.

WHERE is different: it defines the population being measured. It is correct to filter to Internet destinations for an exfiltration signal. Remember, however, that an entity with no matching row is absent rather than zero.

Start with the smallest useful grouping key

Every distinct key owns independent state:

client_ip                         one baseline per client
client_ip, server_ip              one baseline per client-target pair
client_ip, server_ip, server_port one baseline per client-target-service tuple

More dimensions make the detector more specific but divide the history into smaller series. They also increase memory and warm-up time. Use composite keys for behaviours whose meaning really depends on the pair or tuple, such as a vertical port scan.

Max keys bounds the number of result entities accepted by a rule. If one run exceeds the cap, per-entity evaluation is skipped for that run and a throttled meta-alert tells you to narrow the query or raise the cap. The rule editor defaults to 50,000; Seasonal state is much larger per entity, so use a deliberately lower cap when possible.


3. Use statistical operators to discover a signal

NFQL statistical operators help you inspect traffic before automating it. They are aggregates used in STATS; they measure the rows inside each group and return numeric values that can be explored with HAVING.

OperatorWhat it measuresPractical use
COUNT_DISTINCT(col)Number of different values.peers, ports, users, services
ENTROPY(col)Shannon entropy of a numeric value distribution, in bits.how evenly traffic is spread over ports or numeric categories
STDDEV(col)Sample standard deviation around the mean.stable versus variable activity
MAD(col)Raw median absolute deviation from the median.robust spread during exploration
MEDIAN(col)Middle value.robust typical value
PERCENTILE(col, n)Continuous percentile, where n is 1..99.tails such as p95 volume
SKEWNESS(col)Sample asymmetry.diagnosing a long left or right tail
KURTOSIS(col)Excess kurtosis.diagnosing heavy tails and rare extremes

All except COUNT_DISTINCT take a numeric column. In particular, ENTROPY(server_ip) is invalid because an IP address is not numeric. Use COUNT_DISTINCT(server_ip) for destination breadth, or entropy over a numeric field such as server_port.

Distinct count and entropy answer different questions

FROM sessions | LAST 3600
  | STATS distinct_ports = COUNT_DISTINCT(server_port),
          port_entropy   = ENTROPY(server_port) BY client_ip
  | SORT distinct_ports DESC

COUNT_DISTINCT measures breadth. ENTROPY also considers frequency:

  • one value only has entropy 0 bits;
  • n equally frequent values have log2(n) bits;
  • many rare values plus one dominant value have lower entropy than the same values used evenly.

A client making 1,000 connections to port 443 and one to port 22 has two distinct ports but entropy near zero. A source touching 200 ports once each has both a high distinct count and high entropy. Use data from your environment to choose a fixed entropy threshold; values such as 3 or 4 bits are not universal security boundaries.

Standard deviation and MAD measure spread differently

STDDEV is sensitive to extremes because it is centred on the mean and squares deviations. MAD is centred on the median and uses absolute deviations, so one large outlier moves it much less.

FROM sessions | LAST 86400
  | STATS n = COUNT(*) BY client_ip, BUCKET(opened_at, 300)
  | STATS buckets = COUNT(*), average = AVG(n),
          sd = STDDEV(n), mad = MAD(n) BY client_ip
  | HAVING buckets >= 24
  | SORT sd ASC

This can surface clients whose non-empty five-minute buckets are unusually regular. It is only a candidate-beaconing view: buckets with no matching row are absent, so this does not calculate true inter-arrival time or insert zero-count buckets.

Do not confuse two MADs. NFQL’s MAD(col) returns the raw median absolute deviation for exploration. The Median + MAD anomaly baseline multiplies MAD by 1.4826 so that, on roughly normal data, its spread is comparable to a standard deviation.

STDDEV, SKEWNESS and KURTOSIS are undefined on very small samples. Do not interpret their output until each group contains enough values; use a count in the same STATS pipeline to enforce that requirement.

Statistical operators and anomaly rules compose naturally: first use the operators to find a meaningful metric, then let an anomaly rule baseline that metric per entity. The exploratory query may use HAVING; the automated baseline query normally should not.


4. How the anomaly score is calculated

Let x be the raw observation and T the configured transform:

y = T(x)
z = (y - centre) / spread
raw breach when |z| > k

The centre and spread are stored in the transformed working space. The chart converts the centre and band back into raw units, which is why a transformed band can be asymmetric around the displayed centre.

The sign of z has operational meaning:

  • z > 0: above normal;
  • z < 0: below normal;
  • |z|: distance from normal in learned spreads.

k is a distance, not a guaranteed false-positive rate

With ideal independent Gaussian data, k = 3 recalls the familiar 3-sigma range containing about 99.7% of observations. Network metrics are rarely ideal Gaussian samples: counts are discrete, volumes are right-skewed, windows can overlap, and traffic has daily structure.

Treat k = 3 as a starting distance:

  • increase k for fewer, more extreme raw breaches;
  • decrease k for subtler deviations and more noise;
  • do not tune k before fixing the metric’s shape, baseline method and cadence.

A flat baseline is a special case. If the learned spread is almost zero, even a one-unit change is an enormous z-score. Raising k is usually the wrong fix; use a spread floor or choose a deterministic threshold for a truly invariant metric.


5. Choose the baseline method

All three methods are per entity, warm up silently, judge the current value against prior state and freeze on a raw two-sided breach. They differ in what they consider “normal”.

EWMA: a continuously adapting baseline

EWMA keeps a transformed mean, variance and sample count. For a non-breaching observation y:

new_mean     = old_mean + alpha * (y - old_mean)
new_variance = (1 - alpha) *
               (old_variance + alpha * (y - old_mean)^2)

Use EWMA when the metric changes gradually and does not have a strong weekly schedule. It is the default and usually the first method to try.

Strengths: small fixed state, smooth adaptation, simple tuning.

Limits: it does not know the time of day. A large value that occurs during warm-up, or remains inside a broad band, can pull the baseline. A value that actually breaches is frozen and does not pull it.

Understand alpha as memory

alpha is the weight of the newest non-breaching sample. Its approximate half-life in observed samples is:

half_life = ln(0.5) / ln(1 - alpha)
alphaApproximate half-lifeBehaviour
0.0513.5 samplesslow, stable adaptation
0.106.6 samplesbalanced starting point
0.203.1 samplesfast adaptation
0.301.9 samplesvery short memory

Multiply by cadence only when the entity produces a row on every run. With a 15-minute cadence, alpha = 0.1 has a half-life near 100 minutes for a continuously present entity, but much longer for a sparse one.

Lower alpha resists slow drift but takes longer to accept a legitimate regime change. Higher alpha follows a new regime faster but can learn a slow attack as normal. Pair anomaly detection with an absolute threshold when a hard risk limit must never be learned away.

Median + MAD: a robust sliding window

This method keeps the last N accepted transformed observations:

centre = median(window)
spread = 1.4826 * median(|value - centre|)

Use Median + MAD when the baseline may be contaminated by isolated large but non-breaching observations, especially during warm-up, and you want the centre to resist them.

Strengths: robust centre and spread; one extreme value has little influence on the next decision.

Limits: more state per entity; no awareness of time of day; a quiet or discrete series can make MAD collapse to zero unless a spread floor is set.

Window N ranges from 8 to 256 and defaults to 32:

  • larger N: steadier, more robust, slower to adapt;
  • smaller N: more reactive, less stable;
  • warm-up samples cannot exceed N.

The window covers N × cadence only for an entity observed on every run.

Median + MAD is not a general cure for alert noise. On right-skewed byte volumes, its robust spread can be narrower than EWMA’s spread and generate more breaches. On a legitimate recurring backup, freeze-on-breach prevents the burst from ever entering the window, so it can alert forever. Use log1p for skewed volumes and Seasonal for a real weekly rhythm.

Seasonal: one EWMA per hour and weekday

Seasonal keeps up to 168 independent EWMA baselines per entity:

7 weekdays x 24 hours = 168 time-of-week slots

An observation is compared only with previous observations from the same UTC weekday and UTC hour. A high Monday-morning value can be normal while the same value on Sunday night remains anomalous.

Use Seasonal when a legitimate daily or weekly pattern is the cause of false positives.

Strengths: models repeated time-of-week behaviour that EWMA and Median + MAD cannot represent.

Limits: every slot warms up independently, state is much larger, and a schedule tied to local civil time may shift relative to UTC around daylight saving changes.

Warm-up can take weeks. At a 15-minute cadence, a continuously present entity can contribute about four samples to a given hourly slot each week; a warm-up of 10 therefore needs roughly three visits to that week/hour. Sparse entities take longer. A Seasonal rule that shows no alerts but has low slot coverage is not yet proven quiet; it may simply be unprimed.

Method decision table

Traffic shapeStart withMain tuning control
Gradual change, no weekly rhythmEWMAalpha
Isolated baseline contamination or outliersMedian + MADWindow N
Repeated hour-of-week patternSeasonalalpha, plus enough history
Hard invariant or safety limitThreshold ruleraw threshold

Changing the baseline method from the Anomaly Detection page resets learned state and restarts warm-up. That reset is necessary because EWMA scalars, a MAD window and a Seasonal grid are not interchangeable.


6. Make the statistical model fit the metric

Choose the transform and spread floor before fine-tuning k.

Transform: correct the distribution’s shape

Metric shapeTransformTypical examples
Approximately symmetric with stable varianceNoneratios or already-normalised scores
Non-negative discrete countsqrtCOUNT, COUNT_DISTINCT
Non-negative right-skewed magnitudelog1pbyte or packet sums

The transform changes the space in which centre, variance, MAD and z-score are calculated. The chart inverts the result back to raw units.

  • sqrt reduces the level-dependent variance common in count data.
  • log1p(x) = log(1 + x) handles zero and compresses a long volume tail.
  • Both transforms clamp negative input to zero. Do not use them for a signed metric whose negative values carry meaning.

The std.anomaly detectors already use sqrt for counts and log1p for byte volume.

Spread floor: prevent a collapsed denominator

The Spread floor is configured in raw metric units. A floor of 1 for a count says the detector should not trust its learned spread to be finer than one count unit. obserae converts that floor into the estimator’s working space at the current level.

Use it when a metric is discrete or nearly constant:

  • count quantum 1 → start with a floor of 1;
  • a measurement rounded to 10 MB → consider a floor matching that meaningful resolution;
  • 0 disables the floor.

The floor changes the statistical denominator and therefore the raw breach decision. It is different from a dead-band, which acts only after a breach.


7. Decide which breaches deserve an alert

Once the baseline is credible, emission controls turn statistical breaches into operationally useful alerts.

ControlDefaultEffect
DirectionBothemit high breaches, low breaches or both
Dead-band0 offrequire a minimum raw-unit deviation from the centre
Persistence1 offrequire consecutive direction-matching breaches
Cooldownrule settingsuppress repeated notifications for the same entity

Direction

Use High for signals where only growth is suspicious: bytes out, peer count, port count. Use Low when a fall is meaningful and the query still returns a numeric row.

Direction filters alert emission, not baseline protection. A large low-side breach is still frozen even on a High-only rule.

A drop to zero is visible only if zero is returned. If the entity disappears from the grouped result, no anomaly decision runs. Use Heartbeat for that semantic.

Dead-band

A dead-band suppresses statistically large but operationally trivial changes. For example, a count moving from 1 to 3 can be many learned spreads away but still not warrant paging.

Unlike HAVING, dead-band does not remove the observation before the detector sees it. The raw breach remains frozen and tracked.

Persistence

Persistence requires consecutive raw breaches in the selected direction. A normal point or a breach in the other direction resets the streak. Be careful with overlapping query windows: three consecutive values may contain much of the same underlying traffic.

obserae includes two safety behaviours:

  • a material breach at least 2 × k spreads from the centre bypasses persistence and fires immediately;
  • a breach below the dead-band escalates after 10 × persistence consecutive breaches, preventing permanent suppression of a low-and-slow deviation.

The immediate bypass must also exceed the dead-band in raw units, so a collapsed spread cannot turn a trivial one-unit change into a “massive” alert.

Parameter quick reference

ParameterDefault in the editorValid valuesWhen increased
k3> 0 and <= 50fewer, more extreme raw breaches
alpha0.1strictly between 0 and 1faster adaptation, shorter memory
Window N328..256steadier MAD baseline, slower adaptation
Warm-up10at least 2; no more than N for MADlonger silent learning period
Spread floor0 offraw value >= 0wider minimum statistical band
Dead-band0 offraw value >= 0larger material deviation required
Persistence1 offinteger >= 1 in the editormore consecutive breaches required

Severity does not change the score or firing decision. It classifies an emitted alert for prioritisation and routing.

Freeze-on-breach and cooldown

Any primed two-sided raw breach freezes the baseline, even when direction, dead-band, persistence or cooldown prevents an alert. This has two consequences:

  1. notification policy never teaches the model that a deviation is normal;
  2. a genuine permanent regime change can continue breaching indefinitely.

After confirming a permanent change is legitimate, establish a fresh baseline rather than increasing k until the old and new regimes fit one band.


8. Warm-up, missing data and baseline lifecycle

Warm-up counts observations, not elapsed time

The UI default is 10 samples; the built-in detectors use 10 or 12. During warm-up, values are learned and never alert.

For a continuously present entity:

approximate EWMA/MAD warm-up time = samples x cadence

For a sparse entity it takes longer because absent runs contribute nothing. For Seasonal, the sample requirement applies separately to each time-of-week slot.

Raise warm-up when initial data is unrepresentative. Do not raise it merely to silence a bad signal: that postpones the problem instead of fixing it.

Baselines survive normal restarts

Learned baselines are restored after restart, so a routine deployment does not normally send every entity back through cold start. Recent unsnapshotted learning can be lost after an abrupt crash, but the rule simply relearns it.

Per-entity state stays bounded by Max keys, the selected estimator’s fixed state size and optional retention. Any edit that changes the signal or its statistical decisions resets the rule’s learned state centrally, whether the edit comes from the GUI, the Lab or the API. Deleting a rule removes its learned state.

Treat structural signal changes as a new model

Changing the query, metric, group-by key, cadence, transform, method or any statistical decision setting (k, direction, dead-band or persistence) starts a new learning cycle. This is necessary because freeze-on-breach means even a threshold change can alter which observations the old model absorbed. Editing operational metadata such as the name, severity, cooldown or remediation keeps the learned model.

For a structural change, prefer duplicating the rule and letting the new version warm up before retiring the old one. This preserves coverage and prevents old state from being interpreted under a new signal definition.


9. Validate with Anomaly rule preview

Analysis → Anomaly Lab opens Anomaly rule preview. It replays retained data through the same estimator used by the alert engine. Use it before routing a new rule to an output.

The guided view deliberately separates What is unusual? (sensitivity and adaptation) from When should we notify? (direction and consecutive unusual observations). Show expert controls reveals the exact estimator, k, alpha, MAD window, transform, spread floor and emission gates. Both views edit the same configuration; guided mode is not a separate simplified engine.

You can start from an existing anomaly rule, a saved query or ad-hoc NFQL. Pick a representative entity, then inspect how method, k, alpha or N, warm-up, transform, floor and emission controls change the band and fire count.

A disciplined tuning workflow

  1. Validate the signal in Investigation. Check units, direction, entity key and expected row cardinality. Remove HAVING from the baseline query.
  2. Use the engine cadence. The Lab defaults to the rule cadence when possible. A different replay bucket represents a different series.
  3. Test several entities. Include a quiet endpoint, a busy endpoint and a known unusual period. One “easy” host is not representative.
  4. Choose transform and floor. Use sqrt plus floor 1 for counts; log1p for byte volume; confirm the displayed band is plausible.
  5. Choose the baseline method. Use the traffic shape, not the lowest alert count, to select EWMA, MAD or Seasonal.
  6. Set warm-up. Ensure the replay actually leaves warm-up and, for Seasonal, has meaningful slot coverage.
  7. Tune k. Balance missed behaviour against the number of alerts your team can investigate.
  8. Add emission controls last. Direction first, then dead-band for material impact, then persistence for isolated blips.
  9. Validate on a wider period. Include weekdays, weekends, maintenance and known incidents where possible.

The preview first spells the signal out, including its entity key and the fact that a missing query row is unknown, not zero. It reports the selected entity’s simulated detector emissions per observed wall-clock day, the separate data/model coverage, raw anomalies, notifications, peak z-score and how the estimator’s learned spread compares with the series. It does not report accuracy: without labelled incidents, missed attacks and false positives cannot be measured. Per-entity cooldown is applied later by the live evaluator, so routed alert volume can be lower. The Lab refuses to rank a configuration that stayed mostly in warm-up: zero alerts without coverage is not a successful detector.

The chart uses four separate outcomes:

  • Unusual observation — the model judged the point outside its expected range. Alert policy does not rewrite this statistical fact.
  • Suppressed by policy — direction, dead-band or persistence withheld the notification. A persistence marker shows progress such as 1/3.
  • Simulated notification — the current preview settings would notify. An immediate severity bypass is named explicitly.
  • Recorded notification — the historical alert journal says a live rule actually notified; this is evidence, not a reconstructed decision.

Every unusual observation also has a decision sentence with the observed value, exact expected bounds, policy reason, persistence progress and severity-bypass state. Hovering the chart shows the same server-computed z-score and policy decision. After any setting change, Effect of … compares unusual observations, policy suppressions, simulated notifications and immediate bypasses before and after. It is therefore expected for persistence to leave the unusual-observation count unchanged while reducing notifications.

The response labels this result selected_entity, simulated and bucketed_historical_approximation; it never presents one entity as fleet-wide alert volume.

After inspecting representative entities, select Validate all entities. This runs one bounded scan and reports whole-signal notification load, raw anomalies, data coverage, model coverage and the noisiest entities. If the scan cap is reached, the result is marked partial rather than silently extrapolated.

Understand replay limits

The Lab and Anomaly Detection charts reconstruct a series from retained traffic; obserae does not store every historical baseline decision.

  • The Lab uses the rule cadence by default, but widens very small buckets on long horizons to keep replay bounded.
  • The Anomaly Detection chart chooses a display bucket that fits the selected window. Its reconstructed breach markers may therefore differ from live decisions made at the actual cadence.
  • Journalled alerts are authoritative; reconstructed points are explanatory.
  • Replay works for queries ending in a plain STATS <metric> = <aggregate>(…) BY <entity>. Complex pivot cascades, trailing HAVING/SORT, or sources without a time column may not be replayable.

Applying Lab settings writes them to an editable rule. Pack-owned rules are read-only: duplicate one before tuning it.


10. Diagnose a rule from its symptoms

SymptomLikely explanationCorrective action
Rule never leaves warm-upentity is absent on many runs, metric is NULL, cadence is slow, or Seasonal slots lack visitsinspect raw query rows; widen the population; wait for real coverage
Expected zero/drop is missedthe group disappeared, so no zero row was produceduse Heartbeat or redesign the query to emit explicit zeros
Count metric fires on 1 -> 2discrete metric, collapsed spreaduse sqrt and spread floor 1; add a raw dead-band if the change is immaterial
Byte-volume rule is noisyright-skewed heavy tail judged in raw spaceuse log1p before raising k
Every morning or Monday fireslegitimate time-of-week rhythmuse Seasonal; remember slots are UTC and need weeks of coverage
Recurring backup still fires under MADfrozen recurring burst never enters the robust windowuse Seasonal if scheduled, or separate that traffic from the signal
Slow drift is learned awayEWMA adapts too quicklylower alpha; pair with a fixed threshold for hard limits
Accepted permanent regime keeps firingfreeze-on-breach anchors the old baselinewarm up a fresh rule for the new regime
Many alerts remain during cooldowncooldown is per entity; many different keys are firinginspect grouping cardinality and the activity heatmap
Meta-alert says max_keys exceededquery returned more entities than the configured capnarrow the population, simplify group-by or deliberately raise the cap
Chart points do not align with fire ringsreconstructed display bucket differs from live cadencereplay in Anomaly Lab at the engine cadence; trust journalled fires
Persistence changes no yellow pointsyellow means statistically unusual; persistence is notification policyread the before/after panel and the suppressed/simulated markers
Raising k changes almost nothingspread is nearly zero, or breaches are extremely far outset a spread floor, transform the metric, or use a threshold rule

When noise appears, debug in this order:

query meaning -> missing rows -> cadence/window -> transform/floor
-> baseline method -> warm-up -> k -> emission controls

Changing k first often hides the symptom without fixing the model.


11. Built-in anomaly detectors

A fresh install has no active anomaly rule. Install std.anomaly from Rule Sets to add nine maintained detectors:

DetectorEntity and metricDefault metric treatment
Adaptive exfiltrationupload bytes per clientlog1p, EWMA
Egress fan-outdistinct Internet peers per clientsqrt, floor 1, EWMA
DNS exfiltrationDNS sessions per clientsqrt, floor 1, EWMA
Lateral spreaddistinct internal peers per clientsqrt, floor 1, EWMA
Lateral admin surgedistinct internal admin peers per clientsqrt, floor 1, EWMA
Port scan (vertical)distinct ports per client-target pairsqrt, floor 1, EWMA
Host sweep (horizontal)distinct targets per client-port pairsqrt, floor 1, EWMA
Half-open surgeno-reply targets per clientsqrt, floor 1, EWMA
Auth brute forceadmin/auth sessions per client-target pairsqrt, floor 1, EWMA

The pack uses inferred client_ip / server_ip roles and directional byte columns, so canonical storage order is never confused with client/server direction. Its windows match its 5- or 15-minute cadences, and its warm-up is 10 or 12 observations. The shipped rules leave Direction at its default, Both; on a custom copy, High is often more operationally appropriate for volume, peer and port counts if unusual drops do not matter to your use case.

Pack rules are read-only. You can enable or disable them, but duplicate a rule to adapt its query or parameters. The copy has its own baseline and should warm up before replacing the original.

Three starting recipes

Adaptive exfiltration

FROM sessions | LAST 900
  | WHERE server_ip == "internet4" OR server_ip == "internet6"
  | STATS out = SUM(client_to_server_bytes) BY client_ip

For a high-side custom detector, start with entity client_ip, metric out, EWMA, log1p, k = 3, alpha = 0.1, warm-up 12, direction High, cadence 15 minutes.

Lateral spread

FROM sessions | LAST 900
  | WHERE server_ip == "internal4" OR server_ip == "internal6"
  | STATS peers = COUNT_DISTINCT(server_ip) BY client_ip

For a high-side custom detector, start with entity client_ip, metric peers, EWMA, sqrt, spread floor 1, k = 3, alpha = 0.1, warm-up 10, direction High, cadence 15 minutes.

Vertical port scan

FROM sessions | LAST 300
  | STATS ports = COUNT_DISTINCT(server_port) BY client_ip, server_ip

For a high-side custom detector, start with entity client_ip, server_ip, metric ports, EWMA, sqrt, spread floor 1, k = 3, alpha = 0.1, warm-up 10, direction High, cadence 5 minutes. Watch cardinality: every client-target pair owns a baseline.

These are starting configurations, not universal truth. Validate them against your traffic and alert budget in the Anomaly Lab.


12. Operate the Anomaly Detection page

Analysis → Anomaly Detection is the operational view for statistical rules. Deterministic rules remain on the Rules page.

The overview separates rules that are active, still learning or off, and shows tracked entities and recent fires. Open a rule to inspect:

  • recent fires, with the entity and observed value;
  • each entity’s learned baseline and warm-up state;
  • an activity heatmap of deviation by entity and time;
  • an observed-versus-expected chart with centre, band, raw breaches and emitted alerts;
  • a Seasonal 24 x 7 grid when that method is used.

Clicking a fire opens the case at the relevant time. The alert’s recorded value and matched rows are authoritative. The surrounding band may be reconstructed from retained traffic, so the page labels when it is showing reconstructed or current-baseline context.

Entity labels add cartography identity and, for public addresses, available country, ASN, threat-feed and cloud context. This enrichment helps prioritise an alert but does not alter its statistical score.

Switching the baseline method inline asks for confirmation, clears learned state and starts warm-up again. Pack-owned rules offer Duplicate instead of Edit.


13. Which detection type should you use?

NeedUse
Explore and understand a traffic distributionNFQL statistical operators
Enforce a known limit shared by everyoneThreshold rule
Detect unusual behaviour relative to each entityAnomaly rule
Detect that an expected entity or signal disappearedHeartbeat rule
Detect a hard limit and behavioural driftThreshold and Anomaly rules together

A mature workflow is:

hypothesis -> exploratory NFQL -> stable metric -> Anomaly Lab
-> silent warm-up -> limited alert routing -> production tuning

Anomaly detection is strongest when it complements deterministic controls. It can discover deviations you did not know how to threshold, while fixed rules still enforce invariants that must never become normal.

See also:

  • NFQL for query syntax and aggregation;
  • Alerting for rule cadence, cooldown and outputs;
  • Rule sets for installing and managing std.anomaly.