NFQL

NFQL is the query language used by obserae. You can run it from the Investigation page or with obserae-cli query to answer questions such as:

• Which internal systems contacted the internet during the last hour?
• Which clients sent the most data to a database?
• Which sessions did not match any Flow Matrix rule?
• Which conversations crossed several exporters, and did those exporters agree on the observed volume?

An NFQL query reads from left to right:

FROM flows
  | LAST 3600
  | WHERE dst_addr == "internet4" AND protocol == TCP
  | KEEP src_addr, dst_addr, dst_port, bytes
  | SORT bytes DESC
  | LIMIT 100

Read this as: “Take the flows received during the last hour, keep TCP traffic whose destination is on the public IPv4 internet, select the useful columns, then return the 100 largest flows.”

This guide starts with the minimum needed to write useful queries, then covers the complete language and queryable schema. You do not need to know SQL.

Use the page in two ways:

• follow Learn NFQL in five steps for a first investigation;
• use the stage reference, filtering, time windows, aggregation and cross-table lookups while writing queries;
• consult the complete schema to find a source or column;
• start from the investigation recipes and adapt the cartography names to your environment.

Learn NFQL in five steps

1. Choose what one row represents

Every query starts with FROM:

FROM flows

The source determines what each row means. The three sources used most often are:

Start with flows when you need the original export fields. Start with sessions when the question is about a client talking to a server. Start with sessions_consolidated when the same conversation may have been observed by several exporters. See Sessions and NAT for the difference.

2. Narrow the time range

Time filters make a query relevant and keep investigations fast:

FROM flows | LAST 300

LAST is expressed in seconds: 300 is five minutes, 3600 is one hour and 86400 is one day.

Use BETWEEN for an incident window:

FROM flows
  | BETWEEN "2026-07-16T08:00:00+02:00" AND "2026-07-16T09:00:00+02:00"

The time column depends on the source. NFQL selects the correct one automatically; you do not need to name it in LAST or BETWEEN.

The Investigation time filter and saved dashboard panels can bind their window and aggregation width to the viewer’s controls:

FROM flows
  | BETWEEN $from AND $to
  | EVAL bucket = BUCKET(time_received, $interval)
  | STATS bytes = SUM(bytes) BY bucket

$from and $to are timestamp expressions; $interval and $range_s are integer seconds. This means an EVAL can derive a stable bound such as from_far = $from - 600 (ten minutes before the selected start) and a later BETWEEN can use that alias. The editors highlight all four variables. They are resolved safely by the compiler when Investigation or a dashboard runs the query; they are not string substitutions. CLI requests do not have a selected web time range, so use explicit bounds there.

3. Filter rows with WHERE

FROM flows
  | LAST 3600
  | WHERE protocol == TCP AND dst_port == 443

Combine conditions with AND, OR and NOT. Parentheses make mixed conditions unambiguous:

FROM flows
  | WHERE protocol == TCP
      AND (dst_port == 80 OR dst_port == 443)

4. Choose and order the result

FROM flows
  | LAST 3600
  | WHERE dst_port == 443
  | KEEP time_received, src_addr, dst_addr, bytes
  | SORT bytes DESC
  | LIMIT 20

• KEEP selects columns and fixes their display order.
• DROP removes a few unwanted columns while retaining all others.
• RENAME gives columns result-friendly names.
• SORT orders rows; ascending order is the default.
• LIMIT caps the number of rows.

Use SORT ... | LIMIT ... together for a deterministic top-N query. A LIMIT without a SORT returns an unspecified subset.

5. Summarise with STATS

Raw rows answer “what happened?”. Aggregation answers “how much, how often and for whom?”:

FROM flows
  | LAST 3600
  | WHERE dst_addr == "internet4"
  | STATS bytes_out = SUM(bytes), flow_count = COUNT(*) BY src_addr
  | SORT bytes_out DESC
  | LIMIT 10

This returns one row per source address, rather than one row per flow.

At this point you can already write most day-to-day investigations. The next sections explain every construct and its exact behaviour.

Running queries

Investigation page

Open Investigation in the Analysis section of the sidebar. The page provides:

• syntax highlighting and completion;
• a quick, relative or absolute time filter that resolves $from, $to, $interval and $range_s;
• a searchable catalog whose Variables section explains and inserts those time variables, followed by tables, columns, virtual columns and cartography references;
• Ctrl+Enter or Cmd+Enter to run the query;
• sortable, paginated results;
• saved queries and JSON/CSV export.

Interactive queries time out after 30 seconds. The page displays at most 10,000 rows and marks a result as truncated when that cap is reached. Add a time filter, a more selective WHERE, or a LIMIT rather than treating a truncated result as complete.

Command line

Quote the complete pipeline so the shell does not interpret | or >:

# Formatted table
obserae-cli query 'FROM flows | LIMIT 5'

# JSON array: one object per result row
obserae-cli query --json \
  'FROM flows | KEEP src_addr, bytes | LIMIT 3'

# Bind a value to a ? placeholder
obserae-cli query --arg 443 \
  'FROM flows | WHERE dst_port == ? | LIMIT 5'

--arg may be repeated. Values that parse as integers are sent as integers; all other values are strings. Prefix a numeric-looking value with s: to force a string, for example --arg s:443.

The CLI also applies a 30-second server-side timeout. Unlike the Investigation page, it does not add a row cap: use LIMIT when you do not need the complete result.

The pipeline model

An NFQL program contains one or more pipelines. A pipeline starts with FROM; stages separated by | transform its rows from left to right:

FROM flows
  | WHERE protocol == TCP
  | KEEP src_addr, dst_addr, bytes
  | SORT bytes DESC
  | LIMIT 50

The output columns of each stage become the input columns of the next stage. For example, this query is invalid because KEEP removed protocol:

FROM flows
  | KEEP src_addr, dst_addr
  | WHERE protocol == TCP

Place filters early for readability and efficiency, and place the final KEEP, SORT and LIMIT near the end.

Complete stage reference

Most stages can be repeated if the columns currently in scope allow it. Several WHERE or time stages intersect naturally. A final pipeline should contain only one SORT and one LIMIT; if repeated, the last declaration is the one applied to the terminal result. Each pipeline accepts at most one of PIVOT, PIVOT NOT or JOIN.

Lexical rules

• Keywords and named protocol constants are case-insensitive. from, FROM, tcp and TCP are accepted. Use the exact lowercase spelling shown in the catalog for table and column names.
• Whitespace and line breaks are insignificant outside strings.
• Line comments start with --. # is a shell/YAML comment, not an NFQL comment.
• Strings use double quotes. The supported escapes are \" and \\. Single-quoted strings are not valid NFQL.
• Identifiers contain letters, digits and underscores and cannot start with a digit.
• Numbers are integers (42) or decimals (20.5). Scientific notation is not supported. A leading negative sign is reserved for relative BETWEEN bounds; use arithmetic such as 0 - value if a computed negative value is required.
• NFQL has no standalone NULL, boolean, duration-suffix or wildcard literal in ordinary expressions. Missing values use the dedicated IS NULL and IS NOT NULL predicates. * is supported specifically by COUNT(*) and as an open BETWEEN bound.

Filtering with WHERE

Comparisons and types

FROM flows | WHERE bytes > 1000000
FROM flows | WHERE protocol == 6
FROM flows | WHERE src_port < 1024
FROM flows | WHERE dst_port >= 5000
FROM sessions | WHERE role_conf == "LOW"

The comparison operators are ==, !=, <, <=, > and >=.

• == and != require comparable values.
• Ordering is available for numbers, timestamps and strings. String ordering is lexical.
• Integer and decimal values can be compared; numeric values are widened when necessary.
• IP addresses and CIDRs have type INET. Use equality for an exact address and WITHIN for containment.
• UUID values can be compared for equality and used in cross-pipeline lookups, but not ordered.
• There is no general timestamp literal in WHERE; use LAST or BETWEEN to select an event window.

Missing values with IS NULL

Use IS NULL to select rows where a value is absent and IS NOT NULL to keep rows where a value is present:

FROM sessions_consolidated
| BETWEEN $from AND $to
| WHERE ip == "internal" AND client_host IS NULL
| LIMIT 10

FROM sessions | WHERE closed_at IS NOT NULL

The same predicates work on computed expressions and aggregate results:

FROM flows | EVAL ratio = bytes / packets | WHERE ratio IS NULL
FROM flows | STATS deviation = STDDEV(bytes) BY protocol | HAVING deviation IS NULL

NULL is absence, not an ordinary value. Write client_host IS NULL, not client_host == NULL; the latter is rejected with a correction. An empty string "", numeric zero and the text "unknown" are present values and do not match IS NULL.

Boolean logic

FROM flows | WHERE protocol == TCP AND dst_port == 443
FROM flows | WHERE dst_port == 80 OR dst_port == 443
FROM flows | WHERE NOT (protocol == TCP)
FROM flows | WHERE (dst_port == 80 OR dst_port == 443) AND protocol == TCP

Precedence, from tightest to loosest, is:

1. NOT;
2. arithmetic multiplication and division;
3. arithmetic addition and subtraction;
4. comparisons, WITHIN, IN and the IS NULL predicates;
5. AND;
6. OR.

Use parentheses whenever a reader could hesitate.

Set membership with IN

FROM flows | WHERE dst_port IN (80, 443, 8080)
FROM flows | WHERE protocol IN (TCP, UDP)
FROM flows | WHERE src_addr IN ("host:proxy", "group:backends")

An IN list accepts integer and string literals, ? parameters and named protocol constants. Decimal items, column references and nested queries are not accepted. To compare with the output of another pipeline, use PIVOT or JOIN.

For INET values, an IP or CIDR literal in IN uses strict equality. This does not mean containment:

-- Exact values only; this does not match every address in 10.0.0.0/8
FROM flows | WHERE src_addr IN ("10.0.0.0/8", "192.0.2.10")

Use WITHIN clauses joined by OR when you need several CIDR ranges.

Containment with WITHIN

The three comparison operators divide up cleanly:

• == compares a column to one value;
• IN compares it to a list of values;
• WITHIN asks whether it belongs to a range.

A range can be written three ways, and all three read the same:

FROM sessions | WHERE server_ip WITHIN "10.0.0.0/8"      -- a CIDR
FROM flows    | WHERE src_addr WITHIN "2001:db8::/64"    -- IPv6, same syntax
FROM sessions | WHERE server_ip WITHIN "network:office"  -- a name from your map

The last one is the useful one day to day. A cartography name — a network, a host, a group, a role — is a set of addresses, so asking whether an address belongs to it is exactly what you meant. Every reference form works:

FROM sessions | WHERE client_ip WITHIN "workstations"        -- a group
FROM sessions | WHERE server_ip WITHIN "host:srv-db-01"      -- every NIC of a host
FROM sessions | WHERE client_ip WITHIN "lan.dhcp"            -- the DHCP pool only
FROM sessions | WHERE server_ip WITHIN "internet"            -- anything public

A name always goes on the right of WITHIN.

On a CIDR literal the two operators really do differ, and the difference matters:

FROM flows | WHERE src_addr == "10.0.0.0/8"      -- exact equality: the network address itself
FROM flows | WHERE src_addr WITHIN "10.0.0.0/8"  -- any address in the /8

On a name they mean the same thing and return the same rows. == on a name was always a membership test underneath; WITHIN simply says so. Existing queries written with == keep working.

Shortcuts that make queries readable

Named protocol constants

All three forms below are equivalent on a protocol column:

FROM flows | WHERE protocol == TCP
FROM flows | WHERE protocol == tcp
FROM flows | WHERE protocol == "TCP"

The string rewrite only applies to a column named protocol. A string such as bytes == "TCP" remains a type error instead of silently becoming bytes == 6.

Virtual ip and port columns

flows, sessions and sessions_consolidated expose two virtual columns:

FROM flows | WHERE ip == "192.0.2.10"
-- Equivalent to src_addr == "192.0.2.10" OR dst_addr == "192.0.2.10"

FROM sessions | WHERE port == 443
-- Equivalent to server_port == 443 OR client_port == 443

The operator determines how the expansion is combined:

• ip == X, ip IN (...) and ip WITHIN X match when either real column matches;
• ip != X matches only when neither real column matches;
• ordering operators such as ip < X are rejected; name the directional column explicitly.

Virtual columns are filters and lookup keys, not result columns. You cannot write KEEP ip; keep the real source/destination or client/server columns.

Cartography and rule-set references

When an INET column is compared with a double-quoted name using WITHIN, ==, != or IN, NFQL resolves that name to its current set of addresses. WITHIN is the clearest spelling — a name is a set, and you are testing membership — but all of them work and return the same rows.

Examples:

FROM flows
  | WHERE src_addr WITHIN "role:workstation" AND dst_addr WITHIN "internet"

FROM sessions
  | WHERE server_ip WITHIN "group:databases" AND server_port == 5432

FROM flows | WHERE src_addr WITHIN "office.dhcp"
FROM flows | WHERE src_addr WITHIN "office.printers"
FROM flows | WHERE dst_addr WITHIN "host:proxy:wan"

References are live: a later query uses the current cartography and installed rule-set vocabulary.

The numbered reserved sets are per-family; the unnumbered "any", "internet" and "internal" cover both at once, so a dual-stack question needs one clause instead of two. Those three exist in NFQL only — you cannot declare an interface on “both families” — and they take precedence over a cartography network of the same name. If your map has a network literally called internet, write "network:internet" to reach it.

internal4 and internal6 include every non-public address in their family, not only RFC 1918 or ULA space. This includes loopback, link-local, CGNAT, multicast and reserved ranges.

"NAME.dhcp" and "NAME.static" are obserae’s own split of a network. You can name your own slices too — "office.printers", "office.test_device" — by declaring them on the network; see named sub-ranges.

Since host names and historical network names may contain dots, a dotted name stays whole until resolution. A bare name matches an exact network, host or group before falling back to network.subrange. The explicit "network:office.printers" skips hosts and groups: it prefers an exact network, then the sub-range. Validation refuses a network and qualified sub-range with the same spelling. Mistype a slice and obserae lists what the network declares.

Reading names instead of addresses

The references above let you filter by name. To get the name back in the result, use the cartography columns that sessions and sessions_consolidated carry:

FROM sessions | LAST 3600
  | STATS bytes = SUM(client_to_server_bytes)
       BY client_host, server_host, server_service
  | SORT bytes DESC | LIMIT 20

Instead of three columns of raw addresses, you read poste-lmo → srv-web-01 (https).

Three things to know:

• They are free. The values are recorded when the session closes, not looked up when you query. Using them costs nothing and never changes how many rows you get back.
• They are a snapshot of the past. Renaming a host in the cartography does not change sessions already recorded. A query over last month shows the names that were in use last month — which is what you want when reconstructing an incident, but does mean a rename appears as two names either side of the change. Sessions recorded before a host was added to the cartography stay empty.
• Empty means “not in the cartography”. An unknown address leaves the columns empty and the session is still returned — nothing is ever hidden or guessed. server_service is also empty when Obserae could not tell which side was the server.

Unlike the reference syntax above, these columns are not live: that is the trade-off for reading them without a lookup.

Service purposes

A rule-set purpose represents a port and protocol, so it is compared with a pseudo-column rather than an IP column:

FROM flows
  | LAST 300
  | WHERE src_addr == "zone:user"
      AND port_proto == "purpose:std.dns"

FROM sessions
  | WHERE server_port_proto == "purpose:std.postgres"

A purpose matches its standard ports and any deployment-specific service port assigned that purpose in your cartography. The deployment-specific match is scoped to the service address, so a custom PostgreSQL port does not become PostgreSQL everywhere. != negates the complete purpose match. Other operators and IN are not supported for purpose pseudo-columns. See Rule sets.

Time windows

LAST

FROM flows | LAST 60
FROM sessions | LAST 3600
FROM sessions_consolidated | LAST 86400

LAST N requires a strictly positive integer number of seconds. It means “time column greater than or equal to the current time minus N seconds”. Rows exactly on the boundary can fall outside the window because time advances between writing and executing a query; use a small margin for boundary tests.

BETWEEN

FROM flows | BETWEEN -3600 AND -60
FROM flows | BETWEEN -3600 AND *
FROM flows | BETWEEN -3600 AND now
FROM flows | BETWEEN "2026-07-16" AND "2026-07-17"
FROM flows | BETWEEN "2026-07-16T08:00:00+02:00" AND "2026-07-16T09:00:00+02:00"

Each bound accepts:

BETWEEN 60 AND * does not mean “the last 60 seconds”; it means “after Unix timestamp 60”. Use LAST 60 or BETWEEN -60 AND *.

Dashboard timestamps can be shifted by a whole number of seconds. A bound expression must have type TIMESTAMP and must not depend on each input row:

FROM sessions
  | EVAL from_far = $from - 600
  | BETWEEN from_far AND $from

Here 600 is ten minutes. Timestamp subtraction returns a BIGINT number of seconds; timestamp plus or minus an INTEGER/BIGINT shifts it by that many seconds. Fractional offsets are rejected.

Several time stages intersect:

FROM flows
  | LAST 86400
  | BETWEEN "2026-07-16T08:00:00Z" AND "2026-07-16T12:00:00Z"

Time column by source

rules has no event time, so LAST and BETWEEN are rejected on that source.

Parameters

? is a positional runtime parameter. Bind values from left to right:

obserae-cli query \
  --arg 443 \
  --arg '10.0.0.0/8' \
  'FROM flows | WHERE dst_port == ? AND src_addr WITHIN ? | LIMIT 100'

Parameters are supported in predicates, IN lists, arithmetic expressions and BETWEEN bounds. The number of supplied values must exactly equal the number of placeholders.

Use parameters when a program or repeated operational procedure supplies the values. They keep values separate from query structure and avoid quoting or injection mistakes. A parameter is not a column, table or stage placeholder: FROM ? and KEEP ? are invalid.

Shaping columns

KEEP and DROP

FROM flows
  | KEEP time_received, src_addr, dst_addr, bytes

FROM flows
  | DROP flow_id, flow_version, next_hop

Both preserve row count. KEEP sets the exact output schema and order. DROP preserves the order of every remaining column and must leave at least one. Duplicate or unknown column names are errors.

RENAME

The new name is on the left:

FROM flows
  | KEEP time_received, src_addr, dst_addr, bytes
  | RENAME observed_at = time_received, source = src_addr, destination = dst_addr

Renaming is simultaneous and preserves column positions. The old name is no longer available downstream. The result cannot contain duplicate names.

SORT and LIMIT

FROM flows
  | SORT bytes DESC, time_received ASC
  | LIMIT 50

Each sort key has its own direction; ASC is the default. Earlier keys have higher priority. LIMIT accepts a non-negative integer.

Computed columns with EVAL

EVAL appends one or more named expressions:

FROM flows
  | LAST 3600
  | EVAL kilobytes = bytes / 1024,
         bytes_per_packet = bytes / packets
  | KEEP src_addr, dst_addr, kilobytes, bytes_per_packet
  | SORT kilobytes DESC

Supported operators are +, -, * and /. Multiplication and division bind more tightly than addition and subtraction. Numeric expressions support all four. Timestamp arithmetic and string concatenation use the typed forms described below.

• Division always returns DOUBLE, even for integer inputs.
• Division by zero produces NULL inside the pipeline rather than failing the query. That missing value remains NULL in GUI and API results.
• +, - and * preserve integer types unless a DOUBLE operand is present.
• timestamp +/- integer shifts a timestamp by whole seconds; timestamp - timestamp returns BIGINT seconds.
• VARCHAR + VARCHAR concatenates strings. Mixed operands are rejected; wrap non-string values in STR(...).
• An alias cannot replace an existing column.
• Assignments in the same EVAL are evaluated from left to right, so a later assignment can use an alias created earlier in that stage.

INT(number) truncates a numeric expression toward zero and returns BIGINT. This converts the DOUBLE produced by division into an integer-valued column:

FROM sessions
  | BETWEEN $from AND $to
  | EVAL range_bucket = INT($range_s / 100)

For example, INT(1.9) returns 1; a negative result is also truncated toward zero. The function name is case-insensitive and non-numeric arguments are rejected during compilation.

STR(value) converts any scalar value to VARCHAR. Use it to build readable labels from names, addresses, ports, counters or timestamps:

FROM sessions
  | EVAL endpoint = client_host + " (" + STR(client_ip) + ")",
         service = server_service + ":" + STR(server_port)
  | KEEP endpoint, service

String concatenation uses +, but both operands must already be VARCHAR. This explicit cast keeps integer, INET, UUID and timestamp formatting predictable instead of relying on implicit database coercion.

BUCKET(timestamp, width_seconds) returns the UTC-aligned start of a time bucket. BUCKET(number, width) returns the numeric lower bound of a zero-aligned value range:

FROM flows
  | LAST 3600
  | EVAL minute = BUCKET(time_received, 60)
  | STATS flow_count = COUNT(*), bytes_total = SUM(bytes) BY minute
  | SORT minute ASC

For timestamps, the width must be a strictly positive INTEGER or BIGINT expression in seconds. It may therefore come from a preceding EVAL as long as INT(...) converts true division back to an integer:

FROM sessions
  | BETWEEN $from AND $to
  | EVAL x = INT($range_s / 100)
  | STATS sessions = COUNT(*) BY BUCKET(opened_at, x)

A literal zero is rejected during compilation. A computed width must remain strictly positive when the query executes.

Numeric values accept integer or decimal widths. Buckets are half-open and aligned on zero: BUCKET(17, 10) returns 10, representing the integer range 10–19; BUCKET(-1, 10) returns -10. A fractional input or width returns a DOUBLE lower bound and is displayed explicitly as [lower, upper).

FROM sessions_consolidated
  | BETWEEN $from AND $to
  | STATS sessions = COUNT(*) BY BUCKET(coherence_pct, 10)

The result keeps exact 100 in its own [100, 110) group rather than silently merging perfect coherence with 90–99. The web console displays readable range labels, while JSON and CSV retain numeric lower bounds for sorting and filters. Only ranges containing rows are returned.

Aggregation with STATS and HAVING

How aggregation changes the result

FROM flows
  | LAST 3600
  | WHERE protocol == TCP
  | STATS flow_count = COUNT(*), bytes_total = SUM(bytes) BY src_addr
  | HAVING flow_count > 100
  | SORT bytes_total DESC

The order matters:

1. WHERE filters individual input rows.
2. STATS groups the remaining rows and computes metrics.
3. HAVING filters the aggregated rows.

After STATS, the original columns disappear. The new schema contains the BY keys first, then the aggregate aliases. In the example it is src_addr, flow_count, bytes_total; a later reference to protocol would be invalid.

A STATS without BY returns one global summary row:

FROM flows
  | LAST 3600
  | STATS total_bytes = SUM(bytes),
          total_packets = SUM(packets),
          flow_count = COUNT(*),
          unique_sources = COUNT_DISTINCT(src_addr)

Every aggregate needs an explicit alias before =.

Aggregate functions

NFQL result cells preserve SQL NULL as JSON null. They are not coerced to zero, an empty string, or a zero timestamp, so visualizations can choose an honest missing-value policy.

AVG is intentionally integer-valued. For a fractional average, aggregate a sum and count, then divide them with EVAL:

FROM flows
  | STATS total = SUM(bytes), n = COUNT(*) BY src_addr
  | EVAL average = total / n

Time histograms and numeric distributions

BUCKET can appear directly in STATS ... BY. Its implicit result name is bucket:

FROM sessions
  | LAST 3600
  | STATS session_count = COUNT(*) BY BUCKET(opened_at, 60)

With several bucket expressions, names become bucket, bucket_2, and so on. Use EVAL first when you want a specific name.

Time histogram results are automatically ordered from the oldest bucket to the newest. This also applies to a timestamp alias created with EVAL bucket = BUCKET(...) and then used in STATS ... BY bucket. Additional grouping fields are stable tie-breakers inside each bucket. Add an explicit SORT only when you want a different order; an explicit sort always takes precedence.

Direct numeric BUCKET groupings are likewise ordered by lower bound. The visualization studio recognizes their metadata, suggests a bar chart, and uses the aggregate as Y instead of accidentally building a second histogram of the already-aggregated counts.

Behavioural examples

Hosts contacting many different ports tend to have higher destination-port entropy:

FROM flows
  | LAST 3600
  | WHERE protocol == TCP
  | STATS port_entropy = ENTROPY(dst_port),
          distinct_ports = COUNT_DISTINCT(dst_port) BY src_addr
  | HAVING port_entropy > 3.0 AND distinct_ports > 20
  | SORT port_entropy DESC

Do not treat a statistical threshold as a universal truth. Establish a normal range for your environment and tune it. The Anomaly detection guide explains these metrics and baseline-based detections.

Looking across tables

The cascade model

The > operator starts another pipeline. A lookup stage in the right-hand pipeline compares it with the immediately preceding pipeline:

FROM session_matches | LAST 3600 | KEEP session_id
> FROM sessions
  | LAST 3600
  | PIVOT session_id == session_id
  | KEEP client_ip, server_ip, server_port, opened_at

In a lookup predicate:

• the left column belongs to the previous pipeline’s final output;
• the right column belongs to the current pipeline at that stage.

KEEP and STATS in the previous pipeline therefore determine which keys are available to the next one.

Choosing PIVOT, PIVOT NOT or JOIN

Sessions matched by a rule:

FROM session_matches | LAST 3600 | KEEP session_id
> FROM sessions
  | LAST 3600
  | PIVOT session_id == session_id
  | KEEP client_ip, server_ip, server_port, opened_at

Closed sessions that matched no rule:

FROM session_matches | LAST 3600 | KEEP session_id
> FROM sessions
  | LAST 3600
  | PIVOT NOT session_id == session_id
  | KEEP client_ip, server_ip, server_port, role_method

Rule details attached to matching events:

FROM rules | KEEP rule_id, name, tags
> FROM session_matches
  | LAST 86400
  | JOIN rule_id == rule_id
  | KEEP matched_at, session_id, prev_name, prev_tags

JOIN can produce more than one row when either side contains duplicate keys. Use PIVOT when the previous columns are not needed; it expresses membership and does not multiply current rows.

Equality, containment and virtual lookup keys

Lookups support == for equality and WITHIN for CIDR containment:

FROM sessions
  | LAST 3600
  | KEEP client_ip, server_ip, server_port, opened_at
> FROM enrichment_ranges
  | WHERE source == "aws"
  | JOIN server_ip WITHIN cidr
  | KEEP prev_client_ip, prev_server_ip, prev_server_port,
         prev_opened_at, source, cidr

Here the previous pipeline’s server_ip is on the left and the current pipeline’s cidr is on the right: “server IP is contained by range”. Both must have type INET. JOIN is used because the final result needs the previous session columns; they are exposed with the prev_ prefix.

Either lookup side may be virtual ip or port, provided all of its real columns are still present:

FROM enrichment_ips | WHERE nature == "threat" | KEEP ip
> FROM flows
  | PIVOT ip == ip
  | LAST 3600
  | KEEP src_addr, dst_addr, dst_port, bytes

The current ip expands to src_addr or dst_addr, so traffic is found in either direction.

Lookup limits and pitfalls

• A pipeline accepts one lookup stage.
• Lookups compare one logical key. Composite-key lookups are not supported.
• Lookup operators are == and WITHIN, and both sides are columns — one from each pipeline. Range comparisons and cartography references are not accepted here: a name on the right would not mention the current pipeline at all, turning the stage into an all-or-nothing gate.
• Filter cartography in a WHERE stage (| WHERE server_ip WITHIN "internet"), then expose a real key with KEEP.
• JOIN is inner only. There are no left, right or full outer joins.
• Equality PIVOT NOT follows SQL NULL semantics. Remove nullable keys from the previous pipeline when necessary; a NULL in the lookup set can prevent expected anti-matches.
• Each pipeline refers only to its immediate predecessor, which makes longer cascades predictable.

Complete table and column reference

The Catalog panel on the Investigation page exposes this same schema with inline descriptions. Only the columns below are addressable from NFQL.

flows

One row per record received from a NetFlow/IPFIX exporter. Time column: time_received. Virtual columns: ip, port.

sessions

One closed client/server session per exporter. Open sessions remain in memory and are not queryable here. Time column: opened_at. Virtual columns: ip, port.

For upload or exfiltration investigations, aggregate client_to_server_bytes, not both directions. See Sessions and NAT.

sessions_consolidated

One conversation across all exporters. Only closed sessions are consolidated. Time column: opened_at. Virtual columns: ip, port.

Volumes are never summed across exporters because that would double-count the same traffic. The table exposes the minimum and maximum exporter views instead.

session_matches

One rule match for one session. Time column: matched_at.

rules

Operator-facing Flow Matrix rule catalog. This source has no time column.

See Flow Matrix rules.

nat_relations

Aggregated NAT relationships learned from repeated observations. Time column: last_seen.

The *_sampler columns are exporters, not addresses. Grouping by them answers “which probes saw this translation”, never “what was translated to what”. For the latter, group by original_ip / translated_ip:

FROM nat_relations | LAST 86400
  | STATS n=SUM(observation_count) by nat_type, original_ip, translated_ip, direction_method
  | SORT n DESC

A single probe sitting on the NAT box reports the same address on both sides; that is expected, not an error. See NAT.

nat_translations

Recent socket-level NAT evidence. Time column: last_seen.

enrichment_ranges

CIDR catalog obtained from cloud and threat-intelligence sources. Time column: fetched_at.

Use WITHIN in a PIVOT or JOIN because this table contains ranges.

enrichment_ips

Exact observed IPs resolved against enrichment sources. Time column: resolved_at.

See IP Enrichment.

arp

ARP observations collected from supported device connectors. Time column: timestamp.

dhcp

DHCP lease observations collected from supported device connectors. Time column: timestamp.

See Device connectors.

Investigation recipes

The named cartography objects in these examples are illustrative. Replace them with names from your own Catalog panel.

Recent HTTPS traffic touching the load balancers

FROM flows
  | LAST 3600
  | WHERE ip == "group:loadbalancers" AND port == 443 AND protocol == TCP
  | KEEP time_received, src_addr, dst_addr, bytes
  | SORT time_received DESC
  | LIMIT 200

Backends reaching the internet without the proxy

FROM flows
  | LAST 3600
  | WHERE src_addr == "group:backends"
      AND (dst_addr == "internet4" OR dst_addr == "internet6")
      AND dst_addr != "host:proxy:wan"
  | KEEP time_received, src_addr, dst_addr, dst_port, protocol, bytes

Top clients uploading to the database tier

FROM sessions
  | LAST 3600
  | WHERE server_ip == "group:databases"
  | STATS bytes_uploaded = SUM(client_to_server_bytes),
          sessions = COUNT(*) BY client_ip, server_ip, server_port
  | SORT bytes_uploaded DESC
  | LIMIT 20

Public sources reaching non-load-balancer destinations

FROM flows
  | LAST 3600
  | WHERE (src_addr == "internet4" OR src_addr == "internet6")
      AND dst_addr != "group:loadbalancers"
  | KEEP time_received, src_addr, dst_addr, dst_port, protocol
  | SORT time_received DESC

TCP half-open sessions

FROM sessions
  | LAST 3600
  | WHERE close_reason == "no_reply" AND protocol == TCP
  | KEEP opened_at, client_ip, server_ip, server_port, role_conf
  | SORT opened_at DESC

Conversations observed by several exporters

FROM sessions_consolidated
  | LAST 86400
  | WHERE sampler_count > 1
  | KEEP opened_at, client_ip, server_ip, server_port,
         sampler_count, samplers, coherence_pct
  | SORT coherence_pct ASC

A low coherence_pct can result from sampling, dropped exports or partial visibility. It is a prompt to inspect the path, not proof of a fault.

Threat-enriched sessions

FROM enrichment_ips
  | WHERE nature == "threat"
  | KEEP ip, source, detail
> FROM sessions
  | LAST 86400
  | JOIN ip == server_ip
  | KEEP opened_at, client_ip, server_ip, server_port,
         prev_source, prev_detail
  | SORT opened_at DESC

Flow rate per five minutes

FROM flows
  | LAST 86400
  | STATS flows = COUNT(*), bytes_total = SUM(bytes)
      BY BUCKET(time_received, 300)
  | SORT bucket ASC

Reading and fixing errors

NFQL reports the phase and line:column position of the first error:

When a query fails, inspect it from left to right:

1. Confirm the FROM table in the Catalog panel.
2. Confirm each referenced column still exists after KEEP, DROP, RENAME or STATS.
3. Check that compared values have compatible types.
4. Check parentheses and use ==, not =, for equality.
5. For cascades, confirm the left lookup key is emitted by the previous pipeline and the right key exists in the current one.

Current language boundaries

NFQL deliberately exposes a compact investigation language. The following are not currently supported:

• duration suffixes such as LAST 5m or LAST 1h;
• single-quoted strings, scientific notation and general negative literals;
• LIKE, regular expressions and general string functions;
• nested queries inside WHERE or IN;
• wildcards such as KEEP *;
• arbitrary scalar functions other than BUCKET and INT;
• timestamp arithmetic;
• composite-key, range or outer joins;
• selecting a virtual or purpose pseudo-column with KEEP.

These boundaries are usually handled by choosing a narrower source, combining predicates, using EVAL, or cascading pipelines with PIVOT/JOIN.

Compact syntax reference

program     := pipeline ('>' pipeline)*
pipeline    := FROM table ('|' stage)*

stage       := WHERE predicate
             | LAST positive_seconds
             | BETWEEN bound AND bound
             | KEEP column (',' column)*
             | DROP column (',' column)*
             | RENAME new '=' old (',' new '=' old)*
             | EVAL alias '=' expression (',' alias '=' expression)*
             | STATS aggregate (',' aggregate)* (BY group_key (',' group_key)*)?
             | HAVING predicate
             | SORT column (ASC | DESC)? (',' column (ASC | DESC)?)*
             | LIMIT non_negative_integer
             | PIVOT NOT? previous_column ('==' | WITHIN) current_column
             | JOIN previous_column ('==' | WITHIN) current_column

aggregate   := alias '=' function '(' ('*' | column) ')'
             | alias '=' PERCENTILE '(' column ',' rank_1_to_99 ')'

group_key   := column | scalar_call
scalar_call := BUCKET '(' (timestamp_expression ',' positive_seconds
                         | numeric_expression ',' positive_number) ')'
             | INT '(' expression ')'
bound       := '*' | now | unix_seconds | '-' seconds_ago | rfc3339_string | '?'