Operations

This page is for the person who runs obserae as a service: where files live, how to expose the GUI safely, what to monitor, how to back up the instance and what to check first when something looks wrong.

For a first hands-on setup, start with Quickstart. For package and Docker installation, see Installation.

Filesystem Layout

A typical systemd install uses:

/etc/obserae/obserae.yaml          # configuration you maintain

/var/lib/obserae/
  db/obserae.duckdb                # main database file
  data/                            # traffic history, audit log, enrichment, backups, master key
  run/obserae.sock                 # local control socket for obserae-cli

Keep /var/lib/obserae on a disk sized for your retention policy. Traffic history grows with the number of flow records you collect; retention is what makes that growth plateau.

Deploy As A Systemd Service

Fastest path: the one-line installer performs every step in this section for you — download, verify, install the binaries, and create the user, directories, config and unit — see Installation → Option 1. The manual steps below are the same deployment spelled out; use them when you want to review or customise each step.

1. Install the binaries

sudo install -m 0755 obserae      /usr/local/bin/
sudo install -m 0755 obserae-cli  /usr/local/bin/

2. Create the service user and directories

sudo useradd --system --home /var/lib/obserae --shell /usr/sbin/nologin obserae
sudo install -d -o obserae -g obserae -m 0750 /var/lib/obserae

Only the root is created here, and only because /var/lib belongs to root: the service account has to own something before it can write. obserae creates data/, db/ and run/ underneath on the first start, along with every store directory inside data/.

3. Create the config

sudo install -d -m 0755 /etc/obserae
sudo tee /etc/obserae/obserae.yaml > /dev/null <<'YAML'
listen:
  netflow:
    enabled: true
    address: "0.0.0.0:2055"
  ipfix:
    enabled: true
    address: "0.0.0.0:4739"

storage:
  data_dir: "/var/lib/obserae/data"
  duckdb_path: "/var/lib/obserae/db/obserae.duckdb"
  memory_limit: "50%"

control:
  socket: "/var/lib/obserae/run/obserae.sock"

web:
  enabled: true
  address: "127.0.0.1:8080"

logging:
  verbosity: 0
YAML

The web GUI stays on loopback here. Put a TLS reverse proxy in front before you make it reachable from another machine.

4. Install the unit

sudo tee /etc/systemd/system/obserae.service > /dev/null <<'EOF'
[Unit]
Description=obserae NetFlow/IPFIX collector
After=network-online.target
Wants=network-online.target

[Service]
Type=exec
User=obserae
Group=obserae
ExecStart=/usr/local/bin/obserae --config /etc/obserae/obserae.yaml
Restart=on-failure
RestartSec=5

NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/obserae
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
LockPersonality=true
RestrictRealtime=true

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now obserae
sudo journalctl -u obserae -f

If you bind a privileged UDP port, grant CAP_NET_BIND_SERVICE in the unit.

Expose The Web GUI Safely

The GUI requires a login. On first boot, obserae creates the admin user and prints the generated password once in the daemon log. Change it after signing in.

obserae serves HTTP; it does not terminate TLS itself. For remote access, keep obserae bound to 127.0.0.1:8080 and put a reverse proxy with HTTPS in front.

Caddy

obserae.example.com {
    reverse_proxy 127.0.0.1:8080
}

Caddy forwards the original scheme, so obserae can keep the session cookie secure.

nginx

server {
    listen 443 ssl;
    server_name obserae.example.com;

    ssl_certificate     /etc/letsencrypt/live/obserae.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/obserae.example.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Host              $host;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header Upgrade           $http_upgrade;
        proxy_set_header Connection        "upgrade";
        proxy_buffering off;   # stream Server-Sent Events (config-restore progress)
    }
}

The Upgrade and Connection headers are required for live cockpit and cartography updates. proxy_buffering off lets the config-restore progress stream (Server-Sent Events) reach the browser as it is produced. The daemon clears its own write deadline and heartbeats every 15 s during a restore, so the default proxy_read_timeout (60 s) is enough — a long restore never times out.

If the GUI appears to accept the password but sends you back to the login page, you are probably reaching a non-loopback HTTP URL without TLS. Use HTTPS, or for a trusted lab-only plain-HTTP deployment set web.secure_cookies: false.

To record the real client IP in the audit log when a reverse proxy is used, set web.trusted_proxies to the proxy address or CIDR. Without it, forwarded client IP headers are ignored.

CLI Access

obserae-cli talks to the local control socket. Give operators access by adding them to the obserae group:

sudo usermod -aG obserae alice
# alice must log out and back in

obserae-cli --socket /var/lib/obserae/run/obserae.sock status

A shell alias keeps commands short:

alias obserae-cli='/usr/local/bin/obserae-cli --socket /var/lib/obserae/run/obserae.sock'

Treat socket access as privileged. It is the administrative control path.

Monitoring

The quickest health check is:

obserae-cli --socket /var/lib/obserae/run/obserae.sock status

Use --json when integrating with a metrics stack:

obserae-cli --socket /var/lib/obserae/run/obserae.sock status --json

Watch these signals:

The GUI Monitoring page shows the same operational view for administrators. The Cockpit stays focused on analyst-facing health and alert coverage.

Audit Log

Included with Business, Business+, and Enterprise licenses. In Community, Audit log is unavailable. Existing audit history remains stored and becomes available again when a valid license is installed.

The Audit log records sensitive actions: logins, denied requests, cartography changes, rule changes, lifecycle changes, user changes and output/device edits. It is searchable in the GUI and exportable for incident response.

Calls made by an API token to the SOAR profile are recorded too, including the ones that only read. Each entry carries the operation, the permission that admitted it and the status code the server answered, so an auditor asking “which automation queried what, and what came back” can reconstruct it. Filter on the action api.profile.read.

Browser sessions are deliberately not recorded this way: a GUI refreshing a page every few seconds would bury the automation’s trail, which is the one the question is about.

The audit log is tamper-evident. If entries are edited, deleted or reordered, integrity verification reports where the chain broke.

Use:

obserae-cli verify-auditlog

Keep audit retention deliberate. By default, audit entries are kept forever; if you set an age limit, make sure it matches your investigation and compliance needs.

Master Key

obserae creates one master key on first boot, normally under <data_dir>/masterkey.bin. It protects stored secrets such as output tokens, device API secrets and session-signing material.

Back it up to a secret manager:

obserae-cli masterkey export --output /secure/obserae-masterkey.b64

If you rebuild an instance, restore the master key before first boot. Without the matching key, encrypted connector/output secrets cannot be recovered and users will need to sign in again.

You can rotate the key from Settings → Master key or the CLI:

obserae-cli masterkey import new-masterkey.b64

Offline License

Open Settings → License to inspect or install the signed offline license assigned to the instance. Obserae verifies an upload against every public root key embedded in the running binary before it displays or stores any customer field. Business, Business+ and Enterprise licenses include the same professional feature set. It stays enabled for the 30-day grace period after expiry. When grace ends, professional pages and background execution are paused; no user, rule set, output, report schedule or authentication setting is deleted.

Installing, renewing, validating, restoring, or removing a license is recorded in the audit log. Installation and removal require license:manage; read-only access requires license:read. Removal asks the operator to type the exact company name shown in the signed artifact.

The consolidated configuration YAML includes exactly one armored artifact:

license:
  artifact: |
    -----BEGIN OBSERAE LICENSE-----
    …
    -----END OBSERAE LICENSE-----

A renewal replaces that value; removal omits the entire license section from later exports. An import re-verifies the artifact before changing the active license, so a forged or corrupted backup section is rejected atomically. Older history-based bundles remain readable and are normalized to their active artifact, but are never re-exported in the old shape.

Backups

Use the built-in backup runner from the Lifecycle page or the CLI. A backup is a recoverable snapshot of the instance, not just a copy of one file.

Useful commands:

obserae-cli backup status
obserae-cli backup run
obserae-cli backup list
obserae-cli backup plan --point-in-time 2026-05-28T13:00:00Z
obserae-cli backup restore --point-in-time 2026-05-28T13:00:00Z --dry-run --confirm
obserae-cli backup restore --point-in-time 2026-05-28T13:00:00Z --confirm

Before upgrades or large imports, run a manual backup:

obserae-cli backup run

Also export configuration regularly:

obserae-cli config export > /var/backups/obserae/$(date +%F).config.yml

Config export is useful for review and version control. Backup is what you use for disaster recovery.

Upgrades

For a binary install:

obserae-cli backup run
sudo systemctl stop obserae
sudo install -m 0755 obserae /usr/local/bin/
sudo install -m 0755 obserae-cli /usr/local/bin/
sudo systemctl start obserae

For Docker:

docker pull ghcr.io/spartan-conseil/obserae:latest
docker restart obserae

Read the release notes before upgrading production. They call out any migration, configuration or compatibility notes.

Data migrations run automatically during startup. Before changing an existing store, obserae creates a selective rollback copy under migration_backup/. If the process or a migration is interrupted, the next start restores that copy before opening DuckDB and retries the upgrade automatically; no operator command is part of the normal upgrade procedure.

If automatic recovery cannot use the rollback copy, startup stops without clearing the safety marker. With the service down, inspect the state with obserae migrate status --config /etc/obserae/obserae.yaml. The offline migrate restore command performs the rollback manually and is intended only for recovery diagnostics, not routine upgrades.

Troubleshooting

The CLI cannot connect to the socket

Check that the service is running and that the CLI uses the configured socket:

sudo systemctl status obserae
obserae-cli --socket /var/lib/obserae/run/obserae.sock status

Flow count stays at zero

Check the exporter path in this order:

1. The device is configured to send NetFlow/IPFIX to the obserae host.
2. UDP 2055 or 4739 is reachable through firewalls.
3. Packets arrive on the host.
4. NetFlow v9/IPFIX templates have arrived.

Useful checks:

sudo ss -ulnp | grep -E '2055|4739'
sudo tcpdump -ni any udp port 2055 -c 5
obserae-cli --socket /var/lib/obserae/run/obserae.sock status

For a brand-new NetFlow v9/IPFIX exporter, data packets may arrive before the template that describes them. Wait for the next template refresh or restart the exporter.

The daemon will not start: “Failure while replaying WAL file”

open duckdb: … INTERNAL Error: Failure while replaying WAL file
"…/obserae.duckdb.wal": Calling DatabaseManager::GetDefaultDatabase with no
default database set

The database file is intact — it is consistent as of the last checkpoint. Only its write-ahead log cannot be replayed, which a DuckDB defect can cause after an unclean stop (a kill -9, a power cut, an out-of-memory kill) that happened between a version upgrade and the next checkpoint. obserae now checkpoints every schema migration as it is applied, so a data directory upgraded by a current build cannot reach this state.

Your flows, sessions, alerts and audit journal are not affected: they live in Parquet and JSONL files, not in the database. What is lost is whatever the database recorded since the last checkpoint — mostly state obserae rebuilds by itself, such as evaluation cursors and collector results.

With the service stopped:

sudo systemctl stop obserae
sudo -u obserae cp -a /var/lib/obserae/obserae.duckdb     /var/tmp/   # keep a copy first
sudo -u obserae cp -a /var/lib/obserae/obserae.duckdb.wal /var/tmp/
sudo -u obserae strings -n 4 /var/lib/obserae/obserae.duckdb.wal | head -20

The first table and column names printed identify the upgrade step that was in flight; the database file stands at the step before it. Move the log aside, set /var/lib/obserae/data_version to that earlier number, and start the service: it re-applies the pending steps and this time makes them durable.

sudo -u obserae mv /var/lib/obserae/obserae.duckdb.wal /var/tmp/obserae.wal.broken
sudo systemctl start obserae
sudo journalctl -u obserae -n 50        # "data migration complete from=… to=…"

If you would rather not edit data_version by hand, restore the most recent backup instead — the tradeoff is losing everything since that snapshot rather than since the last checkpoint.

Login loops back to the login page

Use HTTPS through a reverse proxy. A browser will drop a secure session cookie received over remote plain HTTP. For a trusted lab-only HTTP deployment, set:

web:
  secure_cookies: false

In Docker Compose, the equivalent override does not require mounting a custom configuration file:

environment:
  OBSERAE_WEB_SECURE_COOKIES: "false"

The session cookie then crosses the network without transport protection. Do not use this setting on an untrusted network.

Disk keeps filling

Enable retention or shorten the retention windows. Then run a cleanup from the Lifecycle page.

df -h /var/lib/obserae
obserae-cli retention status

A rule is quarantined

A rule usually becomes quarantined because it references a cartography entity that was renamed or removed. Either restore the missing entity, update the rule, or delete it.

obserae-cli rule ls
obserae-cli rule show NAME

Memory keeps growing on a small host

Set a memory limit and enable retention:

storage:
  memory_limit: "512MB"
  max_threads: 2

retention:
  enabled: true
  flows_max_age: 168h
  sessions_max_age: 720h

If heap metrics grow steadily for hours, enable the profiler briefly on localhost and investigate from there:

debug:
  pprof_enabled: true
  pprof_address: "127.0.0.1:6060"

Do not expose the profiler on the network.

See Also

• Configuration - YAML keys and tuning recipes.
• Sizing - disk, memory and CPU for your traffic and retention.
• Lifecycle - retention and backup settings.
• Monitoring - GUI operational dashboard.
• CLI - command reference.
• Installation - Docker and binary install paths.