Lifecycle

The Settings group contains three pages that keep an obserae instance healthy over time:

Changes made from these pages are persisted by obserae and survive daemon restarts. The YAML configuration provides the initial defaults; the UI becomes the source of truth once an operator changes a lifecycle setting.

To work out how much disk a given retention window will actually cost before you set one, see Sizing.

Storage

The Storage page answers one question: is this instance keeping data at the rate and depth I expect?

It shows:

• the database location and size;
• free disk on the same mount;
• flow/session history size;
• enrichment and audit-log footprint;
• backup directory size;
• freshness and time span for retained history.

The parquet stores are grouped into two blocks. Parquet stores holds the time-series data (flows, sessions, archives, audit log, alerts); each shows a size, a freshness badge and a small bar timeline of how bytes are spread over time. Reference data holds the stores that have no time axis — enrichment ranges (broken down by source), the anomaly baseline (broken down by rule) and the rendered reports — shown as ranked bars so you can see, at a glance, which sources, rules or documents take the most space.

Reports keep themselves in check

The documents produced by Reports are stored as files, one folder per report, and they appear on this page like any other store. You never have to prune them: the store enforces its own limits and drops the oldest reports first when one is reached.

180 days is chosen so a quarterly pack survives two quarters plus the time an audit takes to get to it. Retention settings do not apply here: retention governs the traffic obserae observed, and a report is a document obserae produced — it can always be generated again from the same window.

Backups carry your report history and schedules, not the PDF and HTML files themselves, which would make every archive far larger for documents that can be re-produced on demand. After a restore, obserae removes the history rows whose files did not come back, so the page never offers a download that cannot work.

Use it during normal operation to spot three common problems:

The exact on-disk format is an implementation detail. Operationally, treat the values as “database”, “retained traffic history”, “audit log” and “backups”.

Retention

Retention is off by default. That makes a fresh install easy to evaluate: you can collect a few days of traffic, estimate daily growth, then choose a policy deliberately.

Turn retention on once you know how much history you want to keep.

Set a value to 0 when that category should be kept forever. Be careful with the audit log: it is the forensic trail for operator actions, login events and configuration changes.

Cleanup

The page shows the last cleanup, the next automatic cleanup and what was removed. Click Clean up now after lowering a max age or when you need to reclaim space immediately.

A manual cleanup does not reset the automatic schedule.

First Cleanup On An Old Instance

If an instance has been running for weeks or months with retention off, the first cleanup can remove a lot of history. Run it during a quiet window if the machine is small or the stored history is large.

Backup

Backups capture the state needed to recover an obserae instance:

• cartography, flow matrix, alerting, outputs, users, connectors and settings;
• retained flow/session history;
• enrichment state;
• audit log history.

Backups run as a list of independent backup processes (see Backup processes below) — each with its own cadence, retention and target. There is no single global backup policy any more: a local snapshot process writes an on-disk full+delta chain to its own directory, and you can run several (for example one to a local disk and one to a mounted NAS partition). Each local process sets its directory, the daily full time, the delta interval, and how many fulls/deltas to keep (by count and/or age).

Every process has a Run now button that captures immediately — handy before an upgrade, a large rule import or a big cartography change — and shows a live progress bar + ETA. Large snapshots can briefly slow ingestion while the instance captures a consistent state; on small installs this is usually short, but schedule production backups for quieter periods when possible.

Restore

Restore runs from the Restore page or the CLI. The Restore page can pull a .tar.zst archive from a configured offsite destination, restore an uploaded local archive (master key first — validated against the archive before anything is replaced), or restore configuration only. From the CLI, always preview the restore plan before applying it:

obserae-cli backup list
obserae-cli backup plan --point-in-time 2026-05-28T13:00:00Z
obserae-cli backup restore --point-in-time 2026-05-28T13:00:00Z --dry-run --confirm
obserae-cli backup restore --point-in-time 2026-05-28T13:00:00Z --confirm

After a successful restore, obserae exits so your supervisor can restart it on the restored state. This is expected.

Keep the master key safe. It protects stored secrets and audit-log seals. See Operations before rebuilding an instance from backups.

Backup processes

The Backup page is a list of independent scheduled backup processes — so you can run, say, a local copy every hour and an offsite copy once a day, each on its own cadence and retention. A status strip across the top gives the at-a-glance picture an operator needs: how many processes exist, how many are enabled, whether any is failing, the next scheduled run, and how long since the last success.

“Add backup” opens one self-contained form. A process is either:

• a Local snapshot — an on-disk full+delta chain that powers point-in-time restore. Each local process has its own directory (a path on the host or a mounted NAS partition), so you can run several independent local backups; the form sets the directory, the daily full time, the delta interval, and the fulls/deltas retention. No destination; or
• an Offsite archive — where the same form also configures where the portable .tar.zst is shipped, so a local backup directory (a single point of failure on a one-host deployment) is no longer the only option:
  • Local folder — a path on the host or a mounted NAS;
  • S3-compatible — AWS S3, MinIO, Backblaze B2, Wasabi, Ceph… (endpoint, region, bucket, prefix, access key + secret);
  • SFTP — host, port, user, path, and either a password or a private key.

An archive process also sets:

• a schedule — every N minutes/hours, daily at a time, or weekly on chosen days (all times UTC);
• a retention policy — either keep the last N, or smart (GFS): keep the newest of the last N days, N weeks and N months (default 7 / 4 / 3).

(A local snapshot process instead uses its own full/delta cadence and fulls/deltas retention, set in the form as described above.)

Click a process to open a detail panel on the right: its configuration, the last-run outcome (a green tick, or a red error chip spelling out the failure), the next run, its run history, and a Run now button. Run now backs up immediately — an offsite process ships the latest point-in-time to its destination, the local-snapshot process takes a snapshot — and shows a live result right there (running → ✓ shipped, or ✗ with an actionable hint, e.g. an egress-blocked address or “take a local snapshot first”), so it is never a silent no-op. It honours the process’s own retention. For an archive process the panel also offers Test connection; the local-snapshot process’s panel hosts the point-in-time timeline (below). Creating, editing or deleting an archive process manages its destination together with its schedule — there is no separate destinations list to keep in sync.

You can also Download the current point-in-time straight from the browser (from the local-snapshot process’s panel). If no snapshot exists yet, the download takes a fresh full snapshot on demand and streams that — so the first download just works.

Connection test & host-key pinning. For SFTP, Test connection returns the server’s host-key fingerprint on first use; confirm it and it is pinned (subsequent connections are refused if the key changes). If a host cannot be pinned you may tick Skip host-key verification (insecure) — it disables the check entirely (man-in-the-middle risk) and is surfaced with a warning; an S3 destination has the same escape hatch for a self-signed endpoint via a CA certificate (PEM) or Skip TLS verification. All secrets are stored encrypted at rest and never shown again (only “set” / “replace”).

Egress guard (SSRF protection) & the “egress: destination address not permitted” error

Every offsite upload goes through an egress guard: by default the daemon refuses to connect to a loopback, private (RFC1918) or link-local address (including the cloud-metadata IP 169.254.169.254). This stops an operator with backup:manage from pointing an upload at an internal service or the host’s own metadata endpoint. The check runs on the resolved IP at dial time, so it also covers DNS rebinding and redirects.

So a Test connection (or a scheduled ship) to a destination on your LAN fails with:

sftp: dial 192.168.10.2:22: … egress: destination address not permitted: 192.168.10.2

This is expected, not a bug — the target is a private address. Note it is an application-level guard inside the daemon: that your own ssh/sftp from a shell works says nothing about it (a different mechanism entirely).

To allow a legitimate internal target (e.g. a LAN MinIO or SFTP host), add its CIDR to backup.egress_allow_cidrs in configs/obserae.yaml, then restart the daemon — this policy is read once at boot and is not hot-reloaded:

backup:
  egress_block_internal: true                 # keep the guard on
  egress_allow_cidrs: ["192.168.10.0/24"]     # …but allow this segment

Prefer the tightest CIDR that covers your target (a /32 for a single host). A malformed entry fails startup loudly rather than silently widening egress. The guard is shared with alert outputs, which has its own outputs.egress_allow_cidrs.

From the CLI, destinations and jobs remain separately scriptable:

obserae-cli backup destinations list
obserae-cli backup destinations add --name nas --kind sftp \
  --config '{"host":"nas.lan","port":22,"username":"obserae","path":"/backups","auth":"password","host_key_fingerprint":"SHA256:…"}' \
  --secret 'the-password' --keep 7
obserae-cli backup ship --destination <id>

Restore from an archive (master key first)

To rebuild a machine from a downloaded or offsite archive, use Restore from an archive on the Backup page (or the CLI). Because the archive excludes the master key by design, you must supply the matching masterkey.bin from the source host alongside the archive. It is validated before anything is touched — archive format, DuckDB build, and a sealed canary that proves the key matches — so a wrong key or an archive from a different obserae release is refused up front with the live database intact. Only then is the database swapped and the master key installed, after which obserae exits for the supervisor to restart it.

obserae-cli backup restore-from-archive \
  --file obserae-backup-20260716T030000.tar.zst \
  --master-key-file masterkey.bin --confirm

The restore host must run the same obserae release as the archive (it pins the DuckDB build). Restoring across DuckDB versions is refused with a clear message rather than risking silent corruption.

Restore page (full restore or configuration only)

Retention, Backup and Restore live together under the Data Protection menu. The Restore page is full width: a Master key card on top, then two side-by-side choices — so it is always clear what a click will do.

• Master key — paste or upload the source host’s masterkey.bin; it stays in the page only and validates the archive. It is required only for a full restore (a configuration-only restore does not use it).
• Full restore (data + config) — pick a configured destination, choose one of its remote archives, and click Restore. obserae downloads the archive, validates it master-key-first (format, DuckDB build, canary) before touching the live database, then restores it — showing a multi-step progress bar + ETA (download → unpack → apply) and the same topbar indicator. Because the archive is the whole instance, this restores flow data and configuration. It replaces the manual “download then re-upload” round-trip for offsite recovery.
• Configuration only — restore just a configuration bundle (cartography, rules, outputs, …) without touching flow data, using the same async, progress-streamed flow as a configuration restore.

A full data restore replaces the live database and is admin-only (system:manage).

Configuration bundle vs Backup

These two features solve different problems:

Use both in production: export configuration for review/version control, and run backups for recovery.

See Also

• Operations - production deployment, master key and recovery.
• Configuration - lifecycle defaults in YAML.
• CLI - backup and restore commands.
• Web GUI - where these pages appear in the interface.