Backups

Nyxus backs up everything it holds that cannot be rebuilt, on a schedule, sealed with a key only you keep, to storage you own. Restoring is done from your own machine with nyxusctl.

This page covers what is backed up, how, setting it up, watching it, and restoring.

What is backed up

The receiver's state directory, which is the only thing in Nyxus that is not disposable:

• state.db: the hosting company's records, and every company's accounts, devices, enrollment tokens, sessions and API tokens.

• companies/<id>.db: each further company's own records.

Not included, because Nyxus does not own them:

• Findings. They live in your log store, and are backed up, or not, with it. The estate history that Nyxus records each day is in the databases above.

• Configuration. Your Helm values, your Compose .env and the files in secrets/ belong in whatever keeps your configuration.

• Grafana, whether bundled or your own.

How it works

The receiver makes each backup itself. It is the only process that writes those files, on a volume only one pod can hold.

1. Each database is copied while the receiver runs, with SQLite's own backup API, and the copy must pass SQLite's integrity check.

2. The copies go into a gzipped tar with a manifest: the Nyxus version, the time, the key's id, and each file's size and SHA-256.

3. The tar is sealed with AES-256-GCM, in 1 MiB chunks, before it is written anywhere. Each chunk is authenticated, and so is the end of the archive: a backup that has been changed, truncated or extended does not open.

4. The archive, a .nyxb file, is written to backups/ on the receiver's volume and sent to your S3 bucket or Azure container. The newest copies are kept on the volume (three unless you choose otherwise), which covers a bad upgrade but not a lost disk.

A backup runs every 24 hours by default, or every 6 or 12. A failed one is tried again an hour later, not a day later. Every run is listed under Deployment › Backups with its size and where it went, or why it failed, and recorded in the audit trail.

Backups are made one database file after another, over a few seconds. A change made in those seconds, such as an account being created, can be in one file and not the next. There is no point-in-time recovery between backups.

The backup key

Nothing is backed up until the receiver has a key. The key is 32 random bytes, base64, in a file. The receiver reads it when it makes a backup and never writes it anywhere, so a backup never carries its own key.

Keep a copy of the key somewhere other than this deployment, such as a password manager or a vault. Without it no backup can be opened, and a key lost with the deployment makes every backup of it useless. With it, anyone holding a backup can read it.

Deployment › Backups shows the key's id, a fingerprint that names it without revealing it. Every backup records the id of the key that sealed it.

Kubernetes

openssl rand -base64 32 > nyxus-backup.key
kubectl create secret generic nyxus-backup-key --namespace nyxus --from-file=key=nyxus-backup.key

Then, in your values:

backup:
  existingSecret: nyxus-backup-key

and run your usual helm upgrade. The key is mounted read-only at /run/secrets/nyxus-backup/key, readable by the receiver's group only.

Docker Compose

From the directory with your docker-compose.yml:

mkdir -p secrets/backup
openssl rand -base64 32 > secrets/backup/key
chmod 640 secrets/backup/key
sudo chown :65532 secrets/backup/key   # not needed on Docker Desktop
docker compose up -d receiver

The Compose file already mounts secrets/backup/ into the receiver. With no key there, the stack starts as before and nothing is backed up.

Changing the key

Give the receiver the new key the same way. Backups from then on are sealed with it. Earlier backups still need the key that sealed them, so keep the old key until those backups have aged out of your storage.

Where backups go

An owner of the company hosting Nyxus sets this under Deployment › Backups. The credentials are stored in the receiver's database and never shown again; they are included in its backups, sealed with the key.

Give the credentials permission to write, and nothing else. A deployment that can only write cannot read old backups or delete them if it is ever compromised. Retention is then the storage's job rather than Nyxus's.

Amazon S3

You need a bucket, its region, and an access key for an IAM identity with this policy, with your bucket and folder:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "NyxusBackupsWriteOnly",
    "Effect": "Allow",
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::nyxus-backups/nyxus/*"
  }]
}

Turn on versioning, or Object Lock, so a backup cannot be overwritten, and add a lifecycle rule to expire old backups. Nyxus sends each file's MD5 and SHA-256 with it, so S3 checks what it received, and a bucket with Object Lock accepts the write.

Azure Blob Storage

You need a SAS URL for the container, with Create and Write only (sp=cw) and an expiry you will renew. It looks like https://<account>.blob.core.windows.net/<container>?sv=...&sp=cw&...&sig=....

Turn on version-level immutability so a backup cannot be changed, and add a lifecycle management rule to delete old backups. When the SAS expires, backups start failing and the page says so. Paste a new one to carry on.

Testing the destination

Test destination writes a small file, nyxus-destination-test-<time>.txt, where backups go. Write-only credentials cannot delete it again; remove it in the storage console, or let the lifecycle rule do it.

Who can do what

This applies to accounts of the company hosting Nyxus. Backups are the whole deployment's, so an account of any other company, and any team viewer, is refused all of it.

Watching it

Deployment › Backups shows the state at a glance. It asks for attention when the key cannot be read, the destination is incomplete, the last backup failed, or none has succeeded for half a day longer than the schedule allows.

The receiver's metrics carry the same:

• nyxus_backup_last_success_timestamp: the Unix time of the last backup that was made and delivered. Alert on its age.

• nyxus_backup_failures_total: backups that did not complete.

- alert: NyxusBackupsStale
  expr: time() - nyxus_backup_last_success_timestamp > 36 * 3600
  for: 15m
  annotations:
    summary: No Nyxus backup has succeeded for a day and a half

Space on the volume

Each copy is roughly the size of the databases compressed. Keeping three copies needs room for three beside the databases themselves. The state volume is 1 GiB by default (managed.persistence.size), which is ample for most deployments; the page shows how much the copies use.

Settings

The destination, folder, schedule and number of copies kept are set on the page, not in the environment.

The archive format

For anyone who needs to open one without Nyxus:

• Header (24 bytes): NYXB, a format byte (1), the key id (8 bytes), a nonce prefix (7 bytes), and the chunk size (4 bytes, big-endian).

• Chunks: AES-256-GCM with a 16-byte tag. Each chunk's nonce is the prefix, a 4-byte big-endian counter from 0, and a byte that is 1 on the last chunk and 0 otherwise. The header is the additional authenticated data for every chunk.

• Key id: the first 8 bytes of SHA-256 over nyxus backup key, a zero byte, and the key.

• Plaintext: a gzipped tar holding manifest.json, then state.db, then companies/<id>.db for each company.