Troubleshooting

Organised by what you can see, not by what is wrong, because when something fails you have a symptom and not a diagnosis. Search this page for the string your terminal or your dashboard is showing you.

Every failure here is one somebody has actually hit.

The collector printed nothing and exited 0

This is usually success. The collectors are quiet when they work, because a script that chatters on every scheduled run trains people to ignore its output.

Check what it did:

# macOS
sudo cat "/Library/Application Support/nyxus/last_scan.txt"
# Linux
sudo cat /var/lib/nyxus/last_scan.txt

claude-code(example.com);codeium [posted=2 suppressed=0] (2026-08-08T21:40:38Z)

• posted=N findings were sent to the receiver.

• suppressed=N findings were found but not sent, because the same finding was already reported inside the throttle window. Info findings repeat daily, warn findings hourly. This is not an error.

• none the scan ran and found no AI tooling the registry knows about.

posted=0 suppressed=2 on a second run is correct behaviour, not a failure. To force a full report while testing, move the state directory aside and run again:

sudo mv "/Library/Application Support/nyxus" /tmp/nyxus-state.bak

Put it back afterwards, or the next scheduled run reports everything as new.

Refusing to scan: an empty scan looks like a clean machine

The collector could not find its configuration, so it stopped rather than producing a scan with no findings. A machine that reports nothing and a machine with nothing to report look identical on a dashboard, and across a fleet that is false confidence.

macOS and Windows take positional parameters, not environment variables, because that is how Jamf and Intune pass them. Parameters 1 to 3 are Jamf's own, so running by hand means passing three empty strings first:

sudo ./nyxus-collector.sh "" "" "" \
  https://receiver.example.com <token> example.com

Linux takes environment variables, because RMM agents and cron set those:

NYXUS_RECEIVER_BASE=https://receiver.example.com \
NYXUS_TOKEN=<token>

refusing to scan without an identifier list

The collector fetched /registry/collector from the receiver and did not get it. It refuses to continue for the same reason as above: without the list of what to look for, every machine looks clean.

Check the receiver is reachable from the machine, with the token:

curl -s -o /dev/null -w "%{http_code}\n" \
  -H "Authorization: Bearer <token>" \
  https://receiver.example.com/registry/collector

• 200 the receiver is fine, so the problem is the URL the collector was given. A trailing slash or a missing scheme is the usual cause.

• 401 wrong token.

• 000 DNS, TLS or network. The collector runs as root, so check root's view of the network rather than yours: a proxy set in a user profile does not apply.

POST failed: HTTP 401

The token the collector holds is not the token the receiver was deployed with.

# Kubernetes
kubectl -n <ns> get secret nyxus-receiver -o jsonpath='{.data.authToken}' | base64 -d; echo
# Compose
cat deploy/compose/secrets/auth_token

Watch for a trailing newline. echo "token" > secrets/auth_token writes one and printf does not, and a token with a newline on the end fails against one without.

Findings are accepted but nothing appears in Grafana or the portal

This is the most misleading failure in the system, so check it first when data is missing.

A failed push answers the collector with a 503, so collectors keep the finding and retry, and the failure shows up as POST-FAILED in collector logs rather than as silence. It is worth knowing what the alternative would look like, because it is the shape this is built to avoid: a receiver that answered 200 when it could not write to the log store would satisfy every collector, the finding would exist in its stdout, and nothing would reach either view. The receiver's own metrics name the cause:

Check the receiver's own metrics:

curl -s http://<receiver>/metrics | grep nyxus_loki_push

nyxus_loki_push_total 1432
nyxus_loki_push_failures_total{reason="HTTPStatusError"} 1432
nyxus_loki_push_last_success_timestamp 0

last_success_timestamp at 0 means it has never successfully written. Then read the logs, where the reason is stated:

kubectl -n <ns> logs -l app.kubernetes.io/name=nyxus --tail=50 | grep -i loki
docker compose logs receiver | grep -i loki

loki push rejected with HTTP 404: this looks like a base URL rather than the
push endpoint, which is /loki/api/v1/push

LOKI_PUSH_URL and LOKI_URL are different values on purpose.

Posting to the base returns 404. Querying the push endpoint returns nothing.

All the receiver's counters are in memory and reset when the pod restarts. So zeros immediately after a deploy mean nothing at all. last_success_timestamp is the one to read, because it distinguishes "restarted a minute ago" from "has never worked".

The portal says the subscription could not be confirmed

The receiver checks with registry.nyxus.co.uk every few hours. This message means those checks have been failing for at least a day, and it is almost always outbound access rather than anything to do with the subscription itself.

It is not urgent on the day it appears and it does not mean anything has stopped. New findings and enrolments continue for fourteen days from the first failed check; the banner counts down. After that they stop, and everything already collected stays readable - the portal, the history and the reports are unaffected either way.

Check the receiver can reach it:

kubectl -n <namespace> exec deploy/nyxus -- \
  python -c "import urllib.request;print(urllib.request.urlopen('https://registry.nyxus.co.uk/updates/latest',timeout=8).status)"

A timeout or a connection error is an egress rule. If you apply the example NetworkPolicy, that is the rule for port 443 to outside the cluster; if the block is a proxy or a firewall allowlist, add the hostname beside the ones you already allow for Entra and your MDM.

403 or 503 means you reached us and we answered - that is our problem, not a rule of yours. It resolves on its own, and the fourteen days are there so it never becomes yours.

To ask again immediately rather than waiting for the next check, an owner can POST to /admin/subscription/check on the receiver.

If outbound access is genuinely not possible, SUBSCRIPTION_CHECK=off stops the receiver asking. The term in the key still applies - it is signed, and expiry is read from the key rather than from us - but a withdrawn subscription can no longer reach the deployment. Tell us if you set it, so we do not read the silence as a blocked firewall.

Pods sit at ImagePullBackOff, or Compose fails on the first pull

The images are private and nothing reached the registry with a valid licence. The pod event says which of the three it is.

unauthorized or denied: no credential was presented. On Kubernetes the release needs a pull secret and needs to be told about it - creating the secret is not enough on its own:

# The pull secret comes from a file only you can read, not from an argument
# anyone on the machine can see.
(umask 077; printf '{"auths":{"registry.nyxus.co.uk":{"username":"licence","password":"%s"}}}' "$NYXUS_KEY" > nyxus-pull.json)
kubectl create secret generic nyxus-registry --namespace nyxus \
  --type=kubernetes.io/dockerconfigjson --from-file=.dockerconfigjson=nyxus-pull.json
rm nyxus-pull.json

helm upgrade nyxus ... --set 'imagePullSecrets[0].name=nyxus-registry'

On Compose it is docker login registry.nyxus.co.uk with the same credential.

manifest unknown: the credential worked but the tag does not exist. Usually a chart pinned to a version your licence does not cover, or a typo in image.tag.

unauthorized on a deployment that worked yesterday: the licence lapsed. Already-running pods keep serving - the key gates pulling, not running - so this surfaces on the next upgrade or the next node that has to pull the image fresh, which is why it often appears as one replica failing while the rest are fine.

The secret lives in the namespace the release is in. A secret in default does nothing for pods in nyxus.

The portal will not start

refusing to start without authentication

The portal names who runs what on which machine, so it will not come up unauthenticated by accident. Either set a password, let the chart generate one, or set PORTAL_AUTH_MODE=none deliberately when something in front of it is doing the authenticating.

kubectl -n <ns> get secret nyxus-portal -o jsonpath='{.data.password}' | base64 -d; echo

The portal starts but every page is empty

LOKI_URL is not set

The portal has no findings to read. If it is set, check it is the base URL and not the push endpoint, and that it is reachable from inside the cluster or compose network rather than from your laptop.

If LOKI_URL is right and pages are still empty, the findings are not in Loki. Go to the section above about accepted findings that never arrive.

A source shows not reporting but I definitely deployed it

The setup page derives what is reporting from findings that carry that source value, not from configuration existing. A source with no findings bearing its name is listed as silent.

Two different causes:

1. It genuinely has not reported. Nothing was scheduled, credentials are missing, or it has run and found nothing.

2. It is reporting without setting source. The findings arrive and are counted everywhere else, but they cannot be attributed to a detector, so the source looks silent while the data is present.

Tell them apart by looking for findings that match the source's surface but carry no source:

sum by (surface) (count_over_time({app="nyxus-receiver", kind="finding"}
  | json source="source" | source = `` [24h]))

A large count on a surface whose source says not reporting is cause 2.

helm install fails with an ownership error

Error: rendered manifests contain a resource that already exists ...
invalid ownership metadata

Something in the namespace has the name the chart wants and was not created by Helm. This is common when adopting the chart alongside an existing hand-rolled deployment: the registry ConfigMap is the usual one.

Point the chart at what already exists rather than letting it create its own:

registry:
  create: false
  existingConfigMap: nyxus-registry
auth:
  existingSecret: nyxus-receiver

Reusing the existing secret also means every collector, extension and scanner keeps its current token, so nothing has to be redeployed.

The receiver crash-loops on unable to open database file

sqlite3.OperationalError: unable to open database file

The receiver keeps its state DB on a PVC, and the PVC arrived owned by root. Most StorageClasses hand volumes over that way; it is fsGroup in the pod security context that makes the kubelet chown the mount to the pod's group. The chart runs the receiver as uid 65532, so without fsGroup that uid cannot create /var/lib/nyxus/state.db and the pod dies before serving anything.

The chart sets fsGroup: 65532 by default. With a values file that replaces podSecurityContext wholesale, add it back:

podSecurityContext:
  fsGroup: 65532

and helm upgrade. The pod recovers on the next start; nothing on the volume needs repair.

ImagePullBackOff

kubectl -n <ns> describe pod <pod> | tail -20

• manifest unknown the tag does not exist. The chart defaults the tag to its appVersion, so a chart version ahead of the published images asks for one that was never built.

• unauthorized a private registry with no pull secret. The published images are public, so this usually means the values were pointed at a private mirror.

The same machine appears twice

The device field changed, so the two halves of its history do not join.

The collectors report the most stable identifier available: a hardware serial, then a machine id, then the hostname. If a machine used to report a hostname and now reports a serial, or the reverse, it becomes two devices.

macOS says so when it happens:

[nyxus] serial lookup failed via ioreg, falling back to hostname for device

Old findings age out of the portal's lookback window on their own. There is nothing to repair, but the count is wrong until they do.

The invite email never arrived

Three different failures wear the same symptom, and the portal tells them apart if you look in the right place.

The portal said nobody was emailed. Then nothing was sent, and it is not a delivery problem. Either no relay is configured, or the account was made without an address. Settings > Account has the relay wizard; once it is configured, the button that resends to everybody who was missed closes the gap for accounts created before it.

The portal reported an error from the relay. That string is what the relay said, passed through. 535 or Authentication unsuccessful is the credential. 550 or Relay access denied means the relay took the connection and refused to carry mail for that sender - the usual cause is From not being an address that relay is willing to send as. A timeout usually means egress: cluster network policy, or a provider that blocks outbound 25.

The relay accepted it and it still is not there. Then it left this platform and the trail continues in your mail infrastructure. 250 Queued mail for delivery is an acceptance, not a delivery. Two things account for most of it: the message went to junk, and the recipient's tenant filtered it as a spoof because it claims to come from a domain that tenant owns while arriving from a relay it does not recognise. Send the test to an address on a different domain to tell those apart quickly - if that one arrives, the problem is your own tenant's anti-spoof handling, not this platform.

Use the test send in the relay wizard rather than creating accounts to experiment. It reports exactly what the relay said, and it makes no account.

Sign in with Microsoft is not on the login screen

It appears once federated sign-in is switched on, and not before, because a button that looks live and is not makes people doubt their own password. If you have configured it and the button is missing, the last wizard step - a real sign-in - has not been completed: nothing takes effect until it has.

A federated sign-in is refused

The refusal page says which check failed. The common ones:

• No account carries that address. Expected: it authenticates, it does not provision. Create the account first, with that address on it.

• That sign-in is too old. The provider answered with the time the person last actually authenticated, and it is outside the window this deployment asks for. Signing in again fixes it. Widen the window under Settings > Account if twelve hours is wrong for you.

• The provider did not say when this person last signed in. The token came back with no auth_time claim, so freshness cannot be established and is not assumed. Setting the window to "every time" is not a workaround; the claim is what the check reads.

• This account is bound to a different identity. Binding is permanent by design, because an address can be reassigned and rebinding would hand somebody else's account away. Unbind it deliberately from Settings > Account.

Locked out after requiring single sign-on

Sign in as the account the setup code created. It keeps its password precisely for this, it cannot be deleted or demoted while enforcement is on, and once you are in you can turn enforcement off from Settings > Account.

If that password is lost too, the receiver's ADMIN_TOKEN credential resets it: POST /admin/password with that token and a new password targets the oldest admin account, no current required. See the endpoint table in How Nyxus is put together. It needs access to the receiver's environment, which is the point - recovery belongs to whoever owns the box and to nobody else. ADMIN_TOKEN is optional and unset by default, so this path exists only if you set one up in advance.

A backup failed

Deployment › Backups gives the reason on the failed run. The common ones:

• no backup key is configured, or there is no key in ...: the receiver has no key file. See Backups.

• the backup key file is not base64 / must be 32: the file is there but is not a key. Make it with openssl rand -base64 32, with nothing else in it.

• S3 refused the upload (HTTP 403) - AccessDenied: the access key cannot write to that bucket and folder. The policy must allow s3:PutObject on arn:aws:s3:::<bucket>/<folder>/*; check the folder matches.

• S3 refused the upload (HTTP 400) - AuthorizationHeaderMalformed: the region is not the bucket's region.

• Azure Storage refused the upload (HTTP 403) - AuthenticationFailed: the SAS has expired or was pasted incomplete. Generate a new one with Create and Write, and paste the whole URL.

• ... could not be reached: ConnectTimeout: the receiver has no route to the storage endpoint. Allow egress to *.s3.<region>.amazonaws.com or <account>.blob.core.windows.net.

A failed backup keeps its copy on the volume, and the next attempt is an hour later. Back up now tries again straight away.

Something else

Report it through whoever your subscription names, with what you can see: the symptom, the component, the version from System health, and the output of the relevant check above. Releases and support says which releases are supported and what that covers; a security fault goes privately instead, by the route the security guidance sets out. A symptom plus the failing string is enough to work with; a description of what you expected instead is not.