Operations: Health Checks & Backups
Once the Connector is installed — by any of the paths in Installation — treat it as a long-running service: monitor it continuously, and back up its database. This guide covers both, and applies to bundle installs, Docker Compose, and Kubernetes deployments alike, except where noted.
Ongoing health checks
Section titled “Ongoing health checks”The Connector is (at minimum) three cooperating services: the API, the dashboard, and PostgreSQL. Each has a cheap, unambiguous probe you can wire into whatever monitoring you already run (Nagios/Icinga, Zabbix, Prometheus blackbox exporter, Datadog, Uptime Kuma, a cron job that pages you — anything that can run a command or fetch a URL).
| Service | What runs it | Probe | Healthy when |
|---|---|---|---|
| Connector API | a10e-bundle.service (bundle installs) or your container orchestrator |
curl -fsS http://127.0.0.1:8080/v1/healthcheck |
HTTP 200 |
| Dashboard | Docker container a10e-bundle-dashboard (bundle installs) |
curl -fsS http://127.0.0.1:3000/api/health |
HTTP 200 |
| PostgreSQL (bundled) | Docker container a10e-bundle-postgres |
docker exec a10e-bundle-postgres pg_isready -U a10e -d a10e_db |
exit 0, prints “accepting connections” |
| Release integrity (bundle installs only) | the a10e CLI |
/opt/a10e/current/bin/env-manager bundle health |
exit 0 — API and dashboard answer and the running unit belongs to the active release |
The last row is the same check the CLI uses during install and upgrade. It’s stricter than the two curls on their own because it also confirms the process serving on those ports is the release /opt/a10e/current points at, so a stale process from a failed upgrade can’t look healthy.
A single script for all three services (bundle installs)
Section titled “A single script for all three services (bundle installs)”Save this as /usr/local/bin/a10e-healthcheck, mode 0755. It exits 0 only when every probe passes and prints one line per service, so it works as a Nagios-style plugin and reads well in a log.
#!/bin/sh# a10e-healthcheck — exit 0 only if every Connector service is healthyset -urc=0check() { # name, command... name=$1; shift if "$@" >/dev/null 2>&1; then echo "OK $name" else echo "FAIL $name"; rc=1 fi}
check "a10e.service active" systemctl is-active --quiet a10e.servicecheck "a10e-bundle.service active" systemctl is-active --quiet a10e-bundle.servicecheck "postgres container running" sh -c '[ "$(docker inspect -f "{{.State.Running}}" a10e-bundle-postgres 2>/dev/null)" = true ]'check "postgres accepting connections" docker exec a10e-bundle-postgres pg_isready -q -U a10e -d a10e_dbcheck "dashboard container running" sh -c '[ "$(docker inspect -f "{{.State.Running}}" a10e-bundle-dashboard 2>/dev/null)" = true ]'check "API /v1/healthcheck" curl -fsS --max-time 5 http://127.0.0.1:8080/v1/healthcheckcheck "dashboard /api/health" curl -fsS --max-time 5 http://127.0.0.1:3000/api/healthcheck "release health (a10e)" sh -c 'cd /opt/a10e/current && ./bin/env-manager bundle health'
exit $rcRun it as root (it needs Docker and the release directory):
sudo a10e-healthcheckIf you set A10E_DB_DSN to point the Connector at your own PostgreSQL, drop the two postgres rows and probe your database server instead.
Wiring it into monitoring
Section titled “Wiring it into monitoring”systemd timer, alerting via the journal. Runs every minute; failures are visible in journalctl -u a10e-healthcheck and can be forwarded by your log shipper.
sudo tee /etc/systemd/system/a10e-healthcheck.service >/dev/null <<'EOF2'[Unit]Description=Alien Giraffe Connector health checkAfter=a10e-bundle.service
[Service]Type=oneshotExecStart=/usr/local/bin/a10e-healthcheckStandardOutput=journalStandardError=journalEOF2
sudo tee /etc/systemd/system/a10e-healthcheck.timer >/dev/null <<'EOF2'[Unit]Description=Run the Connector health check every minute
[Timer]OnBootSec=2minOnUnitActiveSec=1minAccuracySec=10s
[Install]WantedBy=timers.targetEOF2
sudo systemctl daemon-reloadsudo systemctl enable --now a10e-healthcheck.timerjournalctl -u a10e-healthcheck -n 20 --no-pagerHTTP monitors. If your monitoring tool only fetches URLs, point two HTTP checks at the host: http://<host>:8080/v1/healthcheck and http://<host>:3000/api/health, expecting status 200. PostgreSQL is bound to 127.0.0.1 only, so it can’t be probed remotely — cover it with the script above or with pg_isready run through your agent.
Prometheus. Use the blackbox exporter’s http_2xx module against the two URLs, and the node exporter’s textfile collector to publish the script’s result, for example by appending a10e_healthy 1 or a10e_healthy 0 to a .prom file from the timer.
Whichever tool you use, alert on two things: any probe failing for more than two consecutive runs, and (for bundle installs that leave unattended upgrades on) a10e-update.timer reporting a failed run in journalctl -u a10e-update.service.
PostgreSQL data, persistence, and backups
Section titled “PostgreSQL data, persistence, and backups”The bundled PostgreSQL holds all Connector state: configuration, users, request records, audit history. Everything else on the host — release directories, images, containers — can be rebuilt from the release bundle. The database cannot. Treat it as the one thing on the host that must survive.
This section covers the bundled-PostgreSQL case (bundle installs with no A10E_DB_DSN set). Docker Compose and Kubernetes deployments manage their own Postgres persistence — see Deployment Model — but the same pg_dump approach below applies regardless of how Postgres is deployed.
Where the data lives
Section titled “Where the data lives”| Item | Value |
|---|---|
| Container | a10e-bundle-postgres (restart policy unless-stopped; comes back with Docker after a reboot) |
| Docker named volume | a10e-bundle-pgdata, mounted at /var/lib/postgresql/data inside the container |
| Default path on the host | /var/lib/docker/volumes/a10e-bundle-pgdata/_data (or under your Docker data root if you changed data-root in /etc/docker/daemon.json) |
| Database / role | a10e_db / a10e |
| Password | generated on first start and stored in /opt/a10e/shared/env-manager/postgres-password (root, 0600); never regenerated |
| Port | 5433 on 127.0.0.1 only (container port 5432) |
The volume and the password file are deliberately outside every release directory, so a10e upgrade and a10e rollback never touch them. Both must be kept together: a copy of the volume without the matching password file (or vice versa) isn’t restorable without manual intervention.
Confirm where the volume is on your host:
sudo docker volume inspect a10e-bundle-pgdata --format '{{.Mountpoint}}'sudo du -sh "$(sudo docker volume inspect a10e-bundle-pgdata --format '{{.Mountpoint}}')"Do not copy the _data directory while PostgreSQL is running and call it a backup — files change underneath you and the copy may not start. Use one of the methods below.
Recommended: nightly logical backup with pg_dump
Section titled “Recommended: nightly logical backup with pg_dump”A logical dump is portable across PostgreSQL versions, restores cleanly into the bundled container or an external server, and doesn’t require stopping the Connector. Run it inside the container so no client tools are needed on the host.
On a host that runs Podman, use podman in place of docker in every command on this page; the container is named a10e-bundle-postgres either way, for example sudo podman exec a10e-bundle-postgres pg_isready -U a10e -d a10e_db.
sudo install -d -m 0700 /var/backups/a10e
sudo tee /usr/local/bin/a10e-backup >/dev/null <<'EOF2'#!/bin/sh# a10e-backup — logical dump of the Connector database plus its credentialset -euDEST=/var/backups/a10eKEEP_DAYS=14STAMP=$(date +%Y%m%d-%H%M%S)umask 077
# Database (custom format: compressed, supports selective/parallel restore)docker exec a10e-bundle-postgres \ pg_dump -U a10e -d a10e_db --format=custom --no-owner \ > "$DEST/a10e_db-$STAMP.dump"
# The credential the volume depends on, and the operator settingscp /opt/a10e/shared/env-manager/postgres-password "$DEST/postgres-password-$STAMP"cp /opt/a10e/shared/.env "$DEST/env-$STAMP"[ -f /etc/a10e/config.toml ] && cp /etc/a10e/config.toml "$DEST/config.toml-$STAMP"
# Prunefind "$DEST" -type f -mtime +"$KEEP_DAYS" -deleteecho "backup written: $DEST/a10e_db-$STAMP.dump"EOF2sudo chmod 0755 /usr/local/bin/a10e-backup
# Run once to confirmsudo a10e-backupsudo ls -l /var/backups/a10eSchedule it nightly with a systemd timer (a cron entry works just as well):
sudo tee /etc/systemd/system/a10e-backup.service >/dev/null <<'EOF2'[Unit]Description=Alien Giraffe Connector database backupAfter=a10e-bundle.service
[Service]Type=oneshotExecStart=/usr/local/bin/a10e-backupStandardOutput=journalStandardError=journalEOF2
sudo tee /etc/systemd/system/a10e-backup.timer >/dev/null <<'EOF2'[Unit]Description=Nightly Connector database backup
[Timer]OnCalendar=*-*-* 02:30:00Persistent=trueRandomizedDelaySec=15min
[Install]WantedBy=timers.targetEOF2
sudo systemctl daemon-reloadsudo systemctl enable --now a10e-backup.timersystemctl list-timers a10e-backup.timerThen ship /var/backups/a10e off the host with whatever you already use — rsync to a backup server, aws s3 sync to a bucket you own, your enterprise backup agent — and monitor that the timer runs (journalctl -u a10e-backup.service). The dumps contain sensitive data and the env-* files contain JWT_SECRET; encrypt them at rest wherever they land.
If the host has unattended upgrades on (a10e-update.timer), run the backup before the upgrade window — the timer above at 02:30 with an upgrade window at, say, 04:30 gives you a fresh dump to fall back on if an upgrade’s schema change makes rollback unsafe (see the rollback note in Standalone Bundle Install).
Restoring a logical backup
Section titled “Restoring a logical backup”On a freshly installed host, up to and including the point where the Connector is running:
sudo systemctl stop a10e-bundle.service # stop the API so nothing writes during restore
# Recreate the database empty, then restore into itsudo docker exec a10e-bundle-postgres psql -U a10e -d postgres -c 'DROP DATABASE IF EXISTS a10e_db;'sudo docker exec a10e-bundle-postgres psql -U a10e -d postgres -c 'CREATE DATABASE a10e_db OWNER a10e;'sudo docker exec -i a10e-bundle-postgres \ pg_restore -U a10e -d a10e_db --no-owner --role=a10e < /var/backups/a10e/a10e_db-<STAMP>.dump
sudo systemctl start a10e-bundle.servicesudo a10e-healthcheckBecause the dump is logical, the new host’s own postgres-password file is used — you don’t need the old one for this path. Restore the saved env-<STAMP> to /opt/a10e/shared/.env (mode 0600) before starting the service so JWT_SECRET matches and existing sessions/tokens remain valid. If the old host used the config overlay (see Persistent Platform Configuration), also restore the saved config.toml-<STAMP> to /etc/a10e/config.toml (mode 0600) first, so any port or platform setting it carried applies before the service starts.
Alternative: volume-level snapshot
Section titled “Alternative: volume-level snapshot”If you prefer a physical copy — for example, to move the whole host or to snapshot at the block layer — stop the database first so the files are consistent:
sudo systemctl stop a10e-bundle.servicesudo docker stop a10e-bundle-postgres
sudo tar -C "$(sudo docker volume inspect a10e-bundle-pgdata --format '{{.Mountpoint}}')" \ -czf /var/backups/a10e/pgdata-$(date +%Y%m%d-%H%M%S).tar.gz .sudo cp /opt/a10e/shared/env-manager/postgres-password /var/backups/a10e/postgres-password-$(date +%Y%m%d-%H%M%S)
sudo docker start a10e-bundle-postgressudo systemctl start a10e-bundle.serviceA physical copy is only restorable into the same major PostgreSQL version and must be paired with the postgres-password file taken at the same time. To restore: stop both services as above, empty the volume’s mount point, extract the archive into it, put the saved password file back at /opt/a10e/shared/env-manager/postgres-password (root, 0600), then start the container and the service. For most deployments the pg_dump route above is simpler and safer.
If you use your own PostgreSQL
Section titled “If you use your own PostgreSQL”When A10E_DB_DSN is set, the bundled container and volume are not created and none of the above applies — the Connector’s state lives in the database you pointed it at, and your existing backup regime for that server covers it. Still back up /opt/a10e/shared/.env (or your deployment’s equivalent), which holds JWT_SECRET and the DSN itself.
Next steps
Section titled “Next steps”-
Catalog scan triage — run the installed release scanner with private diagnostics, first without AI, then with optional AI.
-
Remote Access — reach the dashboard and API over SSH if 3000/8080 aren’t exposed
-
Secret Isolation — how
JWT_SECRETand datasource credentials are kept apart