Skip to content

Operations: Health Checks & Backups

Once the Connector is installed — by any of the paths in Installation — treat it as a long-running service: monitor it continuously, and back up its database. This guide covers both, and applies to bundle installs, Docker Compose, and Kubernetes deployments alike, except where noted.

The Connector is (at minimum) three cooperating services: the API, the dashboard, and PostgreSQL. Each has a cheap, unambiguous probe you can wire into whatever monitoring you already run (Nagios/Icinga, Zabbix, Prometheus blackbox exporter, Datadog, Uptime Kuma, a cron job that pages you — anything that can run a command or fetch a URL).

Service What runs it Probe Healthy when
Connector API a10e-bundle.service (bundle installs) or your container orchestrator curl -fsS http://127.0.0.1:8080/v1/healthcheck HTTP 200
Dashboard Docker container a10e-bundle-dashboard (bundle installs) curl -fsS http://127.0.0.1:3000/api/health HTTP 200
PostgreSQL (bundled) Docker container a10e-bundle-postgres docker exec a10e-bundle-postgres pg_isready -U a10e -d a10e_db exit 0, prints “accepting connections”
Release integrity (bundle installs only) the a10e CLI /opt/a10e/current/bin/env-manager bundle health exit 0 — API and dashboard answer and the running unit belongs to the active release

The last row is the same check the CLI uses during install and upgrade. It’s stricter than the two curls on their own because it also confirms the process serving on those ports is the release /opt/a10e/current points at, so a stale process from a failed upgrade can’t look healthy.

A single script for all three services (bundle installs)

Section titled “A single script for all three services (bundle installs)”

Save this as /usr/local/bin/a10e-healthcheck, mode 0755. It exits 0 only when every probe passes and prints one line per service, so it works as a Nagios-style plugin and reads well in a log.

#!/bin/sh
# a10e-healthcheck — exit 0 only if every Connector service is healthy
set -u
rc=0
check() { # name, command...
name=$1; shift
if "$@" >/dev/null 2>&1; then
echo "OK $name"
else
echo "FAIL $name"; rc=1
fi
}
check "a10e.service active" systemctl is-active --quiet a10e.service
check "a10e-bundle.service active" systemctl is-active --quiet a10e-bundle.service
check "postgres container running" sh -c '[ "$(docker inspect -f "{{.State.Running}}" a10e-bundle-postgres 2>/dev/null)" = true ]'
check "postgres accepting connections" docker exec a10e-bundle-postgres pg_isready -q -U a10e -d a10e_db
check "dashboard container running" sh -c '[ "$(docker inspect -f "{{.State.Running}}" a10e-bundle-dashboard 2>/dev/null)" = true ]'
check "API /v1/healthcheck" curl -fsS --max-time 5 http://127.0.0.1:8080/v1/healthcheck
check "dashboard /api/health" curl -fsS --max-time 5 http://127.0.0.1:3000/api/health
check "release health (a10e)" sh -c 'cd /opt/a10e/current && ./bin/env-manager bundle health'
exit $rc

Run it as root (it needs Docker and the release directory):

sudo a10e-healthcheck

If you set A10E_DB_DSN to point the Connector at your own PostgreSQL, drop the two postgres rows and probe your database server instead.

systemd timer, alerting via the journal. Runs every minute; failures are visible in journalctl -u a10e-healthcheck and can be forwarded by your log shipper.

sudo tee /etc/systemd/system/a10e-healthcheck.service >/dev/null <<'EOF2'
[Unit]
Description=Alien Giraffe Connector health check
After=a10e-bundle.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/a10e-healthcheck
StandardOutput=journal
StandardError=journal
EOF2
sudo tee /etc/systemd/system/a10e-healthcheck.timer >/dev/null <<'EOF2'
[Unit]
Description=Run the Connector health check every minute
[Timer]
OnBootSec=2min
OnUnitActiveSec=1min
AccuracySec=10s
[Install]
WantedBy=timers.target
EOF2
sudo systemctl daemon-reload
sudo systemctl enable --now a10e-healthcheck.timer
journalctl -u a10e-healthcheck -n 20 --no-pager

HTTP monitors. If your monitoring tool only fetches URLs, point two HTTP checks at the host: http://<host>:8080/v1/healthcheck and http://<host>:3000/api/health, expecting status 200. PostgreSQL is bound to 127.0.0.1 only, so it can’t be probed remotely — cover it with the script above or with pg_isready run through your agent.

Prometheus. Use the blackbox exporter’s http_2xx module against the two URLs, and the node exporter’s textfile collector to publish the script’s result, for example by appending a10e_healthy 1 or a10e_healthy 0 to a .prom file from the timer.

Whichever tool you use, alert on two things: any probe failing for more than two consecutive runs, and (for bundle installs that leave unattended upgrades on) a10e-update.timer reporting a failed run in journalctl -u a10e-update.service.

The bundled PostgreSQL holds all Connector state: configuration, users, request records, audit history. Everything else on the host — release directories, images, containers — can be rebuilt from the release bundle. The database cannot. Treat it as the one thing on the host that must survive.

This section covers the bundled-PostgreSQL case (bundle installs with no A10E_DB_DSN set). Docker Compose and Kubernetes deployments manage their own Postgres persistence — see Deployment Model — but the same pg_dump approach below applies regardless of how Postgres is deployed.

Item Value
Container a10e-bundle-postgres (restart policy unless-stopped; comes back with Docker after a reboot)
Docker named volume a10e-bundle-pgdata, mounted at /var/lib/postgresql/data inside the container
Default path on the host /var/lib/docker/volumes/a10e-bundle-pgdata/_data (or under your Docker data root if you changed data-root in /etc/docker/daemon.json)
Database / role a10e_db / a10e
Password generated on first start and stored in /opt/a10e/shared/env-manager/postgres-password (root, 0600); never regenerated
Port 5433 on 127.0.0.1 only (container port 5432)

The volume and the password file are deliberately outside every release directory, so a10e upgrade and a10e rollback never touch them. Both must be kept together: a copy of the volume without the matching password file (or vice versa) isn’t restorable without manual intervention.

Confirm where the volume is on your host:

sudo docker volume inspect a10e-bundle-pgdata --format '{{.Mountpoint}}'
sudo du -sh "$(sudo docker volume inspect a10e-bundle-pgdata --format '{{.Mountpoint}}')"

Do not copy the _data directory while PostgreSQL is running and call it a backup — files change underneath you and the copy may not start. Use one of the methods below.

Section titled “Recommended: nightly logical backup with pg_dump”

A logical dump is portable across PostgreSQL versions, restores cleanly into the bundled container or an external server, and doesn’t require stopping the Connector. Run it inside the container so no client tools are needed on the host.

On a host that runs Podman, use podman in place of docker in every command on this page; the container is named a10e-bundle-postgres either way, for example sudo podman exec a10e-bundle-postgres pg_isready -U a10e -d a10e_db.

sudo install -d -m 0700 /var/backups/a10e
sudo tee /usr/local/bin/a10e-backup >/dev/null <<'EOF2'
#!/bin/sh
# a10e-backup — logical dump of the Connector database plus its credential
set -eu
DEST=/var/backups/a10e
KEEP_DAYS=14
STAMP=$(date +%Y%m%d-%H%M%S)
umask 077
# Database (custom format: compressed, supports selective/parallel restore)
docker exec a10e-bundle-postgres \
pg_dump -U a10e -d a10e_db --format=custom --no-owner \
> "$DEST/a10e_db-$STAMP.dump"
# The credential the volume depends on, and the operator settings
cp /opt/a10e/shared/env-manager/postgres-password "$DEST/postgres-password-$STAMP"
cp /opt/a10e/shared/.env "$DEST/env-$STAMP"
[ -f /etc/a10e/config.toml ] && cp /etc/a10e/config.toml "$DEST/config.toml-$STAMP"
# Prune
find "$DEST" -type f -mtime +"$KEEP_DAYS" -delete
echo "backup written: $DEST/a10e_db-$STAMP.dump"
EOF2
sudo chmod 0755 /usr/local/bin/a10e-backup
# Run once to confirm
sudo a10e-backup
sudo ls -l /var/backups/a10e

Schedule it nightly with a systemd timer (a cron entry works just as well):

sudo tee /etc/systemd/system/a10e-backup.service >/dev/null <<'EOF2'
[Unit]
Description=Alien Giraffe Connector database backup
After=a10e-bundle.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/a10e-backup
StandardOutput=journal
StandardError=journal
EOF2
sudo tee /etc/systemd/system/a10e-backup.timer >/dev/null <<'EOF2'
[Unit]
Description=Nightly Connector database backup
[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
RandomizedDelaySec=15min
[Install]
WantedBy=timers.target
EOF2
sudo systemctl daemon-reload
sudo systemctl enable --now a10e-backup.timer
systemctl list-timers a10e-backup.timer

Then ship /var/backups/a10e off the host with whatever you already use — rsync to a backup server, aws s3 sync to a bucket you own, your enterprise backup agent — and monitor that the timer runs (journalctl -u a10e-backup.service). The dumps contain sensitive data and the env-* files contain JWT_SECRET; encrypt them at rest wherever they land.

If the host has unattended upgrades on (a10e-update.timer), run the backup before the upgrade window — the timer above at 02:30 with an upgrade window at, say, 04:30 gives you a fresh dump to fall back on if an upgrade’s schema change makes rollback unsafe (see the rollback note in Standalone Bundle Install).

On a freshly installed host, up to and including the point where the Connector is running:

sudo systemctl stop a10e-bundle.service # stop the API so nothing writes during restore
# Recreate the database empty, then restore into it
sudo docker exec a10e-bundle-postgres psql -U a10e -d postgres -c 'DROP DATABASE IF EXISTS a10e_db;'
sudo docker exec a10e-bundle-postgres psql -U a10e -d postgres -c 'CREATE DATABASE a10e_db OWNER a10e;'
sudo docker exec -i a10e-bundle-postgres \
pg_restore -U a10e -d a10e_db --no-owner --role=a10e < /var/backups/a10e/a10e_db-<STAMP>.dump
sudo systemctl start a10e-bundle.service
sudo a10e-healthcheck

Because the dump is logical, the new host’s own postgres-password file is used — you don’t need the old one for this path. Restore the saved env-<STAMP> to /opt/a10e/shared/.env (mode 0600) before starting the service so JWT_SECRET matches and existing sessions/tokens remain valid. If the old host used the config overlay (see Persistent Platform Configuration), also restore the saved config.toml-<STAMP> to /etc/a10e/config.toml (mode 0600) first, so any port or platform setting it carried applies before the service starts.

If you prefer a physical copy — for example, to move the whole host or to snapshot at the block layer — stop the database first so the files are consistent:

sudo systemctl stop a10e-bundle.service
sudo docker stop a10e-bundle-postgres
sudo tar -C "$(sudo docker volume inspect a10e-bundle-pgdata --format '{{.Mountpoint}}')" \
-czf /var/backups/a10e/pgdata-$(date +%Y%m%d-%H%M%S).tar.gz .
sudo cp /opt/a10e/shared/env-manager/postgres-password /var/backups/a10e/postgres-password-$(date +%Y%m%d-%H%M%S)
sudo docker start a10e-bundle-postgres
sudo systemctl start a10e-bundle.service

A physical copy is only restorable into the same major PostgreSQL version and must be paired with the postgres-password file taken at the same time. To restore: stop both services as above, empty the volume’s mount point, extract the archive into it, put the saved password file back at /opt/a10e/shared/env-manager/postgres-password (root, 0600), then start the container and the service. For most deployments the pg_dump route above is simpler and safer.

When A10E_DB_DSN is set, the bundled container and volume are not created and none of the above applies — the Connector’s state lives in the database you pointed it at, and your existing backup regime for that server covers it. Still back up /opt/a10e/shared/.env (or your deployment’s equivalent), which holds JWT_SECRET and the DSN itself.

  • Catalog scan triage — run the installed release scanner with private diagnostics, first without AI, then with optional AI.

  • Remote Access — reach the dashboard and API over SSH if 3000/8080 aren’t exposed

  • Secret Isolation — how JWT_SECRET and datasource credentials are kept apart