Scan Your Catalog
After registering a datasource, scan it to discover schema, sample and de-identify data, and populate the catalog. AI summaries are optional. Start with the complete non-AI procedure below; add AI only after that baseline works.
Part 1: manual scan without AI
Section titled “Part 1: manual scan without AI”Identify the release and execution identity
Section titled “Identify the release and execution identity”The release builder installs bin/env-manager, normally /opt/a10e/current/bin/env-manager. This is a release executable, separate from the a10e management CLI. For a standalone bundle, substitute its actual directory. Record the resolved release path and version. For the reviewed command parser:
/opt/a10e/current/bin/env-manager -version/opt/a10e/current/bin/env-manager catalog scan -hThe scan accepts -config, -sample-size (default 100), and -dry-run. Releases built from aliengiraffe/env-manager#976 list -config in the help; earlier releases omit it there, although the parser accepts it. Put flags after the command (catalog scan -config …), as every command on this page does: on earlier releases a flag placed first (env-manager -config … catalog scan) starts the API server instead of scanning, and releases with #976 exit with flags must come after the command and print the corrected command. There are no scan verbosity/debug, datasource/organization selector, force-refresh, or timeout flags. Diagnostics use LOG_LEVEL=debug; the command creates a fixed ten-minute context for the whole sweep.
Run as the service’s execution user, with equivalent network access, secret-file permissions, and cloud identity. On systemd hosts, inspect only the relevant non-secret properties:
systemctl show a10e-bundle.service -p User -p Group -p WorkingDirectoryUse your actual unit name. An empty system service User means root. An interactive shell does not inherit the service environment; sudo can also remove variables or change cloud credential lookup. Have the operator open an authorized shell as the correct user before continuing. Keep the service’s working directory when relative paths in datasource secret references depend on it; use absolute paths for the executable, config, keystore, and credential files.
Prepare a private triage session
Section titled “Prepare a private triage session”Use Bash for all the following shell blocks, in the same session. This creates private scratch files without changing the release, /etc/a10e, or a running service:
bashset +x # never trace secret assignmentsumask 077release=$(cd /opt/a10e/current && pwd -P)bin="$release/bin/env-manager"triage=$(mktemp -d "${TMPDIR:-/tmp}/a10e-catalog.XXXXXX")cd "$release" # substitute the service WorkingDirectory if differentcp "$release/a10e.toml" "$triage/base.toml"chmod 600 "$triage/base.toml"catalog scan loads platform configuration directly. It does not load release .env/.env.defaults, provision PostgreSQL, or recover the bundled DSN simply because it runs from the release directory. Its precedence is built-in defaults → base TOML → selected overlay → nonempty process environment. See Persistent Platform Configuration for the full configuration model.
Do not source dotenv files: bundle paths use dotenv/literal-file parsers with different shell semantics, and sourcing can execute their contents. Instead, supply the required values explicitly below using Bash’s hidden read -r -s; input is not echoed or stored as a shell command in history. Retrieve the existing values through your approved secret store or private operator configuration editor. Do not paste secrets into command lines or print the environment. These prompts do not import dotenv quotes or expand variables: enter the effective value, without surrounding file-format quotes.
Supply the application database
Section titled “Supply the application database”The application PostgreSQL database holds organizations, datasource registrations, and catalog entries. It is separate from each datasource the scanner connects to and samples. Choose one of these two setups.
External PostgreSQL: reuse the service’s complete existing DSN, including TLS options. If it is already in the copied base/overlay TOML, you may retain that setting and unset A10E_DB_DSN to prevent an unrelated inherited override. Otherwise:
read -r -s -p 'Existing application PostgreSQL DSN: ' A10E_DB_DSNprintf '\n'export A10E_DB_DSNBundled PostgreSQL: use the already-running container and its existing password. State resolution in the reviewed release is:
A10E_BUNDLE_STATE_DIR/postgres-password, if the service specifies a state override./opt/a10e/shared/env-manager/postgres-passwordfor the normal CLI-managed layout.<release>/data/postgres-passwordfor standalone/legacy bundles. Bundle startup may already have copied that legacy file into shared state; use the active shared file when present.
Do not generate a password, recreate the container, or run bundle startup for triage. Verify the published loopback port without dumping container environment variables:
docker port a10e-bundle-postgres 5432/tcp # or: podman port a10e-bundle-postgres 5432/tcpFor the reviewed bundle’s a10e user and a10e_db database, populate the DSN without printing the password. Set pg_port to the port reported above and pg_password_file to the service’s active state path:
pg_port=5432pg_password_file=/opt/a10e/shared/env-manager/postgres-password# Python 3 is used only to read the file and URL-encode the existing password.A10E_DB_DSN=$(python3 - "$pg_password_file" "$pg_port" <<'PY'import pathlib, sys, urllib.parsepassword = pathlib.Path(sys.argv[1]).read_text().strip()if not password: raise SystemExit("Existing bundled PostgreSQL password is empty; stop triage")port = int(sys.argv[2])print(f"postgres://a10e:{urllib.parse.quote(password, safe='')}@127.0.0.1:{port}/a10e_db?sslmode=disable")PY) || exit 1export A10E_DB_DSNThe sslmode=disable here matches the bundled loopback connection; retain the configured TLS policy for an external database. If the expected credential file is missing, stop and reconcile the installation/state path rather than inventing a replacement credential.
Supply the existing secrets and de-identification settings
Section titled “Supply the existing secrets and de-identification settings”Always reuse the service’s exact de-identification key:
read -r -s -p 'Existing A10E_DEIDENTIFY_SECRET_KEY value: ' A10E_DEIDENTIFY_SECRET_KEYprintf '\n'export A10E_DEIDENTIFY_SECRET_KEY: "${A10E_DEIDENTIFY_SECRET_KEY:?The existing service key is required}"The key is any non-empty string, used exactly as supplied. An empty value fails startup. The scanner doesn’t decode base64 or check the length, so a wrong value still starts the service. The openssl rand -hex 32 key above works. Releases before the one that corrected the scan help text describe the key as “base64-encoded” there; it has always been used as-is. Do not decode, re-encode, or generate a replacement key for triage; changing the string changes deterministic de-identification.
Match the backend in each datasource’s stored secret references (see Vaults and Configure Data Sources):
| Backend | Interactive process prerequisites |
|---|---|
| Java keystore | Existing absolute A10E_KEYSTORE_PATH and existing A10E_KEYSTORE_PASSWORD; readable file, correct format/password, and referenced aliases. |
| File | Access to the existing referenced files as the service user; preserve the working directory for relative references. |
| AWS Secrets Manager | Same AWS identity/profile/role and AWS_REGION as the service, with permission to retrieve the referenced secrets and decrypt them where required. |
For Java keystore, for example:
read -r -p 'Existing absolute keystore path: ' A10E_KEYSTORE_PATHread -r -s -p 'Existing keystore password: ' A10E_KEYSTORE_PASSWORDprintf '\n'export A10E_KEYSTORE_PATH A10E_KEYSTORE_PASSWORDThe scanner’s secret manager reads these settings from the environment, not the platform TOML. Preserve the service’s relevant AWS settings/credential-file access explicitly too; do not dump credentials or assume a login shell has the same role. Stored secret-reference types select the backend; A10E_SECRETS_BACKEND is not a switch that rewrites existing references. A configured but broken keystore can fail scanner initialization even when the datasource under investigation uses another backend.
Also preserve environment-only platform requirements from the service. In particular, when the effective tunnel provider is cloudflare, the configuration loader requires the existing A10E_CLOUDFLARE_PROXY_HOSTNAME even for this command. Supply it explicitly if absent; a scan does not start a tunnel.
Ensure AI is actually disabled
Section titled “Ensure AI is actually disabled”An unset or empty AI_PROVIDER does not clear a nonempty TOML provider: empty environment values are ignored. The effective [ai] provider must be "". The scanner has no --no-ai switch and does not use AI_ENABLED=false or [ai] enabled=false to disable summaries.
Make a private copy of the actual selected service overlay, preserving all other keys. If the service uses A10E_CONFIG_OVERLAY_PATH, use that file. Otherwise check its A10E_CONFIG_OVERLAY_DIR (default /etc/a10e): config.toml wins over the legacy a10e.toml. For the default layout:
if [ -f /etc/a10e/config.toml ]; then cp /etc/a10e/config.toml "$triage/overlay.toml"elif [ -f /etc/a10e/a10e.toml ]; then cp /etc/a10e/a10e.toml "$triage/overlay.toml"else printf '[ai]\nprovider = ""\n' > "$triage/overlay.toml"fichmod 600 "$triage/overlay.toml"Privately edit only the scratch overlay (for example, vi "$triage/overlay.toml"). Set provider = "" inside its existing [ai] table, or add the table if absent. Keep its other settings; do not create a duplicate [ai] table. If required service settings came from .env.defaults rather than TOML, carry their effective values into this private configuration or explicit environment as well.
The resulting AI override is:
[ai]provider = ""Now select that overlay and remove any inherited process provider:
export A10E_CONFIG_OVERLAY_PATH="$triage/overlay.toml"unset AI_PROVIDEROnly this process and its children use the scratch overlay. With the reviewed loader, this overrides an AI provider in the base TOML while retaining the copied host settings. On an older release without overlay-path support, do not assume this disables AI; obtain a release-specific procedure first.
Dry-run, then capture an actual scan
Section titled “Dry-run, then capture an actual scan”Optional dry-run:
LOG_LEVEL=debug "$bin" catalog scan -config "$triage/base.toml" \ -sample-size 10 -dry-run > "$triage/dry-run.log" 2>&1dry_status=$?printf 'Dry-run exit status: %s\n' "$dry_status"Privately inspect the log. Require this startup confirmation before continuing:
AI features disabled (AI_PROVIDER not set); catalog scan will skip summariesDry-run still loads/validates platform config, connects to the application DB, initializes the scanner/secret manager, requires the de-identification key, and can initialize AI if misconfigured. It lists organizations/datasources and resolves datasource configuration, including config-secret references. It returns before datasource credential resolution and connection, target discovery, sampling, de-identification, and catalog writes. It is not a datasource connectivity test or proof that the key or AI provider works.
Datasource target configuration determines matches: for SQL, table names support single names, glob patterns such as CARD*, and comma-separated patterns such as sys_*,app_*,users. A scan of a database datasource also removes the catalog entries it wrote for tables the Target no longer matches (a table that was dropped or renamed in the source, that the login can no longer read, or that a narrowed Target leaves out), so they stop being offered in new requests; requests that are already approved or active keep what they were given. Hand-written entries, entries whose scan policy is Disabled, and an Oracle name the scan skips because it is readable in several schemas are kept. Nothing is removed when the scan finds no table at all, cannot list the tables, or the datasource’s own scan policy is Disabled. Widening the Target again brings the tables back at the next scan. S3 discovery resolves file patterns into concrete paths, one catalog entry per file. SSH discovery lists the host without downloading anything, reads the start of each new or changed file to find its schema, and creates one catalog entry per accepted schema (named by its table name), recording each file’s path, day (read from the date token in the name) and size under it; only the newest file of each schema is sampled, and only when it changed since the last scan. A file that fits no accepted schema is skipped and recorded with its reason, and is never loaded. An entry the scan wrote whose name matches no accepted schema is removed; hand-written entries and entries whose scan policy is Disabled are kept. An SSH datasource that hasn’t been updated after an upgrade is not scanned. The scan log has one line per scan with the number of skipped files for each reason. See Schemas In The Catalog and Skipped Files. A missing target or zero matches performs no target update.
Existing datasource catalog_scan_policy and catalog-row catalog_scan_policy_override metadata can select disabled, normal, or force_refresh. A row policy takes precedence. Disabled targets are skipped; normal scans skip unchanged change marks before de-identification/AI/persistence (SQL sampling occurs before that unchanged check). A stored force_refresh policy bypasses the unchanged check; there is no force flag. Do not change production-wide policies just to make a diagnostic call happen.
Capture both output streams in a restricted file. Redirecting directly avoids showing potentially sensitive errors in the terminal and preserves the scanner’s status without a pipeline:
LOG_LEVEL=debug "$bin" catalog scan -config "$triage/base.toml" \ -sample-size 10 > "$triage/scan.log" 2>&1scan_status=$?printf 'Scanner exit status: %s\n' "$scan_status"Run these blocks without Bash errexit (set -e) so a failing scan reaches the status assignment. If you instead choose live output with tee, use Bash’s PIPESTATUS immediately after the pipeline, not $? (which normally reports tee):
LOG_LEVEL=debug "$bin" catalog scan -config "$triage/base.toml" \ -sample-size 10 2>&1 | tee "$triage/scan-live.log"pipeline_status=("${PIPESTATUS[@]}")scan_status=${pipeline_status[0]}printf 'Scanner exit status: %s; tee exit status: %s\n' \ "$scan_status" "${pipeline_status[1]}"That is an alternative actual scan, not an extra required run. A failing tee means the capture may be incomplete even if the scanner succeeds. Normal logging uses production JSON; LOG_LEVEL=debug uses development formatting and enables debug messages. Sanitized examples (timestamps/caller omitted):
{"level":"info","msg":"database connection established"}INFO resolved targets {"datasource":"example","matched":1}DEBUG AI disabled; skipping summary generationINFO created catalog entry {"datasource":"example","target":"example_table","catalog_id":"<redacted>"}Find the failing stage
Section titled “Find the failing stage”Exit 0 and catalog scan completed successfully do not prove every target succeeded. ScanAll logs datasource errors and continues; target loops also log errors and continue. The scanned count can include skips and datasources with failed targets, and the skipped counter is not a reliable target count. Inspect each intended datasource/target and its persistence result.
| Symptom or log message | Stage | Next check |
|---|---|---|
Failed to load platform config |
Startup/configuration | TOML syntax, readable absolute paths, effective overlay/environment, and required Cloudflare hostname if applicable. |
A10E_DB_DSN is required, Failed to connect to database, Failed to ping database |
Startup/application DB | Existing DSN, loopback published port or external host, TLS, authentication, and execution identity. This is not yet a datasource connection failure. |
A10E_DEIDENTIFY_SECRET_KEY environment variable is required |
Startup/de-identification prerequisite | Supply the service’s exact nonempty key. The scanner doesn’t reject a wrong encoding or length, so passing startup does not validate that the key is the one earlier scans used. |
Failed to initialize secret manager / java-keystore backend unavailable |
Startup/secrets | Keystore path, password, format, permissions; distinguish this from a per-datasource lookup failure. |
catalog scan failed to list organizations / list datasources |
Application DB enumeration | DB permissions, schema/migrations, active organizations and datasource registrations. |
scan failed with resolve datasource config, parse config, or resolve credentials |
Datasource config/secrets | Stored configuration, backend type, aliases/references, file permissions, AWS identity/region and access. Config secrets can fail even during dry-run. |
scan failed with connection/authentication errors; S3 access denied/403 |
Datasource connection | That datasource’s host/port, network route, TLS, credentials, temporary token expiry, and access policy. |
scan failed with x509: cannot validate certificate for <ip> because it doesn't contain any IP SANs or x509: certificate signed by unknown authority (SQL Server) |
Datasource connection/TLS | The server’s certificate doesn’t cover the host entered or isn’t trusted; see SQL Server encryption. |
resolve targets, list s3 files, list ssh files |
Discovery | Listing permissions, SQL schema/owner, bucket/prefix or SSH base path and target patterns. |
skipping datasource (no target configured), resolved targets with matched: 0, no files matched target patterns |
Selection/no work | Correct target, schema and case-sensitive names/patterns; this is not successful sampling. |
skipping target due to scan policy, datasource unchanged, skipping |
Policy/change detection | Inspect that target’s effective policy/change mark. No subsequent AI call or catalog update is expected. |
ssh scan summary |
SSH file classification | One line per scan with the counts of files listed, read, accepted and skipped for each reason (unknown_schema, unreadable, too_large). A warning level means a schema failed and the skipped list was not replaced. |
ssh datasource needs an update; not scanned |
SSH update required | The datasource was saved before schemas were accepted per datasource; review its files and save it (see Updating An SSH Datasource After An Upgrade). |
scan target failed with sample ..., or a file-sampling error |
Sampling | Read/query permissions, file format, query/driver error, and sample size. |
scan target failed / process result failed with deidentify: |
De-identification | Check the sample shape and that the key is exactly the one earlier scans used. Do not rotate the key as a diagnostic fix. |
check existing catalog, insert catalog, update catalog, upsert catalog entry failed |
Catalog persistence | Application DB permissions, constraints/schema and availability, using the datasource/target context in the log. |
context deadline exceeded |
Stage named by the surrounding error | Fixed ten-minute sweep budget, datasource latency and number of targets; some initialization/provider operations can have their own blocking behavior. There is no scan timeout flag. |
Verify the intended result
Section titled “Verify the intended result”In the dashboard, select the intended organization, open its datasource and Catalog Targets editor, then inspect the exact target name/location. An SSH entry is named by its schema’s table name and shows its file count, date span and size; its Files button lists the files the scan recorded, grouped by day. Files the scan skipped are in no entry: they are listed under Skipped files on the datasource’s page. Compare its schema and sampled-data fields with the expected source, and correlate its catalog ID with created catalog entry or updated catalog entry in this run’s log. Merely finding an older row is not proof of an update; unchanged/policy skips require a different expectation.
Non-AI scans create new entries with an empty summary and preserve existing summaries when updating. A non-null or nonempty summary alone cannot prove AI ran: it may be empty, manually written, or left from an earlier scan. Check per-target errors and persistence logs, not just summary counts.
Keep scratch configuration and logs private. Before sharing, redact DSNs, credentials, customer identifiers, paths, sample data, and any AI output; de-identification is not a guarantee that every field is safe to share. When triage is complete, securely retain only the diagnostics you need and remove the scratch directory. Exit this dedicated shell to discard its exported settings.
Part 2: add AI and isolate AI failures
Section titled “Part 2: add AI and isolate AI failures”Keep the proven non-AI setup and private overlay. Enable AI only for this process using a supported provider, an explicit model available to it, and its credentials/endpoint. See Installation: Configure the AI provider for provider-specific setup.
| Provider | Required settings/checks beyond AI_PROVIDER and AI_MODEL |
|---|---|
ollama |
AI_OLLAMA_SERVER_ADDR (default http://localhost:11434), reachable from this host/user, with the chosen model installed; no API key required. Container-only hostnames may not work from the host shell. |
openai, anthropic, googleai |
Existing AI_API_KEY, provider/model access, outbound DNS/HTTPS, proxy and TLS trust. Supply the key with hidden input, not a command-line literal. |
vertexai |
AI_GOOGLE_CLOUD_PROJECT and AI_GOOGLE_CLOUD_LOCATION, falling back to GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION/GOOGLE_CLOUD_REGION; working Application Default Credentials for this user/workload. GOOGLE_APPLICATION_CREDENTIALS, if used, must point to the existing readable credential file. AI_API_KEY is not used. |
For an existing local Ollama installation, enter a model already installed there (the prompt does not install one), then run the same sweep with AI enabled:
read -r -p 'Existing Ollama model name: ' triage_modelread -r -p 'Existing Ollama URL (for example http://localhost:11434): ' triage_endpoint: "${triage_model:?Model is required}" "${triage_endpoint:?Endpoint is required}"AI_PROVIDER=ollama AI_MODEL="$triage_model" AI_OLLAMA_SERVER_ADDR="$triage_endpoint" \ LOG_LEVEL=debug "$bin" catalog scan -config "$triage/base.toml" \ -sample-size 10 > "$triage/scan-ai.log" 2>&1ai_status=$?printf 'AI scan exit status: %s\n' "$ai_status"The nonempty process provider overrides the scratch overlay’s empty provider only for that command. This actual scan has the same broad scope and writes as Part 1. To test generation reliably, use a new/changed target in an isolated fixture or an already-approved target policy that permits refresh. Simply enabling AI after a successful non-AI scan does not regenerate summaries on unchanged targets. Do not delete catalog rows or enable production-wide force refresh as routine troubleshooting.
| Diagnostic | Meaning and next check |
|---|---|
Failed to initialize AI pipeline: invalid config: ... |
Startup failed before target processing: check provider, explicit model, API key or Vertex project/location/ADC. |
initializing AI pipeline with provider/model |
Provider initialization was entered, not proof of a successful request. Subsequent startup errors can identify unsupported providers, plugin setup or endpoint/authentication problems. |
| Policy/unchanged/no-match messages | No summarization reached for that target. Fix the reproduction case before diagnosing provider traffic. |
scan target failed / process result failed with summarize: |
The target reached the summarization stage after sampling/de-identification. Inspect the nested error for model availability, authentication/authorization, rate/quota limits, network/TLS, prompt/generation or deadline errors. Other targets may continue and the process may still exit 0. |
| Target persisted, but summary empty or old | Correlate this run’s provider setup, absence of target errors/skips, persistence log, and actual summary content. An empty but non-null field is not successful generation evidence. |
The scanner supplies the provider with serialized schema and de-identified sample rows in its summary prompt. Schema names and unclassified data can still be sensitive. A generation error prevents that target’s catalog upsert; it does not silently fall back to storing a new non-AI result. Non-AI rescans preserve existing summaries; AI-enabled updates replace them when they actually reach persistence.
The CLI’s loadAIConfig passes provider/model/credentials/endpoint/project/location, but does not pass the server’s AI_GENERATION_TIMEOUT / [ai] generation_timeout setting. Do not use that server setting as a promised scan-timeout control. The scan’s ten-minute context is shared across organizations and targets, including summary calls that honor it; it is not ten minutes per target and does not guarantee a hard wall-clock limit for every library operation.
Next steps
Section titled “Next steps”- Create A Data Access Request From A Template — use verified catalog targets in a request.
- Operations: Health Checks & Backups — operational checks and preservation of existing configuration/state.