Skip to content

Configure SSH Datasources

SSH datasources authenticate with a private key rather than a username/password pair, and java-keystore can’t hold one — see Configure Data Sources and the java-keystore section of Vaults for why. That leaves file and aws-secrets-manager as the two usable backends. This guide covers file, since it lets Alien Giraffe read the same key file you already manage for SSH access directly, without re-encoding it into a secrets manager first. For aws-secrets-manager, see Vaults — the mechanics are the same as any other datasource type, with the key’s contents as the secret value.

Registering the credentials for an SSH datasource (see Registering A Datasource) needs:

  • Username (required) — the account to authenticate as on the remote host.
  • Private Key (required) — the SSH private key used to authenticate. This is the field java-keystore can’t hold.
  • Passphrase (optional) — only needed if the private key itself is encrypted.

These show up as Key Mappings once Secret Backend is set to file, same as any other type — see Credentials.

The file backend’s JSON (see file in Vaults) normally holds each secret as an inline string value, which works fine for a username. A private key is different: it’s multi-line PEM data, awkward to flatten into a single JSON string, and you likely already have it sitting in its own file on disk.

For exactly this case, a Key Mapping whose JSON entry name ends in _file changes what the JSON value means: instead of the secret itself, it’s a path, and the actual value is read from that path when Alien Giraffe connects. Set the Private Key mapping to the entry name private_key_file (instead of its default, private_key), and that JSON entry becomes a pointer to the key file rather than the key material itself.

This applies only to Private Key and Passphrase (passphrase_file). Other fields, such as the username, are always read inline.

The path must point at the private key (for example id_ed25519), never the public half (id_ed25519.pub), and the file must be readable by the account the platform runs as.

If you’d rather not keep a separate key file, put the key inline instead: map Private Key to an entry named private_key and paste the full PEM text as its value, with each line break written as \n. For example, jq -Rs '{username: "svc_reports", private_key: .}' id_ed25519 > reporting-host.json builds a correct file from an existing key.

Lay out the private key and the secrets JSON as separate files, both owned by and readable only by the account connector runs as. On a bundle install that is root; if you run connector under its own service account, use that account instead (see file in Vaults):

sudo install -d -m 0700 -o root -g root /secrets/ssh
sudo install -m 0600 -o root -g root /path/to/id_ed25519 /secrets/ssh/reporting-host_id_ed25519
sudo install -m 0600 -o root -g root /dev/null /secrets/ssh/reporting-host.json
sudo tee /secrets/ssh/reporting-host.json >/dev/null <<'EOF'
{
"username": "svc_reports",
"private_key_file": "/secrets/ssh/reporting-host_id_ed25519"
}
EOF

If the key is passphrase-protected, add a passphrase entry — inline is fine since it’s short, or use passphrase_file the same way if you’d rather keep it out of the JSON too:

{
"username": "svc_reports",
"private_key_file": "/secrets/ssh/reporting-host_id_ed25519",
"passphrase": "correct horse battery staple"
}

Then, on the wizard’s Connect step (see Registering A Datasource), next to Host and Port:

  • Secret Backend: file
  • File Path: /secrets/ssh/reporting-host.json — this is the outer JSON file, not the key itself.
  • Key Mappings: Username → username, Private Key → private_key_file (renamed to match the JSON entry above), and Passphrase → passphrase if used.

Click Test connection. Once it succeeds, the button becomes Next, which takes you to the Files step.

If the SSH server runs on the install host itself, it must listen on an address the request containers can reach, not only on 127.0.0.1; see Where Connections Come From.

An SSH datasource has five wizard steps: Type, Connect, Files, Columns and Review.

On Files, enter Base Path and Target (see Base Path And Target below for how they match) and click Check; when you open the step on a datasource that already has a Target, the check runs by itself. The step lists the files Target matches in a folder tree. Double-click a file to preview its first rows.

The files a Target matches don’t have to share one layout. When they don’t, for example two CSV files with different columns plus a JSONL export, Files lists each schema it found, with its number of files, its columns and an example file. A schema is a file format together with its layout and its set of column names: two files with the same columns in a different order have the same schema. The datasource loads every checked schema, each as its own table with its own columns. The checked schemas are the schemas the datasource accepts: later scans load only files that fit one of them, and set the others aside (see Skipped Files). At least one schema must stay checked. In a new datasource, unchecking a schema leaves its files out and narrows Target to the checked schemas, then checks it again. In a datasource that already has saved schemas, unchecking a schema doesn’t change Target: the schema is simply no longer accepted, and its files then show as skipped. Such a datasource also lists the schemas that earlier scans found in its skipped files, marked from skipped files with their number of files, even when the Files check doesn’t return them; an accepted schema the check doesn’t return is still listed, checked, titled with its table name and marked 1 file at the last scan or <n> files at the last scan (the check reads only the newest file of each stream, so a schema whose files are older ones isn’t in it), or no files now when the last scan found none, or not in this check when the catalog couldn’t be read; you can uncheck it to remove it. Such a schema has its own entry on Columns, where you can edit its Table name: it has no sample, and the step reads No sample: this check did not read this schema's files. Columns as recorded: followed by the columns. Review shows it as, for example, AUDIT_JSON (1 file at the last scan). The list is shown even when only one schema isn’t accepted yet. A schema whose files can’t be read is marked and can’t be checked; Uncheck the schema that can’t be loaded (or Uncheck the 2 schemas that can’t be loaded) removes those in one click. In a new datasource, a schema that can’t be loaded blocks the Files step until you uncheck it, with a message such as 5 files matched, with 2 schemas: 1 can't be loaded. Uncheck it below. In a datasource that already has saved schemas, files that can’t be loaded don’t block saving: the Files step warns, for example 5 files matched · 2 schemas: 1 can't be loaded, so scans skip its files. After you create the datasource, the wizard offers to create another datasource from any schema you left out.

When a Target pattern contains a date token such as {YYYY-MM-DD} (see Dated Files), Files also shows the day read from each file’s name, as a tag next to its size, and a summary such as Dates read from 42 of 42 matched files · 2026-10-01 → 2026-11-11. A file whose name has the right shape but no real date (for example app.2026-13-45.log.gz) isn’t matched by Target at all, so a scan never lists it, not even under Skipped files; the summary counts it, for example 1 file has a name that isn't a real date (skipped). Schema suggestions put the date token, written in the format the file names use, in place of a run of digits that reads as a date, and each schema shows the span of days its files cover.

The help text under Target shows the form of a date token: Mark the date in file names with its format, e.g. staging/**/*.{YYYY-MM-DD}.log.gz or exports/events-{YYYYMMDD}.jsonl.gz.

Format sets how plain-text logs (.log files) split into columns:

  • Auto (default) detects known log formats. Any other text loads as one line column.
  • Manual splits each line by the Split By delimiter (bracketed fields, tab, spaces or tabs, comma, or pipe). Max Columns caps the number of columns, with the rest of the line kept in the last one. Tick First row is a header when the first line holds the column names. CSV, JSON and other structured files keep their own columns under either setting.

On Columns, the left-hand list shows each checked schema. Click a schema to load a sample of its rows. Give each schema a Table name: it’s the name templates, requests and the Data Explorer use for the schema’s table. The field starts with a suggestion taken from the schema’s Target pattern, such as APP_LOGS for staging/*/app-logs-*/*.{YYYY-MM-DD}.log.gz, made unique among the schemas. Request templates refer to tables by this name, so the names you give here must be the names your templates use (see Table names and request templates); the suggestion is only a starting point, and a template that names a table the datasource doesn’t have can’t be used for a request. A table name starts with a letter or _, holds only letters, digits and _, is at most 63 characters long, and can’t repeat another schema’s name, whatever the capitalisation. It also can’t be a word SQL reserves, such as ORDER, SELECT, TABLE or GROUP, in any capitalisation: a query could only use such a table in double quotes. A suggestion that would be a reserved word gets _FILES added (ORDER_FILES). The name is kept as you type it (App_Logs stays App_Logs in the catalog, requests and the Data Explorer), and queries can write it in any capitalisation. A schema taken from the skipped files has no suggested name: type its Table name, and its columns show as recorded, without a sample. Next stays disabled while a name is invalid or repeated, and the field says why; when two schemas share a name, both are marked. Next also stays disabled until every schema in the list has been opened and its sample loaded: a schema marked · hasn’t been sampled yet, ! needs attention, and ✓ is ready. To rename a column, edit its name in the sample’s header. A rename applies to that column name in every schema of the datasource.

Review summarizes each step, with an Edit link back to it, and lists the accepted schemas as tables, with their table names. It also holds Scan policy and More settings. Click Create datasource to save it. The datasource is cataloged on the next scheduled scan.

You can rename a table later by editing its Table name and saving the datasource. The datasource remembers the earlier names: requests and request templates that use an earlier name keep working and load the schema under its current name, and the Data Explorer and new requests show the current name. This works as soon as you save; the next scan only updates the catalog entry’s name. On Review, a renamed table shows as APP_LOGS_2 (was APP_LOGS).

You can give a table the earlier name of another table. The field then shows a note, "APP_LOGS" was the earlier name of APP_LOGS_2. Templates that name it will use this schema from now on., and templates that use that name load the schema that has it now.

If you change what a schema is, for example when you accept its files’ new columns under the same name, requests for it are refused until the next scan with schema "<name>" of datasource "<datasource>" changed since its last scan; it can be requested after the next scan. If you change its columns and its name in the same save, it becomes a new schema and its earlier names aren’t kept.

Base Path is the folder on the remote host that the scan starts from. Target says which files under it the datasource covers. Set Base Path to an absolute path, such as /logs. If you leave it empty or start it with ~/, it resolves to the home directory of the user in the SSH credentials.

The Target is a file pattern relative to Base Path, matched one folder level at a time:

  • * matches any part of a single file or folder name and never crosses a /. *.gz matches a.gz but not host1/a.gz.
  • **, used as a whole folder level, matches any number of folders, including none. logs/**/*.gz matches logs/a.gz and logs/host1/api/a.gz.
  • ? matches one character, and [abc] matches one of the listed characters.
  • Matching is case-sensitive.
  • Several patterns go in the same Target, separated by commas. A file that matches more than one pattern belongs to the first one it matches.
  • A date token such as {YYYY-MM-DD} matches the date in a file name; see Dated Files.

For logs laid out as /logs/staging/<host>/<component>/<hourly file>.gz, with Base Path /logs:

Target Matches
staging/*/*.gz Nothing — files directly inside staging/<host>/ only
staging/*/*/*.gz Every .gz file in every host and component folder
staging/**/*.gz Every .gz file at any depth under staging
staging/app*/api/*.gz Only the api folder on hosts whose name starts with app
staging/app*/api/*.gz,staging/gw*/access/*.gz Both sets of folders

Prefer a Target that names the folders you need over a broad one: each scan lists every folder the patterns reach, and a request can load every file a pattern matches.

A scan fails, and the error names the path, when Base Path doesn’t exist or when the user in the SSH credentials can’t read a folder on the way to the files. If a scan finishes but logs no files matched target patterns, the paths are readable but the Target doesn’t match the layout on disk. Check the number of /-separated levels, or use **.

Logs and exports are often written as one file per day, with the day in the file name. Put a date token where that date appears, inside one folder or file name of the pattern. The token is the date’s format between braces, so it says how the file names write the date:

Token Matches
{YYYY-MM-DD} 2026-11-02
{YYYYMMDD} 20261102
{YYYY_MM_DD} 2026_11_02
{DD-MM-YYYY} 02-11-2026
{DD.MM.YYYY} 02.11.2026

A format holds YYYY, MM and DD once each, in any order, separated by -, _, . or nothing.

Target Matches
staging/*/app-logs-*/*.{YYYY-MM-DD}.log.gz staging/web1/app-logs-1/app.2026-11-02.log.gz, one file per host and day
archive/{YYYY-MM-DD}/app.log.gz archive/2026-11-02/app.log.gz, one folder per day
exports/events-{YYYYMMDD}.jsonl.gz exports/events-20261102.jsonl.gz, one file per day

A token matches exactly the digits and separators its format describes, within one folder or file name, and only when they are a real calendar date. A pattern holds one token. Each pattern of a Target carries its own token, so one Target can hold patterns whose files write the date differently:

staging/*/app-logs-*/*.{YYYY-MM-DD}.log.gz,exports/events-{YYYYMMDD}.jsonl.gz

Anything else between braces is rejected when you save the datasource, and the message shows the form to use, such as {YYYY-MM-DD}. A pattern without a token still works: its files have no day and are always loaded whole.

When one folder holds two kinds of log, for example server.2026-11-02.log.gz and access_log.2026-11-02.log.gz, name each kind’s file prefix in its own pattern instead of one broad *.{YYYY-MM-DD}.log.gz:

staging/*/app-logs-*/server.{YYYY-MM-DD}.log.gz,staging/*/app-logs-*/access_log.{YYYY-MM-DD}.log.gz
  • A file that matches more than one pattern belongs to the first one it matches, in the order of Target. A broad pattern listed first takes the files of the narrower ones after it.
  • Which table a file loads into doesn’t depend on the pattern: it depends on the file’s schema (format, layout and column names). Two kinds with different layouts become two tables, each with the Table name you give it. Two kinds with the same layout load into one table, whichever patterns matched them, and source_path tells their rows apart (for example WHERE source_path LIKE '%/access_log.%').
  • To leave a kind out altogether, leave its prefix out of Target. To get a separate table for each of two kinds that share a layout, create a second datasource with the other kind’s Target.

Each schema you accept is one table, and one catalog entry, named by its Table name. The entry covers every file that fits the schema, across all the patterns of Target, so the application logs in staging/*/app-logs-*/*.{YYYY-MM-DD}.log.gz are a single entry, APP_LOGS, however many hosts and days they hold. Under Admin → Data Sources Config, in Catalog Targets, a schema entry shows its name, its number of files, the first and last day and the total size; its Files button lists the files grouped by day (the latest 31 days, then +N earlier days). A schema that has no files at the moment keeps its entry, with an empty file list.

A file fits a schema only when its format, its layout and its set of column names are all the same as the schema’s. The order of the columns doesn’t matter, and neither do column renames. Saving a datasource doesn’t start a scan: the files join their tables at the next scan. Each scan:

  1. Lists the files on the host without downloading them, and records each file’s path, day and size.
  2. Reads the start of every new or changed file (the whole file, for formats that need it) to find its schema. A file that hasn’t changed since the last scan keeps the schema found before, so accepting another schema sorts files again without downloading them again; a file whose schema you stop accepting has its start read again.
  3. Sorts every file into the schema it fits, and samples only the newest file of each schema.
  4. Removes the catalog entries the scan wrote whose name matches none of the accepted schemas. Entries you added by hand, and entries whose scan policy is Disabled, are kept.

The scan log has one line per scan with the number of skipped files for each reason.

A file that Target matches but that fits none of the accepted schemas is skipped. A skipped file is never loaded, in any request, and appears in no file list. A matched file with no content is ignored: it is neither loaded nor listed here. The scan records why, and the datasource shows the reason:

Reason shown Meaning
Schema not accepted The file was read and its schema is none of the accepted ones, for example a log with another layout, or a CSV with an extra column.
Could not be read The file’s format isn’t supported, the file is corrupt, or it couldn’t be downloaded.
Too large to check The file’s format needs the whole file read to find its schema, and the file is larger than the scan reads. If other files in the same folder, named the same apart from their dates or numbers, were read and fit an accepted schema, the file is loaded with the schema of the newest of them and recorded as not checked. Otherwise it is skipped with this reason.

On the datasource list under Admin → Data Sources Config, a badge shows how many files are skipped, such as 3 files skipped (1 file skipped for one). Selecting the datasource shows a Skipped files section that lists them grouped by the schema the scan found in them, each with its reason. A group of files that were read has an Accept this schema button. It opens the wizard on Files, where that schema is listed unchecked: check it, give it a Table name on Columns, and save the datasource. The next scan sorts the files again without downloading unchanged ones, and the files that fit the schema join its file list.

A skipped file never joins the table of another schema, even when its path matches the same Target pattern as that table’s files. Each accepted schema is its own table, so accepting the skipped file’s schema gives its files a new table of their own. A request or template that names only the existing table, such as APP_LOGS, still leaves those files out: add the new table to the request, or to the template, to load them. To have the files in the existing table instead, they must be written in that table’s layout at the source; after that, the next scan loads them.

Requesters and approvers are told about skipped files too: when a request’s dates include files that were left out because their schema isn’t accepted or they are too large to check, the request carries a warning; files that could not be read are shown only here, on the datasource page (see Files That Aren’t Loaded).

A request loads each schema as one table, from the files in the dates the requester picked (see Create A Data Access Request From A Template). The table is named after the schema, with the Table name you gave it, written in capitals in the Data Explorer:

Table name Files it loads
APP_LOGS Every file that fits the schema, such as those matched by staging/*/app-logs-*/*.{YYYY-MM-DD}.log.gz
ACCESS_LOGS Every file that fits the schema, such as those matched by staging/*/access-logs-*/access.{YYYY-MM-DD}.log.gz

Request templates name the schema, not the pattern (see Request Template TOML). The dashboard offers an SSH datasource only as its schemas, so you can’t pick a single file of it on its own. To load one particular file, give a datasource a Target that names just that file and accept its schema. A request or template that names an SSH datasource with kind = "path" is refused with datasource "<name>" loads files through its schemas: use kind "path_list" with a schema name.

Only when two datasources in one request would have a table of the same name does the datasource’s name go in front of the table names, separated by a dot, for every table of both datasources: tables APP_LOGS in datasources log-host and log-host-2 become log_host.APP_LOGS and log_host_2.APP_LOGS (characters other than letters and digits in the datasource’s name become _). Within one datasource the names are always different. The Data Explorer’s Available Datasets panel lists the exact names.

Every table gets two extra columns, unless the files already have a column of that name:

  • source_path — the file the row came from.
  • file_date — that file’s day, empty (NULL) when the file’s pattern has no date token.

Files are downloaded one at a time over one SSH connection and parsed with the same formats, column renames and Manual format settings as the scan. Empty files are skipped, and a column that some files lack is empty in their rows. If a file is missing or can’t be read when the request is activated, that table fails with the file’s name; the request’s other tables still load.

The list of files is fixed when the request is created: it is exactly what the wizard showed. A file that appears on the host later, such as the next day’s log, needs a new request. This differs from database tables, which are read when the request is activated.

Upgrade the API and the env-enzo image together: an env-enzo image from an older release can’t load a request that lists dated files. A bundle upgrade ships both. If you set the env-enzo image yourself ([enzo] image in a10e.toml, see Installation), move it to the matching release. The upgrade’s database migrations run when the service starts.

Updating An SSH Datasource After An Upgrade

Section titled “Updating An SSH Datasource After An Upgrade”

An SSH datasource saved before schemas were accepted per datasource shows an Update required badge on the datasource list, with a Review files button in place of Edit. Selecting the datasource shows this banner, also with a Review files button:

This datasource was saved before schemas were accepted per datasource. Review its files and save it to scan it again.

Until an administrator updates it:

  • The datasource isn’t scanned. At the next scan the catalog entries earlier scans wrote for it, and its skipped-files list, are removed; until then they can still show. Entries you added by hand are kept. After that, Catalog Targets and Create Request offer nothing from it.
  • In Create Request, the datasource shows Needs an update by an administrator and can’t be requested. A template that includes it shows a notice that names the datasource, and a request can’t be created from the template until it is updated.
  • Creating a request that names the datasource through the API is refused with datasource "<datasource>" must be updated by an administrator before it can be requested.
  • Requests created earlier keep their file lists and their data.

To update the datasource:

  1. Under Admin → Data Sources Config, open the datasource and click Review files. The wizard opens on Files. If the old Target had {date} markers, it is already filled in with each marker written as a date token in the date format the datasource used before, such as {YYYY-MM-DD}. If the old Target had no date markers, which is the common case for a datasource created on an earlier release, it opens unchanged: see When the old Target had no date markers.
  2. Wait for the check to finish (or click Check). Uncheck the schemas you don’t want the datasource to accept.
  3. On Columns, give each schema a Table name. Use the names your request templates use (see Table names and request templates), not only the suggestions.
  4. On Review, save the datasource. The next scan picks it up.

The first scan after the upgrade (the service runs one when it starts) removes the datasource’s per-file catalog entries, and the datasource shows Update required. Nothing from it can be requested until an administrator opens it, puts a date token in the Target where the date is in the file names, reviews the files, names the tables and saves. The wizard doesn’t add the token for you.

For logs laid out as staging/<host>/app-logs-<n>/app.2026-11-02.log.gz and staging/<host>/access-logs-<n>/access.2026-11-02.log.gz:

Target Tables
Before the upgrade staging/*/*/*.gz One catalog entry per file
Wizard opens with staging/*/*/*.gz (unchanged)
After you edit it staging/*/app-logs-*/app.{YYYY-MM-DD}.log.gz,staging/*/access-logs-*/access.{YYYY-MM-DD}.log.gz APP_LOGS and ACCESS_LOGS, named as your templates name them

Click Check after changing Target, confirm that the summary reads the dates from all the matched files, and continue to Columns. If one folder holds two kinds of log, see Keeping Two Kinds Of Log Apart.

Request templates written before the upgrade name a file pattern. Until they name schemas, Create Request shows a notice for each dataset the datasource doesn’t have and a request can’t be created from that template. Import them again, naming the schemas instead (see Request Template TOML).

Replace whichever file changed — the key itself, or the JSON if the username or path changes — in place. As with any file-backed secret, there’s no watch/reload beyond what the datasource connection itself triggers (see file in Vaults).