Configure Data Sources
Datasources are registered, edited, and disabled entirely from the dashboard — this guide walks through that UI. It’s a prerequisite for everything else in the data-mode workflow: nothing can be scanned, templated, or requested until at least one datasource exists.
Where To Find It
Section titled “Where To Find It”Open Admin → Data Sources Config in the sidebar (only visible in data mode, and only to organization admins). This opens the Data Sources Config page at its own dedicated screen — “Manage datasources and per-target catalog metadata.”
Only super_admin users can create, edit, or disable datasources here. Anyone else viewing the page sees a “Read-only access” banner and no Add/Edit/Delete controls — they can still see what’s registered, just not change it.
Supported Datasource Types
Section titled “Supported Datasource Types”The Type selector offers seven datasource types: PostgreSQL, Oracle, MySQL, SQL Server, S3, SSH, and Snowflake.
The Datasource List
Section titled “The Datasource List”The Datasources table shows, per row:
- Status — a colored health beacon (green “Healthy”, red “Failing,” gray “Pending”), with a tooltip giving the status, any error message, and when it was last checked. This reflects live connection health, checked periodically — it isn’t the same as catalog scan status (see Catalog Scanning for that).
- Name and Type
- Scrapper Policy — the datasource’s scan policy, shown as a badge (the table literally labels this column “Scrapper Policy,” while the edit form below calls the same setting “Scan Policy”)
- Actions — an Edit button
- Enabled — a toggle switch (see Disabling And Restoring A Datasource)
Registering A Datasource
Section titled “Registering A Datasource”Click Add Datasource. This opens the New datasource wizard on its own page. Its steps depend on the type:
| Type | Steps |
|---|---|
| PostgreSQL, Oracle, MySQL, SQL Server, Snowflake | Type, Connect, Tables, Review |
| SSH | Type, Connect, Files, Columns, Review |
| S3 | Type, Connect, Files, Review |
The footer moves you through them. On a step that checks something, the main button first runs the check (Test connection on Connect, Check on Tables and Files), then turns into Next, which is enabled once the check passes. Editing a field the check depends on brings the check back. Back returns to the previous step, and Cancel leaves the wizard, asking first if you changed anything. S3 has no live checks: its access is checked on the first scan.
- Type: enter a Name (required) and pick one of the seven types. The type can’t be changed later; if you need a different type, register a new datasource instead.
- Connect: the connection fields in the table below, plus Credentials.
- Tables (databases and Snowflake): Target, plus Schema where the type has one. Check lists the tables the login can see, with the ones Target matches highlighted. Click a matched table to preview its columns and rows.
- Files (SSH and S3) and Columns (SSH): Base Path (SSH) or Prefix (S3), and Target. An SSH Target can hold date tokens such as
{YYYY-MM-DD}for daily files, and each SSH schema gets a Table name on Columns. For SSH, see Files And Columns, which covers files with different layouts under one Target, accepting schemas and naming their tables, log formats, dated files and renaming columns. - Review: a summary of every step, each with an Edit link back to it. Scan policy is set here: Auto (default: skip unchanged targets), Disabled (never scan), or Force Refresh (rescan every run, even unchanged). See Catalog Scanning for what scanning does with this setting. More settings holds the Advanced fields and Custom Fields. Click Create datasource to save.
Connection
Section titled “Connection”The fields depend on the type you picked. Schema, Prefix, Base Path and Target are entered on the Tables or Files step; the rest are on Connect:
| Type | Required fields | Also available |
|---|---|---|
| PostgreSQL | Host, Port (default 5432), Database | SSL Mode, Schema, Target, and (Advanced) Connection Timeout, Max Connections |
| Oracle | Host, Port (default 1521), Database | Service Name, Schema (table owner), Target, and (Advanced) Connection Timeout, Max Connections |
| MySQL | Host, Port (default 3306), Database | Charset, Target, and (Advanced) Connection Timeout, Max Connections |
| SQL Server | Host, Port (default 1433), Database | Schema (default dbo), Encryption, Trust Server Certificate, Target, and (Advanced) Connection Timeout, Max Connections |
| S3 | Bucket, Region | Prefix, Target |
| SSH | Host | Port (default 22), Base Path, Target (the schemas are accepted by checking them on Files, and their table names are set on Columns) |
| Snowflake | Account, Warehouse, Database, Schema | Role, Target |
Required fields are marked with an asterisk. For Oracle, Database is used as the service name unless you fill in Service Name, which then takes precedence. If your database team gave you only a service name, put it in Database (or in both fields). Schema (table owner) is optional. Leave it empty and Target is matched against every table the login can read, including tables owned by other accounts, which is the usual case for a read-only login that doesn’t own the tables. Set it to one owner’s name to scan only that owner’s tables; unquoted names are read as upper case, as Oracle does. The catalog lists these tables as OWNER.TABLE, but in the Data Explorer you query them by the table name alone (for example SELECT * FROM ORDERS). Target is optional for saving a datasource but required for scanning it: with no Target the catalog scan skips the datasource and logs skipping datasource (no target configured). It takes names, globs, or a comma-separated list. For a database, * matches every table in the configured schema; for S3 and SSH, use path globs such as **/*.csv (* doesn’t cross /). An SSH pattern can also hold a date token for files written one per day, such as logs/*/app.{YYYY-MM-DD}.log.gz, and each schema you accept becomes one catalog entry (see Dated Files). PostgreSQL, Oracle, MySQL, and SQL Server also get an Advanced section for connection timeout and max-connections tuning; S3, SSH, and Snowflake don’t need one.
SQL Server encryption
Section titled “SQL Server encryption”By default a SQL Server connection uses Encryption = Encrypt and Trust Server Certificate = No: the connection is encrypted, and the server’s certificate must be signed by an authority the Connector host trusts and must name the host you entered. When it doesn’t, Test connection (and later the catalog scan) fails with an error such as:
| Error | Cause |
|---|---|
x509: cannot validate certificate for <ip> because it doesn't contain any IP SANs |
Host is an IP address, and the certificate lists only host names |
x509: certificate is valid for <names>, not <host> |
The certificate doesn’t cover the host name you entered |
x509: certificate signed by unknown authority |
The certificate is self-signed (SQL Server’s own default) or from a private CA the Connector host doesn’t trust |
Either enter a Host the certificate covers, with a certificate the Connector host trusts, or set Trust Server Certificate to Yes. Yes keeps the connection encrypted but stops checking the certificate, so the Connector can no longer tell your server from another machine answering on that address; use it only where you trust the network path to the server. The other Encryption options: Encrypt Optional still checks the certificate while logging in, Strict always checks it (ignoring Trust Server Certificate), and Disable turns encryption off and fails if the server requires it.
Where Connections Come From
Section titled “Where Connections Come From”A datasource is reached from two different places, and the address you enter in Host has to work from both:
| When | Connects from |
|---|---|
| Test connection, the health status, and the catalog scan | The Connector service itself, on the install host |
| Loading data after a request is approved and activated | The request’s own container (a data enclave), which runs on the install host’s container network |
Because the second path starts inside a container, 127.0.0.1 there means the container itself, not the install host. To keep this working for a database or SSH server that runs on the same machine as the Connector, a Host of 127.0.0.1, localhost or ::1 is replaced by the name host.internal when the request container starts, and host.internal points at the install host. The port is kept. This applies to PostgreSQL, Oracle, MySQL, SQL Server and SSH. S3 and Snowflake have no host to replace; they are reached by their own endpoints.
What this means for your setup:
- Datasource on the install host. Register it with Host
127.0.0.1(orlocalhost), as usual. Then make sure the service listens on an address the request containers can reach, not only on127.0.0.1: for example all interfaces (0.0.0.0, such aslisten_addresses = '*'in PostgreSQL), or the install host’s address on the container network. A service bound only to the loopback address passes Test connection and scans, but refuses the connection from a request container. If a host firewall is active, allow inbound connections to the datasource’s port from the container network. - Datasource on another machine. Use its real host name or IP address as Host. It must be reachable from the install host’s containers, not only from the install host’s own shell. Check any firewall, security group or allow-list that restricts the source address; connections come from the install host.
Credentials
Section titled “Credentials”Usernames, passwords, and other secret values are never entered as plain config fields — they’re handled separately here, and are never shown again in the UI once saved.
- Secret Backend — choose file, java-keystore, or aws-secrets-manager. (For SSH datasources,
java-keystoreisn’t offered — it can’t store SSH private keys. See Configure SSH Datasources for how to store one with thefilebackend instead.) See Secret Isolation for how each backend actually stores the value, and Vaults for how to configure each one so Alien Giraffe can reach it — the value itself has to already exist there, managed however your organization already manages that vault. - Depending on the backend, one more field appears: Secret ID (for
aws-secrets-manager), File Path (forfile), or, forjava-keystore, just a note that it’s configured via an environment variable on the server. - Key Mappings — for each credential the type needs (for example Username/Password, or Access Key ID/Secret Access Key/Session Token for S3), a mapping name field sets the name the secret is stored under in the backend, never the credential itself. Do not type a username or password here. When the backend is
java-keystore, enter the keystore alias that holds the value (the defaults are the type’s prefix plus the field, such aspg_username/pg_password,mysql_username,mssql_usernameororacle_username; see Vaults for when to change them; an alias that doesn’t exist in the keystore fails withalias not found in keystore: <alias>); otherwise the defaults are plain names likeusername/password.
Custom Fields
Section titled “Custom Fields”Free-form key/value pairs for anything the other fields don’t cover. They’re under More settings on the Review step, added one at a time with Add field.
After Create datasource, the wizard confirms the datasource was created. The datasource appears in the table immediately, with its health beacon starting at “Pending” until the first check completes. The service checks every datasource when it starts and then every 5 minutes by default (Snowflake every 30), so a datasource you add between checks can stay Pending for a few minutes. Click Test Connection on its row to check it right away.
Editing A Datasource
Section titled “Editing A Datasource”Click Edit on any row to open the same wizard as Edit datasource, pre-filled with its current values. The type is locked; everything else, including credentials, can be changed. Changing a field reopens the checks that depend on it, so run them again before Save datasource.
Disabling And Restoring A Datasource
Section titled “Disabling And Restoring A Datasource”There’s no separate “Disable” button: flip the Enabled toggle in a row off. A confirmation dialog, Disable {name}?, says how many active and approved requests use the datasource. Confirm with Disable.
Disabling a datasource takes effect immediately:
- It is no longer scanned, and new requests can’t include it.
- Every active or approved request that uses it is ended. Each one is revoked with the reason
datasource <name> was disabled, and its environment, tunnel and data explorer are torn down, so its users lose access at once. A request that also uses other datasources is ended as a whole. Pending requests are left alone, since they don’t give access yet.
Disabling through the API (DELETE /v1/admin/datasources/{id}) does the same.
Disabled datasources drop out of the main table. A Show Disabled Datasources (N) link reveals them, styled with a muted background and a gray “Disabled” status dot. Flip the same toggle back on to restore one. Restoring a datasource doesn’t bring back the requests that disabling it ended: their users must request access again.
Permanently Deleting A Datasource
Section titled “Permanently Deleting A Datasource”On a disabled datasource’s row, a trash-icon button opens a confirmation dialog:
Permanently delete datasource? This will hide {name} from the UI and prevent accidental re-enable. This action cannot be undone from the UI.
Confirm with Permanently Delete. Read the wording carefully — this hides the datasource from the UI and blocks it from being re-enabled there; it’s not a claim that the underlying record or any data is erased.
How File Names Become Table Names
Section titled “How File Names Become Table Names”In the Data Explorer, each approved S3 file is one table. Its name is derived from the file’s path, and it isn’t the file name as written:
- The path is taken relative to the datasource: the bucket and Prefix are dropped.
- Every folder name and the file name is upper-cased, and every run of characters other than the letters
A-Zand the digits0-9(dots, dashes, spaces, accented letters) becomes a single_. Underscores at either end are dropped. The extension stays, as_CSV,_PARQUETor_JSONL_GZ. - A name that starts with a digit gets a
T_prefix, because a table name can’t begin with a digit. - The table is named after the file alone. Folders become a prefix only when the request holds files from different folders, so that equal file names stay apart: the folder name (several folders are joined with
__) goes in front, separated by a dot.
| File | Table |
|---|---|
processed_orders/2025_august_orders.csv |
T_2025_AUGUST_ORDERS_CSV |
incoming/raw patient-data.parquet |
RAW_PATIENT_DATA_PARQUET |
incoming/orders.csv and archive/orders.csv in one request |
INCOMING.ORDERS_CSV and ARCHIVE.ORDERS_CSV |
Use the name in a query as shown, for example SELECT * FROM T_2025_AUGUST_ORDERS_CSV or SELECT * FROM ARCHIVE.ORDERS_CSV. The Explorer’s Available Datasets panel lists the exact name of every table in the request. SSH tables aren’t named after files: each SSH schema is one table, named by the Table name an administrator gave it; see Loading Files Into A Request.
Scanning
Section titled “Scanning”There’s no “scan now” button anywhere on this page — scanning is driven by the CLI or a scheduled job, not the dashboard. See Catalog Scanning for how to run one and what the Scan Policy setting above actually controls.
Separately, in a Catalog Targets area, individual scanned targets can carry their own Scrapper Policy override (the same Auto/Disabled/Force Refresh options, with inline descriptions) on top of the datasource-level default — useful if most of a datasource should scan normally but one target should be skipped or force-refreshed. The area lists every target of the datasource you select, by name, with a line saying how many there are. A datasource with more than 30 targets gets a Filter box, and a long list shows 100 at a time with Show more.
Disabled on a target only stops it being scanned. The table stays in the catalog and can still be requested. To take tables out of what users can request, narrow the datasource’s Target: at the next scan, the entries for tables it no longer matches are removed (see Catalog Scanning).
Troubleshooting
Section titled “Troubleshooting”| Symptom | Cause / fix |
|---|---|
After a request is activated, the Explorer or the request’s load status shows a connection refused error naming host.internal and your datasource’s port (for example Unable to connect to Postgres at host=host.internal port=15432 ... Connection refused), while Test connection and the health status pass |
The datasource runs on the install host but listens only on 127.0.0.1. Test connection runs from the Connector service on the host, where that works; the request container reaches the host as host.internal, which a loopback-only service doesn’t answer. Make the datasource listen on an address the containers can reach (for example 0.0.0.0), allow its port through the host firewall, then activate the request again. See Where Connections Come From. |