Database Connections

Open the Connections screen

In the sidebar, open the Data Hub and then Database connections. The screen shows a connector catalogue and your saved connections. You can also choose Connect database from the Data Hub toolbar.

Supported engines

• PostgreSQL, MySQL and SQL Server: relational SQL
• Amazon Redshift: a Postgres-compatible warehouse
• Snowflake and Google BigQuery: cloud warehouses
• MongoDB: a document store; collections appear alongside tables
• REST / GraphQL API: an HTTP endpoint that returns JSON records
• Vector databases: Pinecone, Qdrant and Chroma

A few connection options are not supported yet and the form says so: SSH tunnels, and IAM authentication for connectors that offer it.

Creating a connection

Choose an engine, fill in its fields, and use Test before saving. The fields depend on the engine:

• PostgreSQL, MySQL, SQL Server, Redshift, MongoDB: host, port (defaulted for you), database, schema, user and password, or a connection string, plus an SSL switch.
• Snowflake: account, warehouse and role, with a user and password or key.
• BigQuery: project, dataset, location and a service-account JSON key.
• REST / GraphQL API: a base URL, the path to the records in the JSON response, a pagination style (none, offset or page) and an authentication mode (none, bearer token, custom header or basic).
• Vector databases: the provider (Qdrant, Chroma or Pinecone), a collection and a namespace. Tables, views and collections are then discovered and listed; Re-discover tables refreshes the list.

How credentials are protected

Connection secrets are encrypted at rest with AES-256-GCM on the server and are redacted from error messages. Rows travel to the browser as Apache Arrow. Connecting is something you ask for: nothing is queried until you browse or import. The server refuses private, loopback and cloud-metadata hosts by default; an operator can opt in to private-network hosts with the DB_CONNECT_ALLOW_PRIVATE_HOSTS setting.

Browse, then import

Select a connection to browse its tables and preview rows. When you import a table you choose how the data is held:

• Live: the dataset stays bound to the table and reads from it. The preview is limited (the first rows) and is clearly marked as a preview. Transform needs a snapshot, because it refuses to treat a 200-row preview as the whole table.
• Snapshot: a complete copy of the table is stored as a dataset version. Snapshots are read in batches by key and are capped at 25,000 rows; a truncated copy is labelled as partial.

A table can be imported both ways. A saved connection remembers which tables you have imported and how, and reconnects the live binding after a reload. Imports made before complete table references were saved may need re-importing to reconnect.

Key-based batch reading is not a repeatable-read transaction. Sources that fall back to offsets carry a weaker consistency warning.

Letting pipelines write

Reading a table into a pipeline is always allowed. Writing is off until you turn it on, per connection, under that connection's Writes settings, and the server enforces whatever is saved. Choose one of five modes:

• Off: pipelines can read but never write.
• Ask each time: a run that writes stops and shows what it is about to do. You approve with a click; replacing a whole table makes you type the table's name.
• Type to confirm: every write needs the table's name typed out.
• Dry run first: the write is tried inside a transaction and rolled back, and you see exactly what would be added, changed and removed. Only those rows can then be applied.
• Auto: writes without asking, only to the tables you list and only up to the row limit. For scheduled runs nobody is watching.

The same screen lists a write log of what pipelines have done on this connection: table, kind of write, status, whether it was a dry run, how many rows, how it was approved and when. See the Write to Database step in Pipeline steps.

Tips

• Use a read-only database user for connections you only browse.
• Combine a database refresh step with an upsert step in a pipeline to load only rows that changed.
• Live SQL Server, Snowflake, BigQuery and vector connections have adapter-level tests, but have not all been exercised against real accounts. Report anything unexpected.