Amazon S3 & compatible stores

Amazon S3, R2 and MinIO

An object store has no query language, so a DBCrust “query” for S3 is one of seven typed commands (ls, head, get, buckets, cd, pwd, help). They only read: there is no put, delete, copy, multipart, ACL or presign command, in the REPL, in -c/-f, in --read-only and for the AI assistant alike.

dbcrust 's3://my-bucket/logs/?region=eu-west-1' -c 'ls'
dbcrust 's3://my-bucket?region=eu-west-1&profile=prod-ro' -c 'head logs/2026-10-16.log.gz'
dbcrust 'minio://minioadmin:minioadmin@localhost:9000/data?tls=false' -c 'get report.csv'

S3 is not part of the default (bundled) build, nor of standard. The AWS SDK adds crates and needs Rust 1.94.1 or newer to compile (the rest of DBCrust builds with 1.91). Build it with:

cargo install --path . --locked --features aws     # every AWS connector: S3, DynamoDB, CloudWatch Logs
cargo build --features s3                          # S3 only

A default build answers “Amazon S3 support is not compiled into this build (enable the s3 cargo feature, or install a full build)”. See Build features.

Connecting

s3://[ACCESS_KEY[:SECRET_KEY]@]bucket[/prefix][?options]
r2://[ACCESS_KEY[:SECRET_KEY]@]account_id/bucket[/prefix][?options]
minio://[ACCESS_KEY[:SECRET_KEY]@]host[:9000]/bucket[/prefix][?options]

Part	Meaning
`bucket`	the bucket (where a host would be). Leave it out (`s3://?region=…`) to start without one: only `buckets` works until you `\c BUCKET`
`/prefix`	the key prefix `ls` and `\dt` start from (`cd` changes it)
`ACCESS_KEY:SECRET_KEY@`	static credentials. The secret key is the connection’s password: it goes to the secret store, is bound to the endpoint, region and bucket it was saved for, and is redacted everywhere
`?region=`	the region (`eu-west-1`, `auto` for R2). Order: `?region=`, then the region of an explicit `?profile=`, then the environment (`AWS_REGION`) and the default profile
`?endpoint=`	a non-AWS endpoint: `https://host[:port]` or `http://host[:port]` (a bare `host:port` is HTTPS). It becomes the connection’s host, port and TLS flag
`?profile=`	a named profile of `~/.aws/config` (SSO, assumed roles, keys, `credential_process`)
`?path_style=`	force path-style addressing. On by default with a custom endpoint (MinIO has no `bucket.host` DNS), off for AWS’ own hosts
`?anonymous=true`	unsigned requests, for public buckets
`?session_token=`	the session token of temporary static keys (a secret option, stored with the keys)
`?role_arn=`, `external_id=`, `session_name=`	assume a role on top of the credentials above
`?ca_cert=/path/ca.pem`	trust this CA in addition to the system roots (a MinIO with its own certificate)
`?connect_timeout=`, `operation_timeout=`, `max_attempts=`	seconds to connect (default 10), seconds per operation (default: the query timeout), attempts (default 3)

r2:// builds https://<account>.r2.cloudflarestorage.com with region auto (?jurisdiction=eu or fedramp for the regional endpoints). minio:// builds https(s)://host:9000 with path-style addressing and region us-east-1 (?tls=false for plain HTTP; HTTPS is the default). Anything else S3-compatible (Ceph, Wasabi, Backblaze B2, LocalStack…) is s3://bucket?endpoint=….

DBCrust always records the canonical s3:// spelling (recent connections, saved sessions): minio://localhost:9000/data?tls=false becomes s3://data?endpoint=http://localhost:9000&region=us-east-1. It never contains the secret.

Credentials

Ambient credentials are the recommended way: nothing secret is stored by DBCrust.

You write	Credentials used
`s3://ACCESS:SECRET@bucket`	those keys (plus `?session_token=` if given)
`s3://bucket?profile=prod-ro`	that profile, whatever the environment says: `AWS_ACCESS_KEY_ID` does not override an explicit `?profile=`. An unknown profile is an error, never a silent fallback
`s3://bucket?anonymous=true`	none: requests are not signed
`s3://bucket`	the AWS default chain: environment variables → `~/.aws/config` and `~/.aws/credentials` (`default` profile or `AWS_PROFILE`, SSO included) → web identity → ECS → EC2 instance metadata

Contradictions are refused rather than resolved silently: keys with ?profile=, anonymous=true with keys, a profile or a role, an access key id without its secret.

SSO works through the profile: run aws sso login --profile my-sso once, then connect with ?profile=my-sso. credential_process entries of a profile run as you configured them.

The SDK also honours AWS_ENDPOINT_URL / AWS_ENDPOINT_URL_S3 and the endpoint_url of a profile when the URL names no endpoint, which is how LocalStack setups usually work.

Cloudflare R2, MinIO and friends

# Cloudflare R2 (S3 API tokens: Access Key ID + Secret Access Key)
dbcrust 'r2://KEY:SECRET@0123456789abcdef/media/images/'

# MinIO over plain HTTP, then with TLS and a private CA
dbcrust 'minio://minioadmin:minioadmin@localhost:9000/data?tls=false'
dbcrust 'minio://minioadmin:minioadmin@minio.internal:9000/data?ca_cert=/etc/ssl/minio-ca.pem'

# LocalStack, or any endpoint
dbcrust 's3://test:test@bucket?endpoint=http://localhost:4566&region=us-east-1'

Without a region and with a custom endpoint, DBCrust signs for us-east-1 (the endpoint ignores it). Against AWS itself a missing region is an error that says how to set one.

Commands

One command per line (a ; is a character of a key, not a separator; -f files take one command per line, blank lines and # comments are skipped). Verbs are case-insensitive. Arguments are words with 'single' or "double" quotes (head "my file.txt"), and \ escapes " and \ inside double quotes. Keys are always bucket-absolute; cd only sets the default of ls and \dt.

Command	S3 call	Result
`ls [PREFIX] [-r\|--recursive] [-n\|--limit N]`	ListObjectsV2, delimiter `/` (none with `-r`)	`key, type, size, last_modified, storage_class, etag`; `type` is `dir` for a common prefix (no other cells) or `object`
`head KEY`	HeadObject	`attribute, value`: bucket, key, size, last_modified, etag, content_type/encoding/language/disposition, cache_control, expires, storage_class, version_id, server_side_encryption, sse_kms_key_id, restore, replication_status, object_lock_mode, accept_ranges, and each `metadata.<name>` (only the attributes the object has)
`get KEY [bytes=A-B \| bytes=A- \| bytes=-N \| --bytes N] [--base64]`	ranged GetObject	text: `line, text`; binary: `offset, hex, ascii` (16 bytes per row); `--base64`: `offset, base64` (48 bytes per row)
`buckets`	ListBuckets	`bucket, created, region`
`cd [PREFIX\|..\|/]`	none	the new location; `cd` alone or `cd /` goes to the bucket root
`pwd`	none	`s3://bucket/prefix`
`help`	none	the commands

A leading / of a prefix argument (ls /logs/, cd /logs) is dropped; a key is used exactly as typed.

key           | type   | size | last_modified            | storage_class | etag
--------------+--------+------+--------------------------+---------------+---------------------------------
bin/          | dir    | NULL | NULL                     | NULL          | NULL
docs/         | dir    | NULL | NULL                     | NULL          | NULL
many/         | dir    | NULL | NULL                     | NULL          | NULL
 odd name.txt | object |   35 | 2026-10-16T08:02:45.698Z | STANDARD      | 5d919242955c77b89968c63014a88120
empty.txt     | object |    0 | 2026-10-16T08:02:45.677Z | STANDARD      | d41d8cd98f00b204e9800998ecf8427e
(5 rows)

`get` is always bounded

get never issues a GetObject without a Range, and a hard cap decides how much one call may read:

no range: the first 64 KiB (get_bytes option);
--bytes N, bytes=A-B, bytes=-N (the last N bytes): refused before any request when they ask for more than the cap (1 MiB by default, the max_get_bytes option, at most 64 MiB);
bytes=A-: from A up to the cap;
a server that ignores the Range header is read only up to the requested size, then the connection is dropped.

Text (valid UTF-8; a multi-byte character cut by the end of the range is dropped) is shown as numbered lines (at most max_rows); anything with NUL bytes or invalid UTF-8 as a hex dump. Content is shown as stored: a .gz object is binary, nothing is decompressed. When more of the object remains, a warning in the log (stderr) says how many bytes were shown, from which offset, of what total size; at debug level every get reports that. An empty object answers with no rows.

Listing is bounded too

ls asks S3 for at most 1000 keys per request and stops at max_rows entries (default 1000, ls -n N can ask for fewer, never more) or after max_pages requests (default 100), whichever comes first. A listing cut short is reported by a warning in the log (stderr) with how to narrow it. Listing is billed per 1000 keys: prefer a longer prefix to a bigger limit.

Options (URL query)

Option	Default	Meaning
`max_rows`	1000 (max 1 000 000)	rows per result (listing entries, text lines, buckets)
`max_pages`	100	listing requests per command
`scan_limit`	1000	entries `\dt` and completion list
`get_bytes`	64 KiB	bytes `get` shows without a range
`max_get_bytes`	1 MiB (max 64 MiB)	hard cap per `get`

Browsing with backslash commands

Command	Shows
`\l`	the account’s buckets (`ListBuckets`; needs `s3:ListAllMyBuckets`)
`\c BUCKET`	switch bucket (checked with HeadBucket) and go to its root
`\dt`	the prefixes (`dir`) and objects (`object`) at the current prefix, as full keys you can paste into `ls`, `head` and `get`; bounded by `scan_limit`
`\d KEY`	the HeadObject attributes (a prefix: a sample of its entries)

The connection banner reads Server: Amazon S3 <region> (or S3-compatible store at <endpoint> <region>); the credential source and location are in the debug log. No secret is ever shown.

Tab completion offers the commands, then the prefixes and keys of the last \dt listing.

Read-only by construction

Every S3 command reads, so --read-only accepts them all and refuses anything else (rm, put, SQL…). That is a statement about the client: what the credentials may do is decided by IAM, which a client cannot prove. DBCrust therefore does not claim a server-enforced read-only mode for S3 (the classification is client-structural: no write API call is reachable from any command). For a hard limit, give the key a read-only policy, for one bucket and prefix:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::my-bucket",
      "Condition": { "StringLike": { "s3:prefix": ["logs/*"] } } },
    { "Effect": "Allow", "Action": ["s3:GetObject"], "Resource": "arn:aws:s3:::my-bucket/logs/*" },
    { "Effect": "Deny", "Action": ["s3:Put*", "s3:Delete*"], "Resource": "*" }
  ]
}

Connecting checks the bucket with HeadBucket. A bucket you may read objects of but not head (access denied) still connects; an unknown bucket, wrong region, unknown or expired credentials do not, and say why: the S3 error code, a hint, the request id, and for a bucket in another region the region to use (?region=eu-west-3).

The credential broker (dbcrust broker enroll) refuses S3 for now (unsupported_engine): IAM cannot be proven read-only from the client, and an SDK client would inherit the broker process’ AWS_* environment and ~/.aws. Enrollment will come with explicit, pinned region, endpoint and credentials.

Sessions and secrets

Saved sessions and recent connections keep the bucket, prefix, endpoint, region, profile and the access key id; the secret access key and the session token go to the secret store, bound to the endpoint host and port, the bucket, the access key id, the transport and the region. A key saved for eu-west-1 is never offered to us-east-1, to another bucket or endpoint, or to a plain-HTTP endpoint. Errors, Debug output and logs mask keys, tokens and anything shaped like an access key id (AKIA…, ASIA…).

AI assistant

?? can read an S3 bucket: it is told the seven commands and their costs, may only run what --read-only accepts, and sees the same bounded results.

Desktop app

The desktop app is built without the AWS connectors and does not offer S3 in its connection form. Use the CLI.

What is not supported (yet)

Writing in any form (put, delete, copy, multipart, tagging, ACLs, lifecycle, presigned URLs, SSE-C).
SELECT … FROM 's3://bucket/key.parquet': querying objects with SQL through DataFusion is a follow-up (an object_store adapter over this client).
Object versions (ListObjectVersions), requester-pays buckets, S3 Express directory buckets, multi-region access points, access-point ARNs in place of a bucket name.
Agent enrollment through the broker.

Troubleshooting

Message	Cause
Amazon S3 support is not compiled into this build	rebuild with `--features aws` (Rust ≥ 1.94.1)
no AWS region: add ?region=…	AWS needs a region; set `?region=`, `AWS_REGION` or a profile region
the bucket is in region X: connect with ?region=X	the bucket lives elsewhere
SignatureDoesNotMatch	wrong secret key, or key issued for another endpoint/region (R2 needs `auto`)
ExpiredToken	refresh the session (`aws sso login`, a new token)
an access key id needs its secret access key	`s3://AKIA…@bucket` without a secret and none saved; use `ACCESS:SECRET@`, a saved session, or `?profile=`
AccessDenied on `ls` / `get`	the policy lacks `s3:ListBucket` / `s3:GetObject` for that bucket or prefix
dispatch failure … connection refused	nothing listens at the endpoint (port, `tls=false`, `ca_cert`)