> ## Documentation Index
> Fetch the complete documentation index at: https://docs.adriel.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Data reference

> Import model, field type mapping, refresh cadence, and limits for the Amazon S3 data source in Adriel.

## Introduction

Amazon S3 is Amazon Web Services' object storage service. Customers drop report or dataset files into an S3 bucket, and the Adriel Amazon S3 connector imports those files so their columns can be used as metrics and breakdowns in widgets. The connector points one Adriel data source at a single bucket plus an optional directory prefix, discovers the files under that prefix, parses each one, and writes the resulting rows into a per-data-source cache table. Widgets then query that cache.

As a file-import connector, the available fields are not a fixed catalog — they are the columns of the imported files. Column data types are inferred from the file contents: numeric columns become metrics, while text and other columns can serve as either a metric or a breakdown. The connector supports CSV, XLSX, and Zipped CSV files and is typically used for partner exports, agency report deliveries, or ad-platform reports that Adriel does not connect to natively.

To connect this data source, see [How to connect Amazon S3 to Adriel](/data-sources/a-f/amazon-s3/how-to-connect).

## Data refresh strategy

Amazon S3 is a **file-cache** connector. On a fixed schedule, the connector re-lists the files under the configured prefix, re-reads each matching file, parses it, and writes the parsed rows into the per-data-source cache table. Widgets always read from this cache, never live from the bucket.

**Cadence.** The cache refreshes **twice daily**, at **03:00 UTC** and **15:00 UTC** (cron `0 3,15 * * *`).

**Manual refresh is disabled.** There is no user-triggered refresh for this connector; new or changed files in the bucket become visible only at the next scheduled cycle.

**Partial-failure safety.** A refresh that fails part-way leaves the previous cached snapshot in place rather than producing empty results.

<Note>
  **Import behavior on each cycle**

  Each cycle re-reads every matching file and upserts the parsed rows into the cache table. When **Keep only recent files** is enabled, only the single most-recently-modified file is kept and it replaces the prior dataset on every cycle. When it is not enabled, rows from all matching files are combined in the cache.
</Note>

## Architecture levels

Bucket → directory prefix → files.

The connection binds to these three levels:

1. **Bucket** — a single customer-owned S3 bucket.
2. **Directory prefix** — an optional path prefix within the bucket. A `*` prefix enumerates the bucket root. Discovery is recursive, so objects in subfolders under the prefix are included.
3. **Files** — every object under the prefix whose key ends with the configured file extension (`.csv`, `.xlsx`, or `.zip`).

One Adriel data source binds to one bucket-plus-prefix. All matching files share a single schema and their rows are combined into one cache table. To import a differently-shaped dataset, create a second data source.

## Date range limits

There is no platform-imposed date range limit, and the connector imposes no date axis at the storage layer. Date-range bounds come entirely from the file contents. When a source file includes a date column that is mapped as the date field in **Blend Data settings**, that column drives date-range filtering on the dashboard. When no date column is mapped, all rows are treated as a single snapshot set and the dashboard date range does not restrict them.

## Import model

The import model is defined by the bucket and prefix selection, the file-discovery and filtering rules, and the per-format parsers. There is no SQL query layer — imported rows are cached and served through Adriel's internal cache query engine.

### Bucket and prefix selection

The connection specifies a bucket name, an AWS region, and an optional directory path. Files are discovered under that path; a `*` path enumerates the bucket root. A trailing `/` on the directory path is normalized before it is used as the prefix.

### File discovery and filtering

* **Enumeration.** Files are listed with paginated `ListObjectsV2` requests (1,000 keys per request), recursively, up to a hard cap of **30,000 files** (up to 30 requests). Files beyond that cap are silently ignored.
* **Extension filter.** Only keys ending in the configured file extension (`.csv`, `.xlsx`, or `.zip`) are kept.
* **Include filter (`fileNameToFilter`).** An optional comma-separated list of regular-expression patterns. When set, a file is kept only if at least one pattern matches its key.
* **Exclude filter (`fileNameToOmit`).** An optional comma-separated list of regular-expression patterns. When set, a file is dropped if any pattern matches its key. Exclusion is applied after inclusion and wins on overlap.
* **Keep only recent files.** When enabled, discovered files are sorted by last-modified time and only the single newest file is imported, replacing the prior dataset on each cycle.

### File-format parsing

| File format | Parsing behavior                                                                                                                                       |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| CSV         | Streamed and parsed directly. An object whose S3 metadata contains `rows-count: 0` is treated as empty and skipped without error.                      |
| XLSX        | Buffered, then only the **first non-null sheet** is converted to CSV and parsed; additional sheets are ignored.                                        |
| Zipped CSV  | The archive is unpacked and only the **first `.csv` entry** is parsed. An archive with no `.csv` entry fails with `CSV file not found in zip archive`. |

The first row of each file is treated as the header (the header row and data-start column are configurable per data source). Column names, order, and types must align across all matching files in the prefix; a file whose columns do not match the established schema fails that refresh cycle.

## Filters

Amazon S3 does not expose a SQL WHERE layer. Two filtering mechanisms apply.

### File-selection filters

The `fileNameToFilter` (include) and `fileNameToOmit` (exclude) asset settings narrow which files in the prefix are imported before any rows reach the cache. Both accept comma-separated regular-expression patterns matched against the full S3 key (see [Import model](#import-model)).

### Date-range filtering

Date-range filtering is applied only when a column has been mapped as the date field in **Blend Data settings**. Without a mapped date column, the dashboard date range has no effect and all cached rows are shown as a single snapshot set.

## Field type mapping

Amazon S3 has no fixed metric or breakdown catalog. The available fields are the columns of the imported files, typed at ingestion by sampling up to 10,000 rows and inferring each column's type from its values. Internal bookkeeping columns (`id`, `filename`, `file_updated_at`, `created_at`, `updated_at`, and the raw-data breakdown column) are hidden from the available fields.

<Note>
  **How to read the columns**

  The **Data type** column uses the platform's ten-value vocabulary: **Number**, **Currency**, **Percentage**, **Ratio**, **Duration**, **Date**, **Text**, **URL**, **Array**, **Boolean**. The **Field role** column indicates whether the field can serve as a metric, a breakdown, or both.
</Note>

<Note>
  **ID-like columns are always text**

  A column whose name is exactly `id`, ends in `_id`, contains `_id_`, ends in ` id`, or is `id` followed by digits is forced to a text field even when its values are numeric, so identifiers behave as breakdowns rather than metrics. This override is applied regardless of the detected value type.
</Note>

Each detected column type maps to an Adriel field type as follows.

| Detected column type                                                                                                   | Adriel field type | Data type  | Field role          |
| ---------------------------------------------------------------------------------------------------------------------- | ----------------- | ---------- | ------------------- |
| Numeric values (including currency-symbol-prefixed values such as `$1,200` or `₩5000`, which are converted to numbers) | `number`          | Number     | Metric              |
| Percentage-formatted values                                                                                            | `number`          | Percentage | Metric              |
| Boolean values                                                                                                         | `boolean`         | Boolean    | Metric or breakdown |
| Text or mixed values, or any type that cannot be resolved                                                              | `string`          | Text       | Metric or breakdown |
| All-blank column (no non-empty values sampled)                                                                         | `string`          | Text       | Metric or breakdown |
| Column name is ID-like *(override)*                                                                                    | `string`          | Text       | Breakdown           |

<Note>
  **Date columns**

  The connector has no native date breakdown. A column containing dates is imported as a text field and only participates in date-range filtering when it is mapped as the date field in Blend Data settings (see [Date range limits](#date-range-limits)).
</Note>

### Adriel-added fields

Beyond the file's own columns, the connector exposes the data source name as a selectable field.

| Field            | Description                                                                                                                             | Data type | API Key          |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------------- | --------- | ---------------- |
| Data source name | The name of the Amazon S3 data source the row came from. Useful as a breakdown when several S3 data sources are combined in one widget. | Text      | `datasourceName` |

### Aggregation defaults

When no explicit aggregation is set on a field, the cache query engine chooses one based on the inferred column type:

| Detected column type               | Default aggregation                                                                                  |
| ---------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Numeric                            | `SUM`                                                                                                |
| Text, boolean, and all other types | Single-value collapse — returns the value when a group has a single distinct value, otherwise empty. |

Users cannot manually override the inferred column types; the type is determined at ingestion from the sampled values.

## Limitations

Based on the current connector:

* **Refresh is scheduled, not on demand.** Data updates only at 03:00 UTC and 15:00 UTC; manual refresh is disabled, so changes in the bucket are not reflected until the next cycle.
* **30,000-file enumeration cap.** Only the first 30,000 objects under the prefix are scanned; files beyond that cap are silently ignored with no warning.
* **Uniform schema required across files.** All matching files must share the same columns, order, and types. A mismatched file fails that refresh cycle (`INVALID_CSV_MISMATCHED_COLUMN_COUNT` / `INVALID_CSV_WRONG_COLUMNS`).
* **XLSX — first sheet only.** Only the first non-null sheet of an XLSX file is imported; other sheets are ignored.
* **Zipped CSV — first entry only.** Only the first `.csv` entry inside a ZIP archive is imported; an archive with no `.csv` entry fails with `CSV file not found in zip archive`.
* **No date breakdown at the storage layer.** Date-range filtering requires a date column mapped through Blend Data settings; otherwise all rows are a single snapshot set.
* **One bucket and prefix per data source.** Combining differently-shaped datasets requires additional data sources and Blend Data configuration.
* **Column types are inferred, not user-set.** Types are detected from sampled values at ingestion and cannot be manually specified.
* **Empty-object short-circuit (CSV only).** An S3 object whose metadata contains `rows-count: 0` is skipped for CSV files; XLSX and ZIP files do not check this metadata.
* **Read permission required, or the data source disconnects.** The IAM principal must allow both `s3:ListBucket` (on the bucket) and `s3:GetObject` (on the prefix). A missing `s3:ListBucket` permission surfaces as `EXTERNAL_SERVICE_PERMISSION_MISSING` and disconnects the data source. A bucket name containing `/` is rejected with `WRONG_S3_BUCKET_NAME`.

## API references

* [Amazon S3 product documentation](https://docs.aws.amazon.com/s3/)
* [ListObjectsV2 API reference](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html)
* [GetObject API reference](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html)
* [IAM policies for Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html)

## See also

* [How to connect Amazon S3](/data-sources/a-f/amazon-s3/how-to-connect) (paired how-to)
* [File from SFTP data reference](/data-sources/a-f/file-from-sftp/data-reference) — alternative file-import source
* [Custom URL Import data reference](/data-sources/a-f/custom-url-import/data-reference) — for HTTPS-hosted exports