> ## Documentation Index
> Fetch the complete documentation index at: https://docs.adriel.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Data reference

> Import model, field type mapping, refresh cadence, and limits for the Custom URL Import data source in Adriel.

## Introduction

Custom URL Import ingests data from any HTTP or HTTPS endpoint that returns JSON or CSV. The endpoint URL, HTTP method, request headers, optional body, response format, and a refresh mode are all user-supplied. On a daily schedule the platform fetches the endpoint, parses the response, and caches the resulting rows so widgets can query them alongside platform-native sources.

A typical use case is a partner-hosted export endpoint or an internal reporting API that no native connector covers — for example a vendor-specific performance API, a public dataset, or an internal BI rollup. The connector also handles date-parameterized endpoints and paginated responses.

As a free-format file/storage connector, the available fields are not a fixed catalog — they are inferred at import time from whatever columns the fetched response contains. Numeric columns become metrics and non-numeric columns become breakdowns, so the data can be aggregated in widgets without additional configuration.

<Note>
  **Custom URL Import is a paid add-on**

  It is not included in standard plans. Contact sales for availability and pricing.
</Note>

To connect this data source, see [How to connect Custom URL Import to Adriel](/data-sources/a-f/custom-url-import/how-to-connect).

## Data refresh strategy

Custom URL Import is a **daily-cache** connector (`SimpleDailyCacheV2`). After the initial connection, historic daily data from the last **180 days** is backfilled (`firstTimeFetchDays`, default 180). On each scheduled run the most recent day of data is fetched again and merged into the cache (`refreshDays`, default 1; one day per fetch call).

**Refresh schedule (UTC).** The cache runs twice daily at **02:00 and 14:00 UTC** (`0 2,14 * * *`).

<Note>
  **Refresh schedule under review**

  The back-office reference doc lists the schedule as 5:00 AM and 5:00 PM UTC, while the connector source lists 02:00 and 14:00 UTC. This page follows the source value; the discrepancy is flagged for reviewer confirmation.
</Note>

**No manual refresh.** User-triggered refresh is disabled (`allowUserManualRefresh = false`), and sub-daily refresh is not supported. All dashboard queries are answered from the cache; the configured URL is called only during scheduled refresh cycles.

The refresh mode controls how newly fetched rows merge with the existing cache — see [Import model](#import-model).

## Architecture levels

Each data source binds to a single HTTP or HTTPS endpoint.

Endpoint → response → rows:

1. **Endpoint** — one user-supplied URL (with method, headers, and optional body) per data source.
2. **Response** — the JSON or CSV payload returned by that endpoint, optionally accumulated across paginated requests.
3. **Rows** — the parsed and flattened records extracted from the response and cached for widget queries.

To ingest a second endpoint, create a second data source. Multi-file behavior does not apply — each refresh is a single request (or a sequence of paginated requests to the same endpoint).

## Date range limits

The connector imposes no date range limit of its own. Any rate limit, payload cap, or page-size restriction comes from the source API. The initial backfill covers the last 180 days; each scheduled run refreshes the most recent day.

If the source API paginates and cannot return a full day of data within the pages the connector can accumulate, that data may be truncated. Endpoints in this situation should be split or batched at the source, and the scheduler should be configured so the total number of daily requests stays within the API's rate limits.

## Import model

Each refresh constructs one or more requests to the configured endpoint, parses the response into rows, injects a `_date` column, and caches the result.

**URL and placeholders.** Before each request, the following tokens are substituted in the URL, the request headers, and the POST body:

| Placeholder    | Substituted value                                               |
| -------------- | --------------------------------------------------------------- |
| `$date`        | The request date, formatted with the user-supplied `dateFormat` |
| `$plusOneDate` | The request date plus one day, same format                      |
| `$page`        | Current page number (page-number pagination only; 1-based)      |
| `$offset`      | Current offset (offset pagination only)                         |
| `$limit`       | Offset step size (offset pagination only)                       |
| `$token`       | Access token obtained from the refresh-token flow               |

The `dateFormat` string uses `date-fns` conventions and requires lowercase `yyyy` and `dd`; uppercase `YYYY` or `DD` is rejected at validation.

**HTTP request.** The method (GET, POST, PUT, or DELETE), headers, and body are user-supplied. Static-token auth is passed as an `Authorization` header. Token rotation is supported via a dedicated refresh endpoint, and the rotated token is persisted to the connection.

**Response format and parsing.**

| Format | Parsing behavior                                                                                                                                                                                                                                   |
| ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| JSON   | The row array is located via an optional dot-path (`responseBodyPath`, e.g. `data.results`); the resolved value must be an array. Nested objects are flattened with an underscore (`_`) delimiter. Specific keys can be omitted before flattening. |
| CSV    | The response body is parsed directly and skips flattening. An optional `csvColumnNames` override replaces the source headers with a comma-separated list (lowercased, spaces replaced with underscores).                                           |

Rows can be filtered server-side before insertion (for example, to drop test or null rows). After parsing, a `_date` column equal to the request date is set on every row, so `_date` is always present in the cached schema regardless of what the endpoint returns.

**Pagination.**

| Mode        | Pattern                        | Behavior                                                                                                                            |
| ----------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------- |
| Page-number | `?page=$page`                  | `$page` starts at 1 and increments; the loop exits when a page is empty or the declared total count or total page count is reached. |
| Offset      | `?offset=$offset&limit=$limit` | `$offset = (page - 1) * offsetStep`; `offsetStep` must be at least 1; same exit conditions as page-number mode.                     |

CSV responses return the first page only — paginated results are not accumulated for CSV.

**Refresh modes.**

| Mode               | When to use                                              | Merge behavior                                                                                                                                                                                              |
| ------------------ | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Date-parameterized | The source API supports filtering by date.               | `$date` is substituted with each date that needs refreshing (typically yesterday and the day before), one request per date. Rows are upserted on the declared unique-key columns; unique keys are required. |
| Daily full-list    | The source API always returns the complete current list. | No `$date` is needed. Without unique keys, the cache is truncated and replaced on each refresh (`replaceAllDataOnRefresh`). With unique keys, rows are upserted instead.                                    |

## Field type mapping

Custom URL Import has no fixed metric or breakdown catalog. The available fields are the columns present in the fetched response, inferred at import time by sampling the parsed rows. New columns that appear as the data changes are added automatically (`extraColumnsLogic = 'add'`).

<Note>
  **How to read the columns**

  The **Data type** column uses the platform's ten-value vocabulary: **Number**, **Currency**, **Percentage**, **Ratio**, **Duration**, **Date**, **Text**, **URL**, **Array**, **Boolean**. The **Field role** column indicates whether the field serves as a metric or a breakdown.
</Note>

<Note>
  **`_id`-like columns are always text**

  A column whose name ends in `_id`, contains `_id_`, matches `id` followed by digits, or is exactly `id` is exposed as a text field even when its values look numeric, so identifiers behave as breakdowns rather than metrics.
</Note>

Each parsed column maps to an Adriel field type as follows.

| Parsed column                                             | Adriel field type     | Data type | Field role                                                                                  |
| --------------------------------------------------------- | --------------------- | --------- | ------------------------------------------------------------------------------------------- |
| Numeric values                                            | `number`              | Number    | Metric                                                                                      |
| Currency-symbol-prefixed values (`$`, `₩`, `¥`, `￥`, `€`) | `number`              | Number    | Metric (the currency symbol is stripped and the value coerced to a number before detection) |
| Date-parseable values                                     | `date`                | Date      | Breakdown; enables date-range filtering on the dashboard                                    |
| Any other text values                                     | `string`              | Text      | Breakdown                                                                                   |
| All-blank column                                          | `string`              | Text      | Breakdown (defaults to text when no value is present to sample)                             |
| Column name ends in `_id` (or other id-like patterns)     | `string` *(override)* | Text      | Breakdown                                                                                   |

<Note>
  **Field-type inference is the shared file-import path**

  Column types are inferred by sampling the parsed rows (`getColumnTypesFromCsvRows` / `detectTypeFromString`), the same inference used for other file-import connectors. The exact behavior on Custom URL Import rows is flagged for reviewer confirmation.
</Note>

### Adriel-added fields

Adriel injects one field on top of the fetched columns.

| Field | Description                                                                                                                               | Data type | API Key |
| ----- | ----------------------------------------------------------------------------------------------------------------------------------------- | --------- | ------- |
| Date  | The request date, set on every cached row so date-level breakdowns and date-range filtering work regardless of what the endpoint returns. | Date      | `_date` |

### Aggregation defaults

When no explicit aggregation is set on a field, the default follows the inferred column type: numeric columns aggregate as a sum, and text columns act as breakdowns that group the rows. Date-parameterized mode stores the request date in `_date` for date-level grouping. *(Inferred from the shared file-import aggregation path — flagged for reviewer confirmation.)*

## Limitations

Based on the current connector:

* **Daily cadence only.** The cache refreshes on a fixed twice-daily schedule; sub-daily refresh is not supported.
* **No manual refresh.** User-triggered refresh is disabled (`allowUserManualRefresh = false`).
* **No live querying.** All dashboard queries are answered from the cache; the endpoint is called only during scheduled refresh cycles.
* **Data-size truncation.** The connector applies no payload limits of its own, but large responses may be truncated by the source API's own rate limits, payload caps, or page-size restrictions.
* **CSV pagination is single-page.** CSV response mode does not accumulate paginated results. Endpoints that return CSV across multiple pages need a JSON wrapper or a single-page response.
* **JSON responses must be arrays.** After `responseBodyPath` extraction, a JSON response that does not resolve to an array is rejected.
* **`dateFormat` casing.** The format string must use lowercase `yyyy` and `dd`; uppercase `YYYY` or `DD` is rejected at validation.
* **OAuth2 authorization-code flow is not built in.** The initial token must be obtained outside the platform and supplied. Token-rotation auth is supported via a dedicated refresh endpoint, but the first token still comes from outside.
* **Unique keys required for date-parameterized mode.** Upsert needs unique-key columns to decide between insert and update; without them, use daily full-list mode.

## API references

Custom URL Import has no dedicated vendor API — it calls the endpoint the customer supplies. Refer to the source API's own documentation for request format, authentication, rate limits, and pagination. For platform-side setup, see the paired [How to connect Custom URL Import](/data-sources/a-f/custom-url-import/how-to-connect).

## See also

* [How to connect Custom URL Import](/data-sources/a-f/custom-url-import/how-to-connect) (paired how-to)
* [Amazon S3 data reference](/data-sources/a-f/amazon-s3/data-reference) — for bucket-hosted file exports
* [File from SFTP data reference](/data-sources/a-f/file-from-sftp/data-reference) — for files pulled from an SFTP server