Skip to main content

​
Introduction

Custom URL Import ingests data from any HTTP or HTTPS endpoint that returns JSON or CSV. The endpoint URL, HTTP method, request headers, optional body, response format, and a refresh mode are all user-supplied. On a daily schedule the platform fetches the endpoint, parses the response, and caches the resulting rows so widgets can query them alongside platform-native sources. A typical use case is a partner-hosted export endpoint or an internal reporting API that no native connector covers — for example a vendor-specific performance API, a public dataset, or an internal BI rollup. The connector also handles date-parameterized endpoints and paginated responses. As a free-format file/storage connector, the available fields are not a fixed catalog — they are inferred at import time from whatever columns the fetched response contains. Numeric columns become metrics and non-numeric columns become breakdowns, so the data can be aggregated in widgets without additional configuration.
Custom URL Import is a paid add-onIt is not included in standard plans. Contact sales for availability and pricing.
To connect this data source, see How to connect Custom URL Import to Adriel.

​
Data refresh strategy

Custom URL Import is a daily-cache connector (SimpleDailyCacheV2). After the initial connection, historic daily data from the last 180 days is backfilled (firstTimeFetchDays, default 180). On each scheduled run the most recent day of data is fetched again and merged into the cache (refreshDays, default 1; one day per fetch call). Refresh schedule (UTC). The cache runs twice daily at 02:00 and 14:00 UTC (0 2,14 * * *).
Refresh schedule under reviewThe back-office reference doc lists the schedule as 5:00 AM and 5:00 PM UTC, while the connector source lists 02:00 and 14:00 UTC. This page follows the source value; the discrepancy is flagged for reviewer confirmation.
No manual refresh. User-triggered refresh is disabled (allowUserManualRefresh = false), and sub-daily refresh is not supported. All dashboard queries are answered from the cache; the configured URL is called only during scheduled refresh cycles. The refresh mode controls how newly fetched rows merge with the existing cache — see Import model.

​
Architecture levels

Each data source binds to a single HTTP or HTTPS endpoint. Endpoint → response → rows:
  1. Endpoint — one user-supplied URL (with method, headers, and optional body) per data source.
  2. Response — the JSON or CSV payload returned by that endpoint, optionally accumulated across paginated requests.
  3. Rows — the parsed and flattened records extracted from the response and cached for widget queries.
To ingest a second endpoint, create a second data source. Multi-file behavior does not apply — each refresh is a single request (or a sequence of paginated requests to the same endpoint).

​
Date range limits

The connector imposes no date range limit of its own. Any rate limit, payload cap, or page-size restriction comes from the source API. The initial backfill covers the last 180 days; each scheduled run refreshes the most recent day. If the source API paginates and cannot return a full day of data within the pages the connector can accumulate, that data may be truncated. Endpoints in this situation should be split or batched at the source, and the scheduler should be configured so the total number of daily requests stays within the API’s rate limits.

​
Import model

Each refresh constructs one or more requests to the configured endpoint, parses the response into rows, injects a _date column, and caches the result. URL and placeholders. Before each request, the following tokens are substituted in the URL, the request headers, and the POST body: The dateFormat string uses date-fns conventions and requires lowercase yyyy and dd; uppercase YYYY or DD is rejected at validation. HTTP request. The method (GET, POST, PUT, or DELETE), headers, and body are user-supplied. Static-token auth is passed as an Authorization header. Token rotation is supported via a dedicated refresh endpoint, and the rotated token is persisted to the connection. Response format and parsing. Rows can be filtered server-side before insertion (for example, to drop test or null rows). After parsing, a _date column equal to the request date is set on every row, so _date is always present in the cached schema regardless of what the endpoint returns. Pagination. CSV responses return the first page only — paginated results are not accumulated for CSV. Refresh modes.

​
Field type mapping

Custom URL Import has no fixed metric or breakdown catalog. The available fields are the columns present in the fetched response, inferred at import time by sampling the parsed rows. New columns that appear as the data changes are added automatically (extraColumnsLogic = 'add').
How to read the columnsThe Data type column uses the platform’s ten-value vocabulary: Number, Currency, Percentage, Ratio, Duration, Date, Text, URL, Array, Boolean. The Field role column indicates whether the field serves as a metric or a breakdown.
_id-like columns are always textA column whose name ends in _id, contains _id_, matches id followed by digits, or is exactly id is exposed as a text field even when its values look numeric, so identifiers behave as breakdowns rather than metrics.
Each parsed column maps to an Adriel field type as follows.
Field-type inference is the shared file-import pathColumn types are inferred by sampling the parsed rows (getColumnTypesFromCsvRows / detectTypeFromString), the same inference used for other file-import connectors. The exact behavior on Custom URL Import rows is flagged for reviewer confirmation.

​
Adriel-added fields

Adriel injects one field on top of the fetched columns.

​
Aggregation defaults

When no explicit aggregation is set on a field, the default follows the inferred column type: numeric columns aggregate as a sum, and text columns act as breakdowns that group the rows. Date-parameterized mode stores the request date in _date for date-level grouping. (Inferred from the shared file-import aggregation path — flagged for reviewer confirmation.)

​
Limitations

Based on the current connector:
  • Daily cadence only. The cache refreshes on a fixed twice-daily schedule; sub-daily refresh is not supported.
  • No manual refresh. User-triggered refresh is disabled (allowUserManualRefresh = false).
  • No live querying. All dashboard queries are answered from the cache; the endpoint is called only during scheduled refresh cycles.
  • Data-size truncation. The connector applies no payload limits of its own, but large responses may be truncated by the source API’s own rate limits, payload caps, or page-size restrictions.
  • CSV pagination is single-page. CSV response mode does not accumulate paginated results. Endpoints that return CSV across multiple pages need a JSON wrapper or a single-page response.
  • JSON responses must be arrays. After responseBodyPath extraction, a JSON response that does not resolve to an array is rejected.
  • dateFormat casing. The format string must use lowercase yyyy and dd; uppercase YYYY or DD is rejected at validation.
  • OAuth2 authorization-code flow is not built in. The initial token must be obtained outside the platform and supplied. Token-rotation auth is supported via a dedicated refresh endpoint, but the first token still comes from outside.
  • Unique keys required for date-parameterized mode. Upsert needs unique-key columns to decide between insert and update; without them, use daily full-list mode.

​
API references

Custom URL Import has no dedicated vendor API — it calls the endpoint the customer supplies. Refer to the source API’s own documentation for request format, authentication, rate limits, and pagination. For platform-side setup, see the paired How to connect Custom URL Import.

​
See also