Skip to main content

​
Introduction

File from SFTP is a file/storage data source. It pulls report files from a remote SFTP server, parses their contents, and stores the rows in a per-data-source cache table so widgets can aggregate the imported data alongside platform-native sources. It is intended for partners or internal systems that already publish scheduled exports to an SFTP endpoint — for example, advertising agencies that deliver overnight performance dumps, or ERP systems that push daily reconciliation files. The connector authenticates over SSH using key-based authentication and points at a single directory on the SFTP server. Customers can point at their own SFTP host or use a platform-managed endpoint provisioned per data source. As a file/storage connector, the available fields are not a fixed catalog — they are derived from the columns of the parsed files. Numeric columns become metrics and text columns become breakdowns, so no metric or breakdown schema is predefined. To connect this data source, see How to connect File from SFTP to Adriel.

​
Data refresh strategy

File from SFTP is a simple-cache connector. It does not query the SFTP server on each widget load — instead, a scheduled job re-imports every file and replaces the data-source cache, and widget loads read from that cache. Refresh schedule (UTC). The cache refreshes on a fixed twice-daily schedule at 10:00 and 14:00 UTC (0 10,14 * * *). The schedule is hardcoded and cannot be customized per data source. Replace, not incremental. On each refresh the connector opens an SFTP channel, lists the configured directory, downloads each matching file, parses it, and rewrites the cache-table contents. There is no incremental fetch — every refresh re-imports all matching files, so results reflect the latest published files as of the last scheduled run. Change detection. File changes between refreshes are detected from each directory entry’s modification time (mtime).

​
Architecture levels

SFTP server → upload folder (directory) → files. The connection resolves as follows:
  1. SFTP server — the customer’s SFTP host, or a platform-managed endpoint used automatically when no host is configured.
  2. Upload folder — one directory on the server. The directory is listed flat; sub-directories are not traversed.
  3. Files — every file at the top level of the folder whose name and extension match the configured filters is imported.
One Adriel data source binds to one upload folder. All matching files in that folder are concatenated into the cache table, so their column structures must align.

​
Date range limits

There is no platform-imposed date range limit. Date bounds come from the contents of the imported files, combined with the date column mapped in Blend Data settings. Files without a date column are treated as a single snapshot set on the dashboard.

​
Import model

Imports are driven by a host/path/file-pattern pipeline rather than a per-load generated query. On each scheduled refresh:
  1. Connect. The connector opens a fresh SSH session to the resolved host (customer host or platform-managed endpoint) on port 22, authenticating with an SSH private key.
  2. List. It lists the configured upload folder flat (no recursion).
  3. Match. Each file is kept only if its extension matches the configured file type and its name matches the file-name filter. The file-name filter is a comma-separated set of name fragments; an empty value disables name filtering. When the file type is unset, the extension filter is effectively disabled and all files are considered.
  4. Download and parse. Matching files are downloaded and parsed into rows. CSV files are read as a stream and parsed row by row with the configured delimiter; Excel (XLSX) files are converted to CSV form before caching.
  5. Cache. Parsed rows from all matching files are concatenated and written to the per-data-source Postgres cache table. The first row of each file is treated as the header.
At widget load time, queries run against the Postgres cache table — not against the live SFTP server. A fresh SSH session is opened per file operation (no connection reuse or keepalive).
Column structures must align across filesAll matching files in the folder are concatenated into one cache table, so their columns must line up. Files with mismatched headers can produce missing or misaligned values.
Streamed CSV, buffered XLSXCSV files are parsed as a stream rather than buffered, so very large files do not exhaust memory. XLSX files are buffered and converted to CSV form during ingestion, because the format itself is not streamable.

​
Filters

Widget filters on File from SFTP data sources run against the Postgres cache table using the standard Postgres filter operator set. Because queries hit the cache rather than the SFTP server, filtering does not re-list or re-download files. The all breakdown sentinel is stripped from queries before they reach Postgres, so requesting an “all” breakdown does not raise a database error.

​
Field type mapping

File from SFTP has no fixed metric or breakdown catalog. The connector is free format: the available fields are the columns discovered from the parsed files, resolved from the per-data-source cache table after the first successful import. Customer-supplied columns are typed at ingestion.
How to read the columnsThe Data type column uses the platform’s ten-value vocabulary: Number, Currency, Percentage, Ratio, Duration, Date, Text, URL, Array, Boolean. The Field role column indicates whether the field can serve as a metric, a breakdown, or both.
Parsed columns map to Adriel field types as follows. The numeric-versus-text split (numeric columns become metrics, text columns become breakdowns) is confirmed by the connector. The finer per-value classification that decides whether a given parsed column is treated as Number, Date, or Text is applied at ingestion and is not otherwise configurable per field.

​
Adriel-added fields

The connector does not expose synthetic metrics on top of the imported columns. Instead, several internal cache-bookkeeping columns are excluded from the available-fields list shown to customers:

​
Aggregation defaults

Numeric columns are treated as metrics and aggregate as expected for a metric; text columns act as breakdowns. Date-based breakdown is not supported by this connector, so imported data cannot be grouped by a native date-part breakdown even when a date column is present. Where a date column is mapped in Blend Data settings, it drives date-range filtering rather than a date breakdown.

​
Limitations

Based on the current connector:
  • 50,000-row cap per query. Row queries against the cache are hardcapped at 50,000 rows.
  • Twice-daily refresh only. The cache refreshes at 10:00 and 14:00 UTC on a hardcoded schedule; the cadence cannot be customized, and new files land in Adriel only after the next scheduled run.
  • Full re-import each refresh. There is no incremental sync — every refresh re-imports all matching files and replaces the cache.
  • One folder per data source. The connection binds to a single directory, listed flat. Sub-directories are not traversed. To import from a second folder, create a second data source.
  • Files must share a column structure. All matching files are concatenated; mismatched headers cause missing or misaligned values.
  • CSV is the supported file type in the UI. Only CSV can be selected when creating a data source. Excel (XLSX) is converted to CSV internally when set, and the JSON and ZIP file types are not supported for this connector.
  • Key-based authentication only. Password authentication is not supported; the SFTP server admin must enable SSH key authentication for the dedicated user. Port 22 is used.
  • No date breakdown. Imported data cannot be grouped by a native date-part breakdown.
  • Empty folder is a healthy connection. A directory listing that returns zero files is treated as a valid connection; the absence of data surfaces only at refresh time when zero rows are imported.
  • No connection reuse. A fresh SSH session is opened per file operation (no keepalive), so high-volume folders open and close many short-lived sessions per refresh.
  • Disconnect when no file matches. When no file in the folder matches the configured file type and name filter, the data source can surface a no-matching-file condition (NO_FILE_MATCHING_SETTING) as a disconnect. (Behavior flagged for tech-reviewer confirmation — see reviewer notes.)

​
API references

File from SFTP imports over the SSH File Transfer Protocol; there is no vendor product API. The following are the relevant protocol references.

​
See also