Deployer Connector (Technical Documentation)

Overview

The Deployer connector synchronizes subscriber-level list data from a Deployer account into your data platform.

It pulls from a single logical table:

deployer — one row per subscriber in the configured list group, with status, subscription dates, engagement timestamps, mobile/SMS info, and various metadata. It also includes dynamic custom fields defined on the list.

The connector is read-only and does not modify data in Deployer.

Configuration & Authentication

The connector uses the Deployer XML API over HTTPS.

Required settings

username – Deployer API username.
token – Deployer API user token.
service_url – Full URL to the XML API endpoint.
list_id – List identifier used when fetching custom fields.
list_group_id – List group identifier used when fetching subscribers.
daysToFetchFromToday – Number of days in the past to include in each sync window (relative to “today”).

These values are injected at runtime and used by all API calls (test, discovery, and sync).

Sync Behavior

Date window

Each sync is window-based, not bookmark-based:

Start date: today − daysToFetchFromToday
End date: today

Filter applied:

<filter>
  <last_updated>
    <gt>{start_date} 00:00:01 +000</gt>
    <lt>{end_date}   23:59:59 +000</lt>
  </last_updated>
</filter>

This means every run re-queries all subscribers whose last_updated falls inside that window. There is no persistent per-subscriber high watermark; you control overlap vs. cost with daysToFetchFromToday.

Pagination

For the deployer stream the connector:

Starts with:
- offset = 0
- records_remaining = RECORD_MAXIMUM (a very high ceiling, 99,999,999).
Chooses a limit for each request:
- limit = min(DEFAULT_FETCH_SIZE, records_remaining)
- DEFAULT_FETCH_SIZE = 100
Calls GetSubscribers with the current limit, offset, and field list.
If zero records are returned, it stops.
Otherwise:
- Writes all records.
- Decreases records_remaining by limit.
- Increases offset by limit.
- Repeats.

In practice, the loop terminates when the API sends the last partial page.

Custom fields

Custom fields are discovered and included both in schema and data:

On discover:
- The connector calls GetCustomFields for the configured list_id.
- For each returned field (fieldid), it adds a column named cf_{fieldid} to the schema as text.
On tap:
- The same GetCustomFields call is made.
- The requested field list passed to GetSubscribers includes all base columns plus each cf_{fieldid}.

Any object/array values that do appear are JSON-encoded before writing records so that every column remains a scalar value.

Deletes

The connector sets subscriberid as the key field and exposes it as a delete key in its metadata.
However, it does not call any Deployer delete endpoint or emit explicit delete records.
In practice, this behaves like an upsert-only feed keyed on subscriberid. If Deployer removes a subscriber permanently, that removal will not be reflected automatically.

API Endpoints & HTTP Methods

All Deployer API interactions are performed via HTTP POST with an XML body to the configured service_url. The connector does not use GET, PUT, or DELETE.

1. GetSubscribers (main data endpoint)

HTTP method: POST
URL: <service_url> (configured)

Request body (simplified):

<xmlrequest>
  <username>{username}</username>
  <usertoken>{token}</usertoken>
  <requesttype>subscribers</requesttype>
  <requestmethod>GetSubscribers</requestmethod>
  <details>
    <listgroupid>{list_group_id}</listgroupid>
    <limit>{limit}</limit>
    <offset>{offset}</offset>
    <format>json</format>
    <fields>
      <field>subscriberid</field>
      <field>listid</field>
      ...
      <field>last_updated</field>
      <field>cf_123</field>
      <field>cf_456</field>
      ...
    </fields>
    <filter>
      <last_updated>
        <gt>{start_date} 00:00:01 +000</gt>
        <lt>{end_date}   23:59:59 +000</lt>
      </last_updated>
    </filter>
  </details>
</xmlrequest>

Response:
- Typically JSON (because <format>json</format>), but the connector checks:
  - If it starts with <?xml, it parses as XML.
  - Otherwise it decodes JSON and expects data under data.

2. GetCustomFields (schema discovery & runtime field list)

Used in both test(), discover(), and tap().

HTTP method: POST
URL: <service_url>

Request body (simplified):

<xmlrequest>
  <username>{username}</username>
  <usertoken>{token}</usertoken>
  <requesttype>lists</requesttype>
  <requestmethod>GetCustomFields</requestmethod>
  <details>
    <searchinfo>
      <List>{list_id}</List>
    </searchinfo>
  </details>
</xmlrequest>

Response:
- Parsed as XML or JSON (same detection logic).
- Custom field metadata expected under data->item; each item has a fieldid used to construct cf_{fieldid} columns.

3. Test connectivity

The test() method also posts an XML body with:

requesttype = lists
requestmethod = GetCustomFields
List = "Test"

and checks whether the returned status equals "SUCCESS" to decide if the configuration is valid.

Operational Considerations & Known Limitations

Sync window & incremental behavior

The connector does not maintain a per-record high-watermark.
Instead, it always pulls subscribers whose last_updated falls within the relative window:
- from today − daysToFetchFromToday (start-of-day)
- to today (end-of-day).
If a subscriber changes outside that window (for example, updated 40 days ago and daysToFetchFromToday = 30), that change will not be seen by the connector.

Recommendation: choose daysToFetchFromToday large enough to cover both your sync frequency and any expected back-dated changes in Deployer.

Upsert-only behavior (no deletes)

The subscriberid column is treated as the key for upserts.
The connector does not emit delete records if a subscriber is removed from Deployer.
To handle deletions, you may need:
- A separate reconciliation process, or
- Occasional full reloads of the deployer table.

Pagination and performance

Each API call fetches at most 100 subscribers.
For large lists and wide date windows, this can lead to many requests.
The connector enforces a high internal upper bound (RECORD_MAXIMUM) but in practice stops when the API returns an empty page.

Retry and backoff behavior

Each request to Deployer is wrapped in a retry loop:
- Total retry duration: up to 1,200 seconds (20 minutes).
- Base retry interval: 65 seconds.
- Backoff: the interval increases by 50% each retry (exponential backoff).
On network exceptions:
- The connector logs the failure, sleeps for the current interval, increments the retry count, and tries again until the duration cap is reached.
If it is still failing at the end of the retry period, it raises an “API unavailable” error.

Response parsing & error handling

The connector supports XML and JSON responses:
- If the payload starts with <?xml, it attempts to parse XML.
- If XML parsing fails, it tries to “tidy” the XML and parse again.
- Otherwise, it treats the response as JSON.
If the parsed object is empty or has status = FAILED:
- It raises a descriptive error (including errormessage when provided).

Custom fields & schema drift

Custom fields are discovered on each run. If new custom fields are added in Deployer:
- They appear as new cf_* columns automatically in both the schema and the records.
If a custom field is removed upstream, it will stop appearing in GetCustomFields, but existing warehouse columns and historical data will remain until you clean them up.

Single-list scope

Syncs are scoped to:
- A specific list group (list_group_id) for subscribers.
- A specific list (list_id) for custom fields.
If your Deployer account uses multiple lists or list groups, you may need:
- Multiple connector configurations, or
- A rotation strategy for list_id / list_group_id if you want to cover more than one.