Charity Engine Connector (Technical Documentation)

Protocol: CharityEngine SOAP API (/v/3/Soap, Get{Object} operations).

Sync style: Incremental by date fields per table; the connector windows large ranges into 24-hour slices and backstops the start time by 2 minutes to avoid missing edge records. Enforces a lower bound of 2010-01-01T00:00:00.

Pagination: Offset/limit loop per slice; limit = 2000 until a page returns fewer than 2000 results.

Rate Limiting: The API throttle is respected with adaptive backoff.

Schema handling: Each table has a fixed column map. JSON-shaped substructures are serialized as JSON strings; empty arrays are normalized to NULL.

Retries & errors: Requests are retried on connection issues (with sleep). Timeouts surface a clear message (e.g., check IP allowlist). Non-200 HTTP codes are mapped to specific exceptions, but CharityEngine often returns HTTP 200 with an error payload, which is parsed and handled.

Simple fields: Extracted directly from attribute or element paths (e.g., ["@attributes","Id"] → id).

Boolean/timestamp typing: Columns are typed (boolean/timestamp/string) before records are written; arrays that resolve empty are set to NULL.

JSON fields: Certain nested structures (e.g., “Payment…”, “Attribution…”, address composites) are emitted as JSON strings when modeled as type: json in the schema.

Discovery: For each selected stream, the connector writes the schema (columns) and key properties (indexes) to the singer catalog.

Unique/Delete keys: The connector writes unique_keys and (when applicable) delete_keys via writeMeta so downstream targets can deduplicate and apply deletes.

Incremental sync design:
Each stream uses an incremental date field (such as DateCreated or DateModified). The connector reads the bookmark from the last run, moves the start time back 2 minutes to cover edge records, and requests data in 24-hour windows until caught up.

Earliest allowable start date:
Queries will not go earlier than 2010-01-01 00:00:00, regardless of bookmark state.

24-hour window slicing:
Large backfills are broken into 1-day windows to prevent overly large requests and improve stability. This can cause slower initial syncs for high-volume datasets.

Pagination:
Each window is paginated with a 2000-record limit per page. The tap continues paging until fewer than 2000 records are returned.

Retry and error handling:
The tap retries transient connection errors and times out gracefully. Persistent timeout errors include clear messaging (e.g., “Check if the IP address is allowlisted”). Although CharityEngine sometimes returns HTTP 200 responses containing error messages, the connector inspects the payload and converts these into explicit errors.

Delete support:
Hard deletes are only supported for select tables—specifically:

Nested structures:
Complex nested fields (like primary addresses, organization details, phone numbers, and JSON arrays) are serialized as JSON strings. This ensures consistent schema output but requires downstream flattening or transformation for analytics.

Variable response nodes:
Some API responses return result arrays under unexpected node names (Conversions, Registrations, MessageReturns, etc.). The connector handles known cases but may require code updates if new objects or naming conventions are introduced in the API.

Schema drift:
CharityEngine’s SOAP API may occasionally add new fields or rename elements. Since the connector uses a fixed schema map, new fields will not appear until the connector mapping is updated.

Performance and rate limits:
Pagination, retry logic, and 24-hour windowing help throttle requests, but extensive historical syncs can still be slow. CharityEngine rate limits are handled gracefully but may extend total sync time.

Environment sensitivity:
Some CharityEngine environments (sandbox vs. production) handle acknowledgments differently. Test or sandbox runs might not mark records as processed upstream.

Timezone and date normalization:
All timestamps are processed as provided by the API. If data mixes UTC and local times or omits offsets, you may observe minor timestamp inconsistencies in downstream data.