This page contains the process for sharing data with Kohort via Parquet files uploaded to an AWS S3 bucket.
Prerequisites
Before you start, you'll need:
A source table that contains validated, cleaned and mapped data ready for export as per the CLF data specification.
Kohort will provide the following details:
S3 bucket location
AWS access keys
Your org_id and dataset_id
Upload Path and Naming Strategy
The uploads must follow the following path and naming strategy:
<org_id>/<dataset_id>/clf/v3/YYYY_MM_DD_hh_mm / product_name*.parquet
Parameters:
<org_id>and<dataset_id>: Unique identifiers for the organisation in conjunction with a given dataset. This therefore allows for multiple datasets within a single organisation.clf: The Cohort-long-form specification to which the data aligns.v3: Major version number of the current spec.YYYY_MM_DD_hh_mm: The timestamp in the path is used to group the files for a given export.product_name*.parquet: The product_name can be set to your liking and the * denotes the automatically numbered files after they have been partitioned.
Examples:
Here’s an example of two valid file names and paths which were exported at 14:30:00 on the last day of January:
org_1234/source_dataset/clf/v3/2022_01_31_14_30 / best_product1.parquet
org_1234/source_dataset/clf/v3/2022_01_31_14_30 / best_product2.parquet
Note: Timestamp is in the file path not name.
File Creation Guidelines
To best leverage the benefits of this file format some important aspects need to be considered when creating parquet files:
The files should be partitioned into smaller files so that no single file is larger than 256MB.
Each column should have the appropriate data type. The column field names and types should be used as detailed in the cohort long form table below.
Choose the correct compression algorithm. We recommend using the snappy compression algorithm when generating parquet files. This algorithm is the most widely supported by different programming language implementations.
A short list of common ways to generate parquet files is listed below:
Data Source/Language | Platform/Package | Relevant Guides |
Google Cloud | BigQuery | Exporting to parquet ( note compression type) |
AWS | RedShift | Exporting to parquet |
Snowflake | SQL | |
Python | Pandas | |
Apache | Arrow | |
Java | SpringBoot |
Segment Guidelines
Limit to 250 segments to manage file transfer time and costs.
Roll up smaller segments (e.g., instead of 200 countries, use 40 plus 'Other').
For more details, see our Introduction to Segmentation article.
Update Frequency
Ideally the Kohort system would prefer daily updates; this allows the system to keep its reporting up to date and relevant (data uploads automatically trigger an event for file processing). The exact schedule can be discussed with your implementation manager.
During implementation you may of course send test files on an ad hoc basis and move to daily processing when all setup is complete and tested.
Note: There’s no need to delete old files as Kohort will automatically remove stale files.