Our data ingestion process automatically validates the format and content of the supplied data against our data specification, and for internal consistency. During the course of ingestion, errors and warnings will be produced if the data does not pass this validation.

Regardless of whether you pass your own Cohort Long Form (CLF) file to S3, or whether we are creating it directly from your supplied BigQuery data, the CLF parquet file passes through the following validation stages, with associated errors.

There are two message types: Errors and Warnings. An Error will prevent processing of the file until resolved. A Warning will allow the file to continue processing, but indicates there may be data accuracy issues in the resulting output.

The file will only move on to each stage if it passes the previous one, so for example if you receive an error at Row validation stage, you will know that File Validation passed.

Stage	Description	Error codes
1. File Ingestion	Any technical errors encountered during initial ingestion of the file will return as an Internal Server Error. The details of these errors are passed to the Ingestion team to resolve.	1000
2. File Validation	The CLF file is validated for issues with the file itself e.g. that the file is using the correct extension, the name is in the right format, and it contains the expected header columns.	2000
3. Row Validation	File data is then validated to ensure it conforms to row and field-level requirements e.g. checking for null dates or negative users.	3000
4. Dataset Validation	The dataset is then validated as a whole to identify higher level data errors e.g. gaps or missing data.	4000
5. Categorical Mapping	Errors encountered during the process of segmentation rollup aka Categorical Mapping will be handled by the team internally.	5000
6. Product Validation	Product Validation is performed as the final round of validation after data is rolled up per the segmenation mapping rules, prior to importing into the WebApp as Product Actuals.	6000

File Ingestion: 1000 Codes

Any technical errors encountered during initial ingestion of the file will return as an Internal Server Error. The details of these errors are passed to the Ingestion team to resolve.

Code	Type	Description	Advised Action
1001	Error	Internal Server Error `Internal Server Error. Our support team has been notified and will be in touch.`	This type of error would indicate a problem in the initial file ingestion itself e.g. an error internal to S3. The ingestion team has been made aware of the issue and is on it. If they cannot resolve the issue in good time, they will be in touch.

File Validation: 2000 Codes

The CLF file is validated for issues with the file itself e.g. that the file is using the correct extension, the name is in the right format, and it contains the expected header columns. If you are exporting the file to the S3 bucket, check for any issues in file generation. If we are generating the file for you from BigQuery, our team will address the issue.

Code	Type	Description	Advised Action
2001	Error	Invalid File Path Length `The file path provided does not match the expected pattern. It must match the pattern {org_id}/{dataset_id}/{file_spec}/{file_spec_version}/{partition_date}/{file_name}.`	Returned when the file path does not have the expected number of segments. Ensure your file path matches the requested pattern. Most commonly this error occurs when clients leave out the partition date or make it a part of the file name. org_id: your unique organisation ID provided to you during onboarding dataset_id: your unique dataset ID provided to you during onboarding file_spec: clf file_spec_version: the current major version e.g. v3 partition_date: timestamp in the format YYYY_MM_DD_hh_mm file_name: file name with no spaces
2002	Error	Blank Path Segments `Blank strings were found in the file path. Please check that your file path contains no blank strings, and matches the pattern {org_id}/{dataset_id}/{file_spec}/{file_spec_version}/YYYY_MM_DD_hh_mm/{file_name}.parquet`	As above, check that your file path matches the required format and contains no missing data between segment separators.
2003	Error	Invalid File Spec `The file specification is not supported. Currently supported specs are: clf. Please check that your file path matches the pattern{org_id}/{dataset_id}/{file_spec}/{file_spec_version}/YYYY_MM_DD_hh_mm/{file_name}.parquet`	Check that the ‘file_spec’ segment of the path contains a supported file spec type. The supported types are listed in the error message.
2004	Error	Invalid File Spec Version `The file specification version is not supported. Currently supported versions are: clf/v3. Please check that your file path matches the pattern {org_id}/{dataset_id}/{file_spec}/{file_spec_version}/YYYY_MM_DD_hh_mm/{file_name}.parquet`	Check that the ‘file_spec_version’ segment of the path contains a supported file spec version. The supported versions for each spec are listed in the error message.
2005	Error	Invalid Partition Date `The timestamp in the file path cannot be parsed. Please check that your file path contains a timestamp in the correct format, and matches the pattern {org_id}/{dataset_id}/{file_spec}/{file_spec_version}/YYYY_MM_DD_hh_mm/{file_name}.parquet`	Check that your partition date is present in the file path, and is in the format YYYY_MM_DD_hh_mm
2006	Error	Invalid File Extension `The file extension is not supported. Currently supported extensions are: parquet.`	Check that the file extension matches a supported format. The supported formats are listed in the error message.
2007	Error	File Size Too Large `The file size of {fmt_size} MB exceeds the maximum allowed size of 256 MB. Consider splitting the file into multiple smaller files.`	File processing goes faster if larger files are partitioned into several smaller files so that they can be processed in parallel.
2008	Error	Required Fields Missing `The file does not contain the required columns: registration_date, activity_date,active_users`	Missing required columns will be listed. Please ensure these columns are included in your dataset. You may get this error if the column is present but named incorrectly. Note that column names are CASE SENSITIVE.
2009	Error	Revenue Fields Missing `The file does not contain a revenue column. At least one of the revenue fields gross_revenue_iap orgross_revenue_ads must be provided`	At least one gross revenue column is required to process data. If your data includes only net revenue, you may pass this in the gross revenue column (please note this to your integration manager). If you believe that you are passing a valid gross revenue column, check that the column name has the correct prefix, spelling and case.
2010	Error	Category Fields Missing `The file does not contain any categorical columns. Categories should be prefixed with category_<name>`	At least one category column is required for segmentation. If you believe you are passing a valid category column, check that the column name has the correct prefix, spelling and case.
2011	Error	Invalid Column Types `The file contained invalid column types:` `The column registration_date must be a date field.` `The column activity_date must be a date field.`	Columns require data in a specific format. All invalid formats will be listed.
2012	Error	File is Empty	File contains 0 bytes.

Row Validation: 3000 Codes

Row Validation errors relate to invalid values within fields, or invalid values within the context of a row. Check the data at source and resolve.

Code	Type	Description	Advised Action
3001	Error	Null Dates `The registration_date and/or activity_date column in the file contained null values. Date fields require a value in the format yyyy-mm-dd.`	All registration and activity dates require a value. Null date rows should be corrected or removed.
3002	Error	Invalid Activity Date `The file contained rows where the activity_date is earlier than the registration_date. Cohorts cannot have user activity prior to the cohort registration date.`	Activity dates must be equal to or greater than the registration date on the same row. Activity cannot predate registration. Invalid rows should be corrected or removed.
3003	Error	Revenue without Users `The file contained rows with zero active_users, but where a revenue value is greater than zero. Revenue values must be attributed to active users.`	Revenue figures should be supplied on rows with active users, as revenue is expected to be the result of user activity. Either correct the active users values on these rows, or ensure revenue is being allocated to the correct rows.
3004	Error	Negative users `The file contained rows with negative values in active_users. Active users must be greater than or equal to zero.`	Active users cannot be a negative value. Affected rows should be corrected, removed or set to zero as deemed appropriate.
3005	Error	Marketing Spend `The file contained rows where marketing_spend is provided on activity dates after the cohort registration date. Marketing spend can only be provided on rows where the registration_date is equal to the activity_date.`	Marketing spend is related to user acquisition and is therefore expected to be allocated to the cohort’s registration date row. This is the row where registration date is equal to activity date. Check the logic for marketing spend allocation and amend.
3006	Warning	Negative Revenue `The file contained rows where a revenue value is negative. Revenue must be greater than or equal to zero.`	Revenue is expected to be a positive value. This is classed as a warning as some negative rows will not cause the overall process to fail if total revenue per cohort and segment is positive, but these should be checked for correctness.
3007	Error	Null Users `The file contained rows where active_users has null values. Active users must be greater than or equal to zero.`	The active_users field requires a value on all rows. Null user rows should be corrected to zero, or removed.
3008	Error	Revenue not cumulative `The file contained rows where an newer revenue_dx value is less than older revenue_dx value.`	Aggregate file only: revenue values should be supplied as cumulative i.e. revenue_d30 should be the sum of all revenue for those 30 days, and therefore greater than the corresponding revenue_d7 value.
3009	Error	Invalid Category Name `Category name '{field}' contains invalid characters. Category names can contain only [A-Za-z0-9_] characters.`	Category names should follow database column naming rules, containing only simple alphanumeric characters and the underscore character.
3010	Error	Invalid Category Value `Category values contain invalid characters. Category values cannot contain '\|' character.`	Category values cannot contain the '\|' (pipe) character as this is reserved character used in segment display. Replace pipes with underscores or dashes.

Dataset Validation: 4000 Codes

Dataset Validation errors relate to issues encountered when looking at the dataset as a whole across all rows, or in comparison with the previously uploaded file. Many of these are classed as warnings as they are indicators that something may have changed unexpectedly, and are often encountered during initial data setup, while you may be in the process of resolving row-level issues. If the message is not expected, check the data at source and resolve.

Code	Type	Description	Advised Action
4001	Error	Max Registration/Activity Date `The latest registration/activity date ({max_timestamp}) in this file is earlier than that in the previous file ({last_timestamp}). Later files are expected to contain activity up to the same or later dates.`	Each subsequent file is expected to contain either the same or more recent data than the one before it. This error occurs if the latest file generation or the source data itself has changed to exclude recent dates.
4002	Error	Duplicate Batch Entries `The dataset contains duplicate rows. A row entry must be unique per registration date, activity date and segment combination.`	Active user data should be aggregated by registration date, activity date, and distinct category column values. Check aggregation logic to ensure no duplicate rows exist.
4003	Warning	Missing Rows `The file contains {count} fewer rows than the previously uploaded file. Row count is expected to increase for subsequent files. Please investigate source data for missing entries.`	Similar to the above error for dates, each subsequent file is expected to contain more rows than the previous one. This is classed as a warning as it may be encountered when correcting errors during initial data onboarding e.g. if invalid rows were removed or aggregation logic was corrected.
4004	Warning	New Segments `The file contained data for unexpected new segments, which are not currently mapped and will be excluded. Please contact us to advise if the segment mapping should be updated to include them.` `New segments found: "Paid\|BS\|AppLovin\|Android", ...`	Segment or Category values are agreed and mapped as part of the onboarding process. If new category values are introduced at a later date, depending on the category mapping rules applied, these may need to be mapped explicitly to be included in the dataset. If the new segment value is invalid, please check your source data. If the value is valid, please advise us how it should be mapped.
4005	Warning	Missing Segments `The file was missing segments present in the previous file. Please check source data for issues, or advise if any segments should now be excluded.` `Segments not found: "Paid\|BS\|AppLovin\|Android", ...`	Segment or Category values are not expected to be removed from the dataset once it is established. Check your source data for issues. If you wish to change the segmentation rules for your product, please advise us and we can discuss the best approach for doing so.
4006	Warning	Marketing Spend Variance `This file contained a 60% variance in marketing spend from the previous file, which is above the expected day-on-day variance. Please investigate source data or feed for any issues.`	Marketing spend variance is checked from one file to the next as an indicator that file content may have changed beyond expected thresholds. If this warning is received it is recommended that you check your source data for any major issues.
4007	Warning	Data Gaps `The file is missing consecutive data points for 8 day(s) in the registration date column. Days that have no user registration should be still included with zero active users.`	The file is expected to contain registration data for every consecutive date. If you did have a period of no user registration during early launch, it is recommended to include missing rows with zero active users. If this gap is unexpected, check your source data for issues.
4008	Warning	Unchanged Row Count `The number of rows found in the file is equal to that found in the previous file. Please investigate data feed for missing entries.`	The number of rows in the file is the same as the previously received file. You may have sent the same file twice, or your source data may not be updating. If unexpected, check your source data or file generation logic for issues.

Categorical Mapping: 5000 Codes

This error code range is currently reserved for any errors encountered during segmentation mapping setup. These errors are internal and received by the onboarding team. If further input or information is required to resolve any of these issues, the team will be in contact.

Product Validation: 6000 Codes

Product Validation is performed as the final round of validation after data is rolled up per the segmentation mapping rules, prior to importing into the WebApp as Product Actuals.

Code	Type	Description	Advised Action
6001	Error	No Data `The file contained no rows. Please investigate source data or feed for issues`	If this occurs after file and row validation has passed, it is likely an internal issue and will be resolved by the ingestion team.
6002	Error	Active Users exceed Cohort Size `The file contained entries where the value of active users is greater than the cohort size (number of active users at the registration date). User activity for a cohort should not exceed the number of registered users.`	The cohort size is determined by the number of users on the row where registration date = activity date. The number of active users on subsequent rows for the same registration date cannot exceed this number. Check source data and resolve any issues where user activity may be attributed incorrectly, or registered user count may be too low.
6003	Warning	No Cohort Size `The file contains activity data where there is no registration date entry for the cohort. Cohort size cannot be derived for these cohorts and they will be excluded`.	For a given cohort, the registration date row is missing i.e. the row where registration date = activity date. This row is required to set the size of the cohort, which is critical for retention calculation. This is a warning as the product data will process, however these cohorts will be excluded from forecasting. As such, failure to remove these cohorts from the Actuals may result in a disjoint between actuals and forecast, especially in Daily Active Users.

Data Ingestion Error Codes