Accuracy is one of the key attributes of a good forecast, and we believe it's important to be transparent about forecast accuracy. Headline accuracy metrics are available for any scenario on the Accuracy Metrics report page.

Knowing your forecast accuracy gives you:

Note: forecast accuracy is a combination of accurate modelling and accurate planning of the product roadmap. If the expected effect of UA campaigns and feature changes is not added to the forecast using the Milestones feature, the accuracy will appear worse than it is. An accurate baseline combined with an inaccurate roadmap will still produce an inaccurate forecast.

In this guide we will explain in detail how our headline accuracy metrics are calculated.

Across all industries that perform forecasting, the agreed gold standard in accuracy measures is Mean Absolute Percentage Error, or MAPE. The equation for MAPE is:

MAPE = ∑{( |actual - forecast| ) / |actual| } x 100 x (1/n)

A MAPE of 5% indicates that a typical forecast result is within 5% of the actual measure, in either direction.

In our accuracy reporting, MAPE is calculated by taking the following steps:

In cohort-based forecasting, the most important KPI to measure for accuracy is Lifetime Value (LTV). As LTV is the product of both retention and monetisation forecasts, an accurate LTV requires accuracy in the forecast as a whole.

We calculate MAPE for d90, d180 and d365 LTV, for forecasts over the previous 13 cohort weeks. If you don't yet have d365 actuals, the accuracy will show as ∅.

Read on for a more detailed explanation.

The age of the cohort at the time of forecast is important for cohort-based forecast accuracy, and informs our selection of forecast data points. LTV forecasts that are based on a single day of actuals would obviously not be accurate, but you shouldn't need weeks of actuals either.

For our forecast data points, we take the average LTV forecast at a weekly cohort granularity i.e. the average result for all cohorts with their installation date in the same calendar week. This "cohort week" standardises results across forecasts that might be run at either daily or weekly frequency, and backtests that run at weekly granularity.

In order for a cohort week to qualify as having 'baked' sufficiently, we require that the youngest cohort in the week has had at least 7 full days of actuals to contribute towards the forecast. Cohort ages thus range from from 8-14 days at the time of the forecast.

Most importantly, we take a snapshot of the forecast result when the cohort is old enough, so a cohort week from 90 days ago will be measured on how accurately it was forecasting at that time, rather than at present when more actuals are available.

How far into the forecast should we measure? A LTV at d7 (7 days since install) or d14 is likely to be very accurate, but not likely to represent the most commonly-used horizon for planning and decision making. We have chosen d90, d180 and d365 as our default cohort forecast horizon, to support a range of different activities and decisions reliant on LTV.

Each forecast data point thus represents the average d90, d180 or d365 LTV forecasts for a given cohort week.

This is compared to the average actual d90, d180 and d365 LTV of that same cohort week, once those values are available, to calculate the APE.

Lastly we must decide how many weeks of data points to include in our mean. Given that marketing teams also rely on LTV being sensitive to recent changes, the performance of relatively recent results is most telling. As such, we include only the most recent 13 qualifying cohort weeks in our MAPE calculation. This represents approximately 3 months of forecast results.

Bear in mind these will be the 13 cohort weeks that have both a forecast and actual LTV to compare, so "last 13 weeks" is relative to the most recent week that has actuals for the metric in question, and will therefore be less recent for longer forecast horizons. For d365 LTV, it is therefore quite possible that fewer than 13 weeks of data are available.

Top level metrics serve as a long-term indicator of calendar date modeling accuracy primarily used for budgeting and scenario planning accuracy calibration.

We calculate MAPE for New Users, Active Users and Revenue at a forecast horizon of 90 days (D90), for forecasts over the previous 365 days.

In a cohort forecasting model, calendar-based metrics are derived from the cohortized results by aggregating the forecast activity on a given calendar day across all cohorts. As such, any one metric contains a mixture of cohort ages and cohort forecast horizons.

Since any limitation on cohort age would change the calendar date result, we snapshot calendar date forecast results as they are at the time of forecast, with all relevant cohorts included.

These are aggregated by calendar week in the breakdown table (below the headline figure) only for convenience of comparison alongside the LTV figures; no weekly aggregation takes place in the calculation itself.

As top level metrics are of primary value to budgeting and scenario planning, accuracy for the coming quarter is considered most informative.

We have therefore chosen 90 calendar days into the forecast (D90) as our calendar date forecast horizon, for all top level metrics.

Forecast values are summed over the first 90 days of the forecast, and their totals compared to the actual totals for the same 90 days, once available. The MAPE therefore corresponds to the typical accuracy of a quarterly total e.g. total revenue in Q1.

As calendar date metrics include a mixed aggregation of both recent and older cohorts, more data points are needed to determine top level accuracy trends.

We therefore include all dates within the last 365 days, starting from the most recent calendar date for which we have both actual results and a prior D90 forecast snapshot.

When a new dataset is onboarded into the system, we perform an operation called "backtesting", in which the baseline scenario is backdated one week at a time, to produce a series of historical forecast snapshots. These represent what our system would have forecast with the data available at that time, using the current model settings.

These snapshots may then be compared to the existing actuals to produce an initial assessment of forecast accuracy given the current model settings. This process is used to fine-tune and optimise model settings to ensure your baseline scenario is configured correctly.

Warning: backtesting accuracy is typically based on the baseline forecast at each prior date, typically without any historical milestones in place. This means that backtested accuracy will always represent the upper bound of accuracy; by adding Milestones into your forward-looking forecast, the deviation from baseline due to product features and UA campaigns can be taken into account, which will increase accuracy going forward.

Backtesting is typically only performed on the baseline scenario, so when you first begin forecasting with our system, this is the only scenario that is likely to already have accuracy metrics available. Other scenarios will show no results on the accuracy metrics report page.

It should be noted that initial backtesting is typically only done for one year prior, so d365 metrics on backtested scenarios will not be reliable (as they are based on very few data points) until a few more weeks of forecasting have passed.

As time passes and forecasts are accumulated, and actuals eventually catch up, accuracy metrics on other scenarios will begin to appear. Bear in mind that without backtesting, it will take a minimum of 90 days of forecasting for the first accuracy metrics to begin showing on a scenario, and a further 90 days before these are based on sufficient data points to be considered significant.

To support further analysis of accuracy, we provide tables and charts of the underlying data points used in the process of calculation, further down on the Summary page. These may be interrogated and manipulated by using the Explore option on the Looker charts.

On the Analysis tab, we also provide our LTV Prediction Accuracy report. This detailed report allows for more intensive analysis of the relative LTV accuracy metric in particular, with additional options for forecast horizon, as well as accuracy methodology. More information can be found in the detailed guide.