Contents
What are Milestones?
Milestones are a tool for adding the impact of future Product changes and UA plans to your forecasts.
A milestone describes an event or change in the roadmap that is predicted to have a step change impact on either of Spend, New Users, Retention or Monetisation forecasts, for all segments or specific segments.
There are 2 types of milestones that can be added, being a Relative and Absolute milestone with summary below
Relative Milestone | Absolute Milestone |
Increase by provided percentage at a certain time e.g.
| Specify an explicit value to use at certain time e.g.
|
Milestones are applied as a relative increase to the baseline forecast. This relative impact allows you to review results and adjust assumptions over time, as new data comes in and the baseline forecast changes. | Absolute milestones override the baseline forecast. |
The effects of milestones are compounded, so 15% + 15% = 32% instead of 30%. Use the Preview chart to easily see the impact of milestones together | Global absolute milestones will give all segments that same value. This should rarely be used |
Relative milestones can be used after Absolute milestones, but not before. e.g. you can set a starting monetisation value and lift that 30% over the next year but not the other way around | Latest by overlapping is given priority. e.g. a milestone from July to hit $1 by the end of the year would be overridden by a milestone that starts in November to hit $1.2 by May the following year |
Adding a New Milestone
To create a new milestone, click on + Add Milestone at the top right of the page.
Milestone Details
Enter the following details to create a milestone:
Status: This allows you to turn an milestone off from the list if it has been de-prioritised and gone back onto the backlog
Start Date: the date on which the feature goes live. This defaults to the first day of the forecast
Segments: select Global to apply the milestone to all segments, or select specific segments e.g. type in “paid” in the search and select all paid segments for a UA campaign
Optional Name and Description: Use name for a short description of the milestone and the description field to give more context for why the values in the milestone may have been chosen
Choose the Pillar (New Users, Retention, Monetisation, Spend) that you can to add. You can add multiple pillar adjustments on the same milestone. The availability of Spend and New User milestones will depend on whether you are in Target New Users or Target Spend User Acquisition mode
New User, Spend and Monetisation Settings
Relative Milestone
Adjust baseline by: the positive or negative percentage that represents the change expected on top of the baseline forecast over the duration of the milestone
Absolute Milestones:
Target Value: the exact value will be achieved at the end of the milestone duration
All Milestones:
Duration: how long it will take for the milestone to reach this full effect e.g. a planned feature is expected to increase monetisation up to 5%, over the course of 30 days.
Monthly is a special-case duration that will apply the percentage adjustment as a month-on-month basis. e.g. Spend has stopped and organic New Users are declining at a rate of 2% per month
Curve Shape: the shape that most closely describes the growth or decline behaviour (disabled if duration is ‘immediate’ or ‘monthly’).
Logarithmic is a growth curve with fast initial uptake followed by gradual decline. This is best used where diminishing returns are expected as the milestone continues.
Parabolic is a growth curve with better-than-linear growth initially, and a leveling-out close to the end of the campaign.
Linear is the default curve and provides evenly-paced growth or decline over the duration.
Early sigmoid is an s-curve with an inflection point earlier in the duration, best used for milestones that might take a short duration to start building up, then experience diminishing returns.
Late sigmoid is an s-curve with an inflection point later in the duration, best used for milestones that might take longer to build momentum, but then grow rapidly once they have.
Top Tip! If you expect that a milestone will show an uplift followed by a decline (e.g. a temporary promotional listing on the app store increasing organic users for a period), add one milestone with a positive adjustment, and another that starts after that one ends, with a negative adjustment.
Retention Settings
If the milestone is expected to have an effect on user retention (e.g. improvements, level tuning, reward ads), enter the adjustment in the Retention section. You can enter adjustments for one or more dx points, if the change is expected to have different effect sizes at different cohort ages. See Understanding Retention Adjustments below for more information.
Apply to: select whether to apply the adjustment to:
All cohorts: change will affect both new and existing users e.g. new end-game prompts
New cohorts: change only applies to new users from the milestone start date e.g. improved new-user tutorial
Old cohorts: change only applies to pre-existing users e.g. reactivation campaign
Adjust baseline by: enter the relative % increase expected at the target dx point.
Note that this is the relative increase on retention e.g. a 5% increase on a 10% d30 retention will result in a 10.5% retention, not 15%.
Days since install: select the target dx point that corresponds to the adjustment.
Milestone Tabs
The milestones page is split into tabs to allow focusing on a specific pillar and seeing a preview of the results
Overview
See all milestones and their settings sorted by start date and segment. Click into any adjustments for a quick change.
Upcoming and In-Progress milestones are displayed by default. Update the Status filters if you want to see Disabled and Past milestones
Milestone Previews
On all the pillar tabs also provide a Preview that updates live with any changes. Open this from the top right Preview button
The preview shows the effect of milestone adjustments based on the filter criteria on the current forecast results (as per the last scenario run). This will update in real time as you edit your milestones.
Filtering of segments is important for what you want to see
Global: when selected alone, shows effect of all milestones with global segment only
Specific segment: when selected alone, shows all milestones whose explicit segment selection includes this segment and its affect on the selected segment, excluding effects from Global milestones.
Global + Specific segment: when selected together, this will return all milestones that have their segment selection set to either Global or the selected segment. The preview will show the combined effect of all selected milestones on the specific segment only.
Note! You will still need to run the scenario to see the effect of your current milestone settings on your scenario and product reports.
The Absolute tab shows the absolute effect of the milestones after multiplying by the baseline forecast or forecast override; use this tab to check if the magnitude of the effect and resulting values are as you expected. If not, adjust your percentage value below, and the preview updates automatically.
One year of actuals data is included for comparison, in grey.
For Spend, New Users and Monetisation, the baseline forecast in blue is also included
Change the granularity on the right to see the monthly aggregated values
The Relative tab shows the resulting multiplier from both absolute and relative milestones that has been applied to the baseline forecast; use this tab to examine the behaviour of overlapping and subsequent roadmap effects and check your workings e.g. the image below shows the global effect of a 30% Spend early ramp over a year followed by a 10% decline from the peak over 6 months
Tip: You can click on any point along the preview curve to add a milestone at that date.
Understanding Retention Adjustments
While Spend, New Users and Monetisation adjustments are relatively straightforward adjustments on a calendar date basis, the retention multipliers are applied at a cohort level, and deserve a deeper dive.
Retention Curve smoothing
A retention adjustment of 5% at d60 adjusts the retention curve for each affected cohort as follows:
Curve from d0 to d60 is adjusted so that it passes through a point that is 5% (relative) higher than before i.e. if d60 retention was 10% it will now be 10.5%, not 15%.
Retention curve from d60 onwards is lifted by 5% relative without changing shape, assuming that the cohort will still fall off at the same baseline rate thereafter.
This slightly-exaggerated graph shows the principle in action, using a larger adjustment and higher retention to illustrate the effect more clearly:
Subsequent adjustments are treated as new target points and the curve is adjusted to also pass through the new target e.g. here a second adjustment predicts a smaller uplift at d180 than at d60:
To restrict the retention uplift effect to specific dx points only, you can set a 0% target at the point where you expect the uplift to no longer apply e.g. here the adjustment effect peaks at d60 and reverts back to baseline from d180:
New Cohorts
By setting an adjustment for New Cohorts only, on the Relative tab of the retention preview, you can see exactly how the retention curves for all new cohorts after the milestone start date are being multiplied.
Days since install | Adjustment |
1 | 0.17% |
7 | 1.17% |
14 | 2.23% |
30 | 5% |
60 | 5% |
In this example you can clearly see all the later dx curves at 5%.
By comparison, going to the absolute tab and zooming in on the milestone date, and hiding all but d1-d60, we can see just how small the absolute result is when applied to the actual retention curve. Here you can see an almost imperceptable uplift at d1, growing to maximum amplitude at d30, and showing the same amplitude at d60 (gradient seen here is an artifact of zooming in).
Old Cohorts
If we switch the adjustment to Old Cohorts only (e.g. in the case of a reactivation campaign), we can now examine the effect on cohorts that installed before the milestone date e.g. were already playing when the feature launched.
Note! because this chart is by cohort date, there is no effect on or after the milestone date, and the multiplier goes to zero at that point. To view effects on a calendar date basis, you will need to run the scenario and check the Retention Trends report in calendar view.
As with the New Cohorts, we see the expected smaller multiplier at lower dx points, rising to maximum effect on the selected dx point (d30 in this case), and the same for points thereafter.
Additionally, there is a reduced effect size for older cohorts, as a greater portion of those cohort's activity will already be within the actuals range at the time of the milestone's occurance. Cohorts that reach the specified dx point before the milestone start date are thus unaffected. This creates the variously-sized triangular shapes seen in the diagram above.
And finally, we take into account that for users who installed the day before the feature release, the retention on the milestone date is already determined before they start playing; they only experience the impact of the change after they already made the decision to play, and their activity is already captured. So old cohort effects will stop just short of the full adjustment value (which is expected to occur on the release date), and do not show a d1 adjustment effect.
On the absolute view shown above, you can see that the Old Cohort portion of the adjustment lifts up prior cohorts and then stops at the milestone date. Be cautious as this can look like a 'drop' in retention (rather than a rise that stops) in more complex graphs like the one below:
All Cohorts
Finally, by setting the adjustment to All Cohorts, we see that this is simply a combination of the Old and New cohort adjustment effects.
In-Progress Milestone Calculations
Once the milestone start date has passed into the Actuals data range, the milestone is considered In Progress until the milestone duration has passed, after which it is considered Past and no longer applied.
During the In-Progress phase, the portion of the milestone uplift that falls within the actuals range is assumed to have occured as planned, and that a corresponding uplift or drop will have been seen in the recent actuals, causing the baseline forecast itself to change. As such, only the remaining uplift or decline % is applied for an in-progress milestone.
Here is a simple example of a 10%, 14-day uplift, when the milestone date is the first day of the forecast:
Fast-forward one day, and we assume that the first day's uplift (0.71%) has occurred as expected, and therefore the resulting baseline users must be 0.71% higher. The system calculates the multiplier required to lift the remaining days to the original 10%, with the following result (100 x 1.0071 x 1.0922 = 110):
For retention milestones, there is no ‘duration’ to the adjustment. Instead, a retention cooling period is what determines a similar gradual reduction of the milestone effect. This cooling period is twice roughly 60 days. Once this period has passed, it is expected that the full effect should be present within the baseline forecast, and there will be no further adjustment applied.
Frequently Asked Questions
In the retention preview, why are some dx lines shorter than others?
Later dx points will always have an earlier final cohort i.e. last cohort that reaches that age within the forecast range, thus higher dx point lines will cut off at earlier points before the end of the forecast.
Some may not be included in the milestone at all, as can be seen with d1095 in the New Cohorts example below, as the forecast range is less than 3 years.
Why does my retention drop when I didn't set a negative adjustment?
"Old Cohorts" adjustments, which raise up cohorts that installed prior to the milestone date, can look like a dip when viewed in combination with other adjustments, as in the example below. This is not a drop, but an uplift that does not impact cohorts on or after the milestone start date, and therefore stops.