All Collections
Setup & Project Configuration
Customising Effective Experiments
Experiments
Experiment Health Score Complete Breakdown
Experiment Health Score Complete Breakdown
Brendin du Plessis avatar
Written by Brendin du Plessis
Updated over a week ago

In this article we'll take a look at the different components that make up the experiment health score

Understanding Your Experiment Scorecard

Your experiment health score is designed to track critical parts of the experimentation program and to ensure that the most important principles are being adhered to. This ranges from having proper sources for your experiments, prioritising and hypothesising at the right stages and not tampering with experiments post-completion.

A Multi-Stage Scoring System

There are 3 stages in the typical experiment's lifecycle and that is; pre-live, live and post-live. The experiment health score is focused on 2 of these phases:

Pre-Live Scoring (Total: 60 points)

Here we track all your planning, prioritisation and critical info added to your experiment and the scoring goes as follows:

  • [Coming Soon] Objectives preset

    • Linked to objectives & targets: 10,

    • Retrospectively linked to objectives & targets: 5,

    • No linking (orphaned experiment): 0

  • Source of Experiment

    • Research: 10

    • Ideas: 5

    • Ad-hoc: 0

  • Prioritisation Completed

    • Fully done: 10,

    • Partially done: 5,

    • Not done at all: 0

  • Experiment Planning

    • Clear dates defined and went to plan: 10,

    • Clear dates defined but didn't follow that plan: 0,

  • Hypothesis Added

    • Added anytime during ideas & hypothesis stage: 10,

    • Added later once experiment has already been created: 5,

  • Process Adherence

    • Followed kanban flow and process: 10,

    • Skipped various statuses: 5,

    • Off course (kept going back and forth): 0

Post-Live Scoring (Total Score: 40 points)

This stage is all about capturing learnings & recommendations and ensuring winning experiments get implemented

  • Analysis (Insights & Learnings)

    • Present: 5,

    • Absent: 0

  • Next Steps (Recommendations)

    • Present: 5,

    • Absent: 0,

  • Experiment Completion

    • Ended on or days within expected end-date: 10,

    • Ended too early: 5,

    • Ended significantly earlier than expected end-date: 0

  • Rollouts

    • Set as Implemented or Not Implemented: 10,

    • Set as Pending or Not Sure: 5,

    • Not Set: 0,

  • Dependencies

    • Finished without unresolved blockers: 10,

    • Finished with unresolved blockers: 0

    • No dependencies: Skip (auto-10)

Thus, an experiment that perfectly followed process and best practice, will have the ability to reach a 100/100 score

Tracking High Alert Integrity Issues

Introducing the new integrity checker. With the sole focus of improving your overall experiment quality, the integrity checker will point out some critical issues that might compromise the integrity of your experiment as well as warning you not to tamper with finalised experiments

Pre-Emptive Warning

As you're about to edit or update a finalised experiment, a warning will pop up warning you that this might affect the integrity of your experiment.

Post-Completion Changes

Once an experiment is finalised, it's highly recommended not to make any further changes as to avoid corrupting the integrity of your data. Any changes to key fields made once an experiment is finalised, will result in point deductions:

  • Changed or altered hypothesis: -10,

  • Changed, removed or altered goals: -10,

  • Changed the already set outcomes: -10,

  • Removing previously added Insights or Recommendations: -10,

  • Changed, replaced or removed variation screenshots: -10

Highlighting Critical Integrity Issues

HARKing

To ensure the integrity of your experiment, your hypothesis should never change once the experiment goes live. It's for this reason that we will check for changes made to your hypothesis.

You will be warned that possible Hypothesising After Results Are Known has happened. This will mark your experiment with an integrity issue and subtract from your overall health score

Peeking

We all get excited when our experiment starts out really promising with an increase in conversion of 150%. However... It's never not how the experiment starts out, but how it finishes.

Your experiment needs to run long enough to reach a significance level to ensure that your results are no fluke. It's for this reason that we will check how early you finished your experiment

You will be warned that possible Result Peeking has happened. This is caused when an experiment stops significantly earlier than anticipated. Your experiment will be marked with an integrity issue and subtract from your overall health score

Instantly Spot Problematic Areas

Should any changes be made to the critical fields, as mentioned above, not only will the experiment be deducted but you'll also be able to see the field changes as visualised on your Experiment Timeline. This will help to track, train and reinforce best practices throughout your team.

Did this answer your question?