Deciding on which method to measure results.
Before delving into the specifics of how we measure pricing performance, it's crucial to understand the rationale behind the methods we've chosen. It's essential to determine not just the outcome of pricing changes, but also the accuracy and validity of these outcomes. In a sector as nuanced as ecommerce, there are several approaches to evaluate pricing strategy, each with its own set of advantages and challenges.
Simply observing a rise in profit from one quarter to the next isn't conclusive evidence that our pricing adjustments are effective because profit can vary season to season. A mere glance at the Profit and Loss statement is therefore insufficient. To truly ascertain the efficacy of price changes, one needs to rely on solid data and employ stringent experimentation and statistical analysis to discern genuine results from mere coincidences or external factors. Here we explain some ways in which this can be done:
Within the realm of ecommerce, A/B testing is often heralded as the gold standard for gauging results. Here, one group of customers is exposed to one price, while another group sees a different price. The outcomes are then compared post-experiment. However, platforms like Amazon throw a wrench in this approach, as they lack the tools to segment prices for different customers. A frustrating limitation, indeed.
Switchback experiments present another option, wherein prices are altered daily (like dayparting). Yet, this too has pitfalls. A price change on one day might influence product rankings the next, thereby skewing the results of the subsequent price switchback. Here again, the challenge of separating genuine results from background noise arises.
Comparative experiments offer another avenue, where one subset of products (ASINs) undergoes price changes, while another remains unchanged. However, this method is fraught with complications. No two ASINs are identical, making direct comparisons challenging. Moreover, for this method to be effective, a substantial number of ASINs is required in each group. This poses issues, as some clients might have a limited number of ASINs, while others might wish to adjust prices across all their ASINs. An inherent issue is that the unchanged group essentially isn't utilizing our tool, Profasee, which isn't ideal for some of our clients.
Given these constraints, we opted for a different testing method. Our approach leverages data from the same period in the previous year, accounting for seasonality. This data, supplemented by information from several days before initiating with Profasee, helps us forecast the likely outcomes had Profasee not been used. In the realm of statistics, this technique is termed "synthetic control." Of course, we're well-aware that every year brings its unique challenges and opportunities. Therefore, we incorporate several statistical adjustments to enhance the accuracy of our predictions. Now, let me guide you through the intricacies of how we evaluate results in our results spreadsheet.
How we measure results:
Building on our earlier discussion, let's delve into the detailed steps of our analytical process:
Data Collection: We gather raw data for each ASIN, capturing metrics such as the number of units sold, revenue, and cost of goods sold (COGS).
Data Cleaning: The raw data inevitably contains anomalies that can distort our analyses. We meticulously cleanse the data, flagging and rectifying outliers. Furthermore, for periods when a product is out of stock, we account for these gaps. In instances where data is missing, we employ estimation techniques to fill in the blanks, ensuring a comprehensive dataset.
Aggregation for Seasonality: While individual years can exhibit unique characteristics, overarching seasonal trends often remain consistent. By aggregating data across all ASINs, we can capture these seasonal patterns. This aggregation helps in smoothening out anomalies specific to individual products and provides a clearer picture of broader market trends.
Computing Day-over-Day Changes: To understand the nuances of daily sales patterns, we compute day-over-day changes for the same period in the previous year. This gives us insights into the day-to-day fluctuations in sales, revenue, and COGS.
Extrapolation: Armed with the day-over-day changes from the previous year, we extrapolate this data forward. This helps us predict what the sales, revenue, and COGS might look like if the current year followed the same patterns as the last. Note, as explained in 3., although every ASIN is unique, we expect that in aggregate current year patterns should be the same as the last.
Setting the Starting Point: For the extrapolation to be meaningful in the current year, we need a solid starting point. This foundation is based on the average values of units sold, revenue, and COGS over the 30 days leading up to the implementation of the repricing strategy. By using this 30-day average, we ensure that our starting point is both recent and representative of the product's performance just before the pricing adjustments. It is from this starting point that we project last years patterns forward.
Producing a P&L Statement: After all the data processing and extrapolation, we synthesize the information into a Profit and Loss (P&L) statement. This statement offers a consolidated view of the financial performance post-repricing. It summarizes: revenues, costs, ad spend, profits and BSR.
P&L Statement per ASIN: For each individual ASIN, we allocate a portion of the overall P&L based on its historical contribution or performance. This means that if an ASIN historically accounted for a certain percentage of the sales or costs, we use that percentage to assign a corresponding part of the total P&L to that specific ASIN. The contribution profit or net profit in this per ASIN P&L we call the Total Contribution Profit Per ASIN (TCPPA).
This step-by-step approach ensures a robust and thorough analysis, allowing us to make informed decisions and provide valuable insights into the efficacy of our pricing strategies.