Service: Wevox Engagement
Who can access: All members
Service function: The following is an explanation of the algorithm used to calculate "Significant Variation." For an overview of material variation, please click here.
Summary
The "Significant Variation" algorithm detects score changes by synthesizing the results of three different statistical methods, each of which plays a different role.
“Long-term change detection" which detects changes that have occurred over a long period of time (90 days)
“Short-term fluctuation detection" detects important changes by comparing the results of the current and previous surveys.
“Abnormal Level Detection" detects unusual conditions by taking into account fluctuations over the past year.
As a result, it has the following characteristics
Detect noteworthy fluctuations regardless of the size of the team
Detect noteworthy changes, including score fluctuations due to changes in team members
Detects both rapid short-term and gradual long-term changes
As more data is accumulated, the system learns more about each team and evolves into a more individually optimized change detection logic.
※For more details, please refer here
What should I do in such a situation?
※ Articles(Japanese only) by Sugiyama, Wevox Data Scientist
Scores are changing but not detected.
Sometimes it happens that the score is changing but the change is not detected. For example, in the following cases.
This time the algorithm detects changes of a frequency that occur only about 5 times or less in 100 deliveries. Therefore, we can say that even this level of change in transition was a statistically non-rare change.
Scores are falling, but they are detected as "rising”
As of May 14, the score has dropped by 6 points from the previous time, but the short-term change detection is detecting an "increase". Why is this?
This is not a false alarm. In fact, a closer look at the data shows that there was a change in team members at this time, and the composition of the team members has changed significantly. The members who did not move had their scores increase by 6 points. Therefore, the change detection logic, which was focused on that side, judges the score as "up".
Long-term change detection and Short-term fluctuation detection are discrepant.
If there is a change in score as described above, as of December 15, the detection result will be "Short-term change is up" and "Long-term change is down". This is a rare case, but it may happen (in this case, it will be displayed as "down" on the screen).
The possible interpretations are: "The trend is down, but the blurring has returned," or "The downward trend has ended and the price has begun to rise.
Unfortunately, from the data alone, I cannot tell if it is one of these or another possibility, so we will leave it to the judgment of the team members on site.
It detects temporary blurring of scores.
Even with this transition, this change in score will say it's a "Significant Variation" and then Wevox detects it. However, it appears that the score was just a little bit blurred in the subsequent transition.
Inevitably, the timing of the detection does not tell us whether the score will return in the future (i.e., it was a blip) or whether the score will remain low in the future (an essential change).
In this way, it is possible to detect something that, in hindsight, was no big deal.
However, we do not yet have the power to predict "in hindsight," and it would be extremely difficult to make predictions based solely on survey data and without knowing the mood of the team, including in the future.
We still believe that the best thing to do here is to utilize the detection of variation as an opportunity for Wevox users who know the actual team to reflect whether or not something really noteworthy is happening.
(Again, this is one form of AI-human collaboration.)
Unable to detect very long term fluctuations
In actuality, the fluctuation detection did not work at this timing.
To be more precise, it was detected several times in the middle of the downward period of the score, but not at this lowest score timing.
The "Abnormal Level Detection" is designed to detect long-term fluctuations, but if the downtrend continues for a very long period of time (about one year in our sample), the algorithm becomes accustomed to the downtrend itself and does not detect it anymore.