Open text questions can be a great addition to your survey. They allow respondents to answer in open-text format, giving them the freedom and space to answer in as much (or as little) detail as they like. You can use them to allow respondents to elaborate or expand on their closed answers or to ensure that you are not limiting your responses to a certain question to a fixed number of options. The analysis of open text questions can also be more resource intensive. Translating the many responses into actionable data and incorporating them into reports can take some skill and time.
Automated text analysis
Great news! Analysing open-ended responses has just become a whole lot easier.
We’ve partnered up with our data science team to build an automated text analysis tool. Our automated text analysis tool will scan all your responses, pick out the most important keywords & phrases and group them for you, so that you can discover insights more easily without needing to plough through all of your individual responses.
How does it work?
When your results are in, you'll see an "Automated text analysis" link on any open ended questions (English only).
When you click on this link you’ll be taken to the text analysis page of your survey. It can take up to 1 minute for the analysis to be run, but this will only be the first time you access this page.
When the text analysis has been completed, you’ll see a list of all the keywords and phrases that the model has recognized for a particular question. If you want to change your questions, or see responses for a different wave or country, you can change this on the right hand side.
You can also see your keywords visualised as a word cloud, which is a great way to visualize text data and get an easy to digest overview view of most frequently mentioned words. A word cloud shows a cluster of words, with the size of the word indicating how often this word has been mentioned in your responses.
You can remove certain keywords (e.g. "nothing") from your word cloud by hovering over the word on the left hand side and clicking on the eye icon. You can also adapt the number of keywords visualised in your word cloud (ranging from 1-50) by using in the input field to the bottom-right of your word cloud.
To see the individual responses that include each keyword, you can click on the arrow next to the keyword. This will show all the original responses that contain this keyword. It’s possible that a single response can have multiple keywords. In this case e.g. "entertaining" and "destructive".
Editing your text analysis
You can choose to merge different keywords together to create a new group. For example because your want to combine keywords that are very similar into one group such as "Good" & "Very good".
Once you have selected two or more keywords, you'll see the option to "merge" them at the bottom right of the list. When you click this the keywords are combined into a group that you can give a new name, in this example "Good".
When you click on "save" the keywords will now be combined into one group "Good" and the list and word cloud will update immediately.
If you want to undo your merge, you can click on the chevron next to the grouped key words and you'll see the option to "undo merge". You will also see the list of keywords that are included in this group, as well as all individual responses.
Limitations
At the moment automated text analysis is only available for English surveys, run on or after 1/1/2021. The model is run on all your responses, so it will not take into account any filters that you have applied on your dashboard.
Sentiment analysis
Next to the key words, you can also see how your respondents have answered to a certain question: positive, negative or neutral. Simply toggle on sentiment analysis on the text analysis page for any question where you might be interested in seeing the sentiment as well. You can read more about how sentiment analysis works here.
What is the algorithm behind automated text analysis?
We’ve spent months building and training our own data model for this feature, based on the millions of open-ended responses that we’ve gathered over the years.
There are two important steps in the way the model works:
Key phrase extraction
Key phrase grouping
The key phrase extraction algorithm is a deep-learning model that has been trained to extract important/relevant keywords in open text.
The key phrase grouping algorithm takes these key phrases and then groups similar key phrases into the same group. What’s unique about our key phrase grouping is that it isn’t just based on a simple text match, but based on deep-learning. This means that it’s able to also recognize terms that are semantically similar (for example, ‘amazing’ and ‘wonderful’).
We’re still aiming to further improve this model, so keep in mind that while it’s built on a solid foundation of data science, certain words might not be picked up by our AI model as a keyword. Also, while our model implicitly understands that words that are spelled similarly (i.e typos, doge <> dog) have a high similarity, this doesn’t mean that the model will be able to pick up on any type of spelling mistake.
If you feel that there’s a significant different between the automated text analysis and your original responses, it would be great to receive this feedback. We are always looking at how we can improve our data model and this might be useful information. Reach out to your CSM or contact us via intercom.