Should you collect data from hard-to-reach populations using Voice (IVR) or SMS?
If you’re not sure, you’re not alone: Unilever recently faced the same question in Indonesia. After sending around 26,000 SMS and IVR messages and questions to about 4,000 people, Unilever found that IVR performed better than SMS—on every metric—including cost per survey!
Keep reading to find out what Unilever learned about data collection.
Fighting malnutrition
Between October 2019 and February 2020, Unilever used the engageSPARK platform to help fight child malnutrition in rural Indonesia, where 23 million children (42% of the child population) are malnourished. Unilever’s Royco brand, known in many markets as Knorr, ran the program.
The program’s results provide essential lessons for practitioners in developing and emerging markets who implement behavior change programs that involve monitoring and evaluation (M&E) and collecting feedback.
Unilever used SMS and Voice/IVR to …
- encourage rural moms to cook healthier meals,
- monitor the effectiveness of the program, and
- get feedback about the program from the moms.
Takeaways for surveys in Indonesia
The Royco team ran A/B tests on various factors, with the key data-collection takeaways being:
- Voice gets more valid responses than SMS: Questions sent via Voice/IVR get a 2.7x valid answer rate compared to questions sent via SMS.
- Voice is Cheaper than SMS: Even though each Voice minute costs 1.5 times more than each SMS message, each valid answer was 8 times cheaper for Voice questions than for SMS questions.
- Multiple-Choice Questions get more valid responses than Open-Ended ones: Multiple-choice questions get a 1.7x valid answer rate compared to open-ended questions that asked for the same information; this is true for both Voice & SMS, though the gap is more significant with SMS.
- Retry Calls significantly increase valid response rates: Retry calls—where the engageSPARK system calls people again if they don’t pick up the call—really work; they increase the valid answer rate by 47%, at no additional cost.
- People won’t reply to SMS, even if they previously got an incentive for replying to Voice: Getting an incentive for replying to a question is not enough to overcome the apparent burden of replying to an SMS question. Only 33% of people who validly answered a Voice question (and got an airtime top-up incentive for doing so) replied to a later SMS question. In all other scenarios, the “repeat response rate” (people who replied to a later question after replying to a previous question) was about 80%.
While these results are from just one program in one country, the sample size was large (almost 4,000 people), and there are multiple, varied data points: the campaign included six questions that were sent one by one over several weeks, with respondents receiving a mix of SMS and Voice/IVR questions. We also have seen similar results in other projects in other countries.
Now let’s dig into the results!
Reinforcing habits with a message series (Drip campaign)
The campaign was part of Unilever Indonesia’s program on fighting child malnutrition. The Royco brand ran a cooking program for Indonesian mothers living in rural areas where they learned how to cook healthy and affordable meals from a chef during a group face-to-face cooking class.
But how do you make sure that the moms continue to cook nutritious food even after learning the recipes? To make this happen, Unilever forged partnerships with two tech startups: Powata and engageSPARK.
The program implementation was straightforward. Royco Indonesia offered to give away a free cookbook in exchange for the mothers’ participation in a 21-day campaign. During the face-to-face cooking classes, Unilever staff entered the mothers’ contact details into an Android App built by Powata. By giving their details, the mothers also agreed to be part of Unilever’s Nutritious Cooking Challenge.
That’s where engageSPARK came in: the Powata app automatically sent the moms’ first names and phone numbers to the engageSPARK system using engageSPARK’s APIs, which subscribed the moms into one of a few campaigns. The campaigns sent the moms reminder text messages about the program, survey questions, and encouraging messages to motivate them to keep cooking healthy food. The mothers were then rewarded with prepaid mobile phone credits, in the form of “airtime top-ups,” for their participation.
Unilever’s campaigns were A/B test experiments to determine which kind of content and which medium is most effective in influencing the end-users to change their behavior. To do this, Unilever created several versions of the campaign:
- An SMS Drip Campaign that sent all questions only via SMS
- An IVR Drip Campaign that sent all questions only via Voice/IVR
- A “mixed” Drip Campaign that alternated between sending the questions via SMS and IVR
Unilever assigned unique contacts to each version and monitored their response rates throughout the campaign. The goal was to find out which content and medium leads to the best results.
Voice vs. SMS A/B Test: Voice Wins!
Unilever found that IVR surveys had a 270% higher response rate than SMS – 270%! We have several theories:
- Being on a call drives urgency. People get bored or busy with other things: So, if a person doesn’t answer an SMS survey question right away, then they might end up never replying. If you answered the phone, you’re on the hook, and chances are you’ll reply.
- Calls are free for the recipients. Although Unilever ensured that SMS Surveys were also free for participants, often people are still afraid to be charged for messages they send, and they therefore don’t reply.
- IVR has richer content. Hearing someone’s voice on the other end of the line makes a significant amount of difference in terms of engagement. People tend to connect with the campaign more because of the human touch.
Interested in doing a similar Voice (IVR) campaign?
Snatch up a copy of our IVR guide for non-profits!
Cost Analysis: Unintuitively (sort of), Voice is MUCH Cheaper!
Yes, you read that correctly! The cost per answered IVR question was 8 times cheaper than the cost per answered SMS question. How come?
- You can pack a lot more into one minute of an IVR call than in a single SMS. An SMS is limited to 160 characters (or 70 characters for Unicode languages such as Chinese, Arabic, Hindi, etc.). A voice message asking five questions is much cheaper than sending five SMS messages.
- SMS Answers cost, too. The recipients can only respond to SMS Surveys by sending back a message. To make the campaign free for the recipients, Unilever Indonesia was charged for each message sent and received.
- With IVR, unanswered calls are not charged. With SMS Surveys the first question will still incur a cost even if no one replies.
- Call retries are free. The system can just keep calling the recipients without added cost. Each repeat of a question with SMS adds to the survey cost.
Multiple-Choice vs. Open-Ended A/B Test: Multiple-Choice Wins!
Unilever also compared response rates to two ways of asking a question: multiple-choice and (2) open-ended. Multiple-choice questions had a 1.7x higher response rate than open-ended questions.
For the open-ended version, they ask, “How many days do you cook from the recipe book this week?” The mom could then reply with anything.
For the multiple–choice version, they asked the same question but provided choices:
- 0 days
- 1-3 days
- 4-5 days
- 6-7 days
To respond to the question, the mom simply had to respond with a number. In this case, if they answered 4-5 days, all they had to do was reply with 3 (for SMS) or press the number 3 button on their phone (for IVR).
Multiple-choice questions had significantly higher response rates than open-ended questions for both IVR and SMS! Want to learn more? Read this blog post about how to ask better questions.
Retry Call Analysis: Retries Are Awesome!
People don’t always pick up the phone. That’s why we developed a feature that allows you to try again a few times.
Setting this up made all the difference for Unilever: call retries increased the valid answer rate by 47%, at no additional cost.
Repeat Respondents: Voice Wins Again!
One unexpected thing that Unilever learned was that answering a question once did not mean that the person would answer a later question sent to them via SMS, even if they received a nice incentive for answering the first question.
Unilever analyzed the likelihood that a person would answer a second question after they already answered a previous question.
- If the first question they answered was IVR, and the second question they got was IVR, there was an 80% chance that they’d reply to the second question.
- First question SMS and second IVR – 80% again.
- First question SMS and the second SMS – 80% again.
- BUT, first question IVR and second SMS – the reply rate dropped to 33%
This was a particularly interesting take-away because the respondents already knew that they’d be getting an incentive for answering the second question, as they had received one for the first question. But they still didn’t reply. The only explanation was that there was something about SMS that caused them to pause and choose not to reply.
Conclusion: Voice is Better than SMS
Based on the results of the campaign, Unilever found that voice is significantly better than SMS on every metric: (1) response rates, (2) cost, and (3) repeat respondents.
They also learned that multiple-choice questions are more effective than open-ended questions.
Lastly, this project validates our hypothesis that setting up call retries increases response rates—at no extra cost!
Does that mean that SMS is useless? Far from it—it depends on the use-case! Read here how Grameen engaged more than 20,000 women with SMS.
Interested in surveying hard-to-reach populations? Schedule a demo now!