I am a “control freak”. Now, I do not mean that I try to control and manipulate every detail- those of you who know me well know that I am more of a broad vision strategist. What I do mean is that when it comes to research and to surveys I insist on having a control group.
What is a “control group”? In science, it is a group of test subjects in an experiment that do not receive any of the treatment or the type of input that is received by the group that is the focus of the test. If you want a market research survey to be scientific- if you really want to know that you have valid results- you need a control group.
Why? If you intend to prove that your activities are the cause of impact it is necessary to prove that the same impact does not occur in the absence of your activity. This may all seem a bit murky, theoretical, and very irrelevant to your work, so let me clarify this with an example.
Suppose you run a hypothetical college ministry in the nation “Anystan”. Your ministry has launched a new evangelism program that you believe would arouse a greater interest in knowing God among college students. So, your ministry does a typical research study on the campus where you work and measures the student interest in knowing God before and after going through this new evangelism program. At the end of this research study you discover that before the program that students’ interest in God is a “2 out of 10” and that after the program this same measured interest in God jumps to “7 out of 10”.
A glowing success? Why not? Isn’t it good enough to show that there was an increased interest among the students that you work with? Yes, it is good that there is measurably more interest in God, but is this impact really the result of your new program?
Let’s now imagine that a control group was in the same research study on another campus in Anystan where you are not running this new program. These students are given a “before” survey at the same time as the students are given their survey on the other campus where you work. Likewise, they are given an after survey at the same time- even though there was no “after” at this campus since there was no program at this campus. So, if the results of the control group show that the students have a “2 out of 10” interest in God beforehand and a “2 out of 10” interest in God afterward then you can confidently make the claim that your program has made an impact among the students.
But what if the control group of students got exactly the same scores as the students on the campus where you are running the new program: a “2 out of 10” interest in God at first and then a “7 out of 10” interest in God? What does this mean? It means three things: 1) Students in Anystan are simply starting to become more interested in God, 2) your ministry may have nothing to do with this new interest, and 3) you just might be wasting money on a program that really has no impact.
Now, if you think that this is a dismal scenario, then consider the other option. What if the scores in the control group of students- the students not exposed to your program- are a “2 out of 10” interest in God before and “9 out of 10” afterward? In other words, the students at the campus where you are not working have experienced a greater increase in a desire to know God than the students at the campus where you are working. This would mean that there is probably a work of God among the students of Anystan and your new program may actually be hindering it.
If the control group was never tested, then your ministry would probably assume that you are doing a good work. But there is still the very real possibility that the program has no impact or may even be having a slightly negative impact. The strategic decision without a control group would be to fund this program while in reality you may be spending resources on an ineffective program or even a program that negatively impacts the vision of your ministry.
So why hasn’t any other researcher talked to you about control groups before? I must confess that Clarity Research is one of the few research firms serving ministries that regularly advocates for using control groups. While control groups are a standard practice in market research among businesses, it is mostly absent from ministry research. Personally, I do not recall running a single market research study without a control group in more than a decade of work in a Fortune 100 company.
So why don’t we have control groups in our ministry research? I think that there are three honest and legitimate reasons. The first two are time and money. It is simply more expensive and time consuming to find and test a survey among a control group as well as your target group. The additional cost in time and resources to run a control group should be carefully weighed against the possibility that God may be working in other unknown ways among your target audience. But realize the consequences of foregoing a control group: your results are a reasonable and data-based assumption but they are not necessarily true.
The other valid reason for not running a control group? It’s the danger of uninformed respondents. I will unpack this interesting phenomenon that I discovered in Moldova in next month’s update.