It can happen all the time and of course it did happen before, that we had to refuse to start an online study because of a very, very bad study design. Nevertheless, this time I thought I’d write some short notes on it, simply because I think that this issue is very important and somehow underestimated.

And of course I’m posting this blog-entry because I hope to get a discussion started and hear back your thoughts on this issue!

In a nutshell this is what happened: We won a project from a client, which sounded fair and easy in the very beginning. During our project managers test of the survey we found the whole study design to be more or less one big grid question and including a plausibility-check at the very end of one page. This check would exclude all panelists who would not answer a certain grid-item with a certain scale-item (where the right answer had nothing to do with a candid answer on that certain item).

Don’t get me wrong, I completely agree with plausibility-checks but I do also believe that researchers need to make sure, that surveys are engaging or – if this is not possible for any reason – at least not disencouraging. But the combination of a very long and boring survey and a plausibility-check if respondents would not keep up their concentration over a bunch of grid questions with up to 30 items, isn’t something which I think is helpful for better data quality.

I agree that online panel providers are responsible for panel quality and maintenance but my point is that good data quality can only be delivered if – next to a quality panel – the study design delivers good quality as well. If there is a lack on questionnaire-quality, we cannot be responsible for high drop-out rates or cheaters. BTW: I think we cannot even name those cheaters who are “straightlining” due to fact that the same-looking odd grid question is asked (of course slightly modificated) 3-5 pages in a row where people would have to scroll down on every page to see the last items. Would you remain to answer seriously and with the same concentration after the 20th item?

Back to the story: We finally decided not to start this project and – as it looks right now – lost an interesting client. It wasn’t just the fact that the study design was that bad, but also that participants would have been excluded if they – in my opinion understandably – would have lost concentration at some point of the survey and therefore would have been the ones to be blamed for a disencouraging and bad survey design. Putting high quality expectations on a fieldwork provider but in return not delivering quality in the study design didn’t seem to be a fair deal for us. Especially not, as we would have been the ones to pay for it due to high dropout- and “cheater”-rates.

Now I would be interested in your thoughts: Would you field such a survey? Would you – as a client – expect your panel-provider to start a project like that? Would you accept even major changes on your survey design? How do you cope with issues like that?