Poor Questions Lead to Bad Results

Ask the wrong question and your results will be flawed. Whether you're asking an online survey question or conducting a face to face interview the question you ask can make or break your entire experiment or interview.

Recently, I had the idea to run a poll for the greatest Super Nintendo games of all time using a Verify Memory test. It seemed like a great fit, the actual test asked people for five games they remembered. The report grouped responses into the top five related phrases. I was surprised at the results.

More than half of the people that took the test did not follow the instructions.

The plan was to simply instruct users to enter their five favorite Super Nintendo games into the form, rather than their memories of the screenshot. Seems pretty straight forward right? Memory tests in Verify require an image, so a picture of a Super Nintendo seemed appropriate. I didn't want to skew the results by using a picture of an actual game cartridge. So after reading the instructions and clicking the button to start the test, people saw this for five seconds. After which they were presented with this form, which asked them to deposit the names of their most fondly remembered games. I published the test and patiently waited to bath in a sea of 64-bit nostalgia. I was not naive, I expected a few people to skip or misinterpret the instructions, and thus was very surprised that the second most popular phrase was video game, indicating that many people simply put what they remembered of the image, and not their favorites games.

Why is this? To put it generally, the test was poorly designed.

To be more specific:

• We asked a question about software (games), and then showed a picture of hardware (Super Nintendo)


• The survey did not follow the established pattern for a memory test, of collecting memories from an image


• The instructions were not attached to the form


• An image of a Super Nintendo was disimilar enough from the instructions to cause people to question what they were doing


• Images are more powerful then words, people followed the instructions implied by the image, rather than the explicit written instructions

So what?

Our takeaway from this fun little experiment is to think long and hard when phrasing a question. Whether it's Verify or a Q&A interview your question is what will make or break the results. When dealing with Verify tests make sure to:

• Have just one introduction/instructions screen


• Make your instructions a maximum few lines long


• Make the question/instructions as clear and concise as possible. You grandmother should understand them.


• Shoot the preview of your test to a few colleagues to see if they can understand and take your test without your help