Things to keep in mind when conducting a user study

I am currently evaluating a privacy label for IoT (Internet of Things) products, the process includes a user study that consists of a questionnaire (both sides of an A4 paper) and a follow-up interview (with audio recording).

Here are some notes for my future self, or anyone else who happens to deal with the same challenge. Note that this is not meant to be an exhaustive list - these are just some observations I made after reading other people's checklists and books about conducting such studies.

BEFORE YOU START: working with people is interesting and maybe you've always dreamed of running usability studies or other studies that bring you face to face with people, I get it. However, such studies will produce a lot of data, and the data will require a lot of time for analysis. Conversations will require transcripts, handwritten notes have to be digitized and kept in a spreadsheet. It takes a lot of effort to organize these materials and prettify them into a form that your software can digest. This process will not be fun, it will be boring and exhausting, you'll have to listen to some conversations over and over in order to figure out what it is that they're saying. At some point you'll regret that you wanted a user study and you'll wish you could go back and re-think your strategy.

Before you run this type of a study, make sure that you really need it and that there are no easier ways to achieve the same goal.

• Obviously, test the questionnaire with at least one colleague, before taking it to the public. Keep a diary of the modifications you make, and review the new versions with another colleague.
• Ideally, you'll have to pilot it with a normal human (i.e. not a colleague), as your colleagues are likely to miss some details, because they have more things in common with you than you think.
• If the questionnaire spans across several pages, make sure you have "please turn the page around for more questions" somewhere at the bottom. Otherwise, some participants will stop without going forward, and your data will be incomplete.
• Make sure that all questions where multiple options can be chosen are marked with "multiple choices possible".
• Carry multiple pens with you, as some participants will not have their own.
• Keep several copies of the study materials, in case some participants make markings on the originals, that should not deter you.
• Make sure the consent form provides opt-out checkboxes for people who agree to participate, but do not want their voice to be recorded.
• If you're distributing prizes, also include "I opt-out of the lottery" or "I do not want the reward", as some people will be willing to contribute in the name of science, with no other expectation.
• Prepare a table for collecting data about each session as it progresses; otherwise you'll have to do this in post-processing and it will take a lot more time. Moreover, you will most likely forget to perform certain actions (e.g. I forgot to mark the time it took each person to fill out the questionnaire); however - if you have a table to fill out, there will be a field to fill in - reminding you that it needs to be done.
• When you're done composing the questionnaire, go through each question and make sure it deserves to be there. More questions = more time = fewer people will go for it. Always minimize the time it takes, remember that a lot of people were enthusiastic about it, until they learned it would take up to 25 minutes.
• Prepare aesthetically pleasing promo-material that conveys the key facts about the study, so people can see it at a glance. Show it as you approach them to ask them to participate - some might listen, while others prefer visual information.
• Prepare some kind of a "thank-you note" that will be given to each participant after the study. The note has to include a reference to a site where the study results will be published, your contact information (in case they want to provide more feedback), and information about the project itself, to disseminate some facts about it. Remember that otherwise you cannot use their contact information to get in touch, unless the consent form had something to say about this.
• Shut up and listen to people when they talk. Let me make it clear - as you listen to the recordings, you will realize you interrupt them quite often, possibly stopping them from sharing important ideas. No matter how good of a listener you are, you can do better, shut the hell up. Yes, there will be awkward silence, but you're not an MC hosting a goddamn party, you're eliciting qualitative data in a scientific study.

Do you still want to run such a study? If you pull it off, you'll earn the right to brag about it ;-)