Lean user tests — how to find the right candidates

TL;DR

9 min readMay 9, 2016

This part of the Lean User Tests series focuses on acquiring the right candidates for the interviews, answering questions like:

How many applicants are required to fill an interview slot?
How to reach out to prospective candidates among your user base?
How to set up the application form?

There are differences in approaching candidates in the B2B versus the B2C space. However, the following general principles apply to all user interviews.

How to Identify the Right Interviewees for Lean User Tests?

The result of any user test completely depends on the selection of the right interviewees. You need to pick the right candidates, or otherwise you will waste resources on a useless user test session.

The company, for which I organized and ran lean user tests, stood out due to two characteristics:

On the hand, many users had either never made a purchase (50%), or purchased only once. (25% of the total user base.)
On the other hand, a small group of users drove a large part of the overall sales.

Consequently, most of the lean user tests involved two questions: a) how to get new users to make the first and then a second purchase and b) how to keep the regular clients happy, that represented more than 75% of the overall revenue?

The Process

The acquisition of participants for the user interviews took place by advertising a landing page with an application form. The compensation for participating in the lean user tests was a voucher of the company at a value of €20.

This voucher could be redeemed anywhere on the company’s platform. It could also be transferred to third parties. Last, but not least: The value of the voucher could not be exchanged for cash.

The Required Number of Applicants

Not every applicant is suitable for an interview. Even such a moderate compensation of just €20 still attracted applicants, who were only interested in the voucher.

Therefore, the number of applicants had to exceed the required number to fill the schedules interview slot by a factor of at least two or three. Thus, we could sort out those applicants, who were not suited for the task, and we could respond in a timely manner to those we wanted to invite.

Note: You cannot expect that those who apply — even if you respond to them quickly and inform them about the date of the interview — will actually confirm the invitation and show up to the interview.

Marketing for Participation

We used three methods to attract potential interviewees to apply for participation in lean user tests:

Email marketing: A list of prospective interviewees was selected jointly with the marketing department and invited by an email campaign. The recipients were mainly customers based in Berlin, who had either purchased many items or who had never bought anything. The emails contained the date and purpose of the user tests, but they otherwise referred to the landing page, where the applicants registered their interest. It took the marketing department usually a week to organize an email campaign.
Social media marketing: This acquisition channel was also provided by the marketing department. (And it meant mainly Facebook.) The social media marketing campaigns were a lot less targeted and returned a poor outcome.
Messaging: The internal messaging system of the platform in question was by far the best method to target a specific group of users. The selection of those being contacted was controlled by a specific set of rules that were matched in real-time with former customer data. The only requirement for this messaging was, that the client had to log-in to the website.

The interviewee marketing started around 3–4 weeks prior to the scheduled user test. The launch depended on the time of year, taking into account school holidays etc. It was the general perception that no lean user tests should be carried out in December and January, as the number of applicants would drop too far below the required number.

Also, the number of canceled appointments (or no-shows) would be significantly higher. This also applied to the two weeks before and after the Easter holiday.

During the school summer holiday, it is recommended to increase the size of the target group by 20–30%.

For the acquisition of prospective interviewees, we did not use any of the following sources that are popular in the United States:

Craigslist,
User forums or
Market research agencies, that offer professional castings for interviewees.

Please hit the “heart button” 💚 below, if you found this post useful–it would mean a lot to me!

Numbers, data, and statistics:

To fill 30 interview slots (at 60 minutes each) for a compact test campaign over a period of six days, we needed approximately 100 applicants.

The user tests were intended to support the introduction of a new responsive frontend design in February 2014. The 100 applicants were picked from a group of over 4,200 customers. In the end, we had to pitch approximately 140 users to fill each interview slot.

Conclusions: Best practices to acquire interviewees are highly dependent on the product and its market. B2C is rather different from B2B. Hence, the “hiring” process usually has a steep learning curve.

Therefore, I suggest organizing the first user test sessions independently from each other, so that you can improve your approach from session to session.

Download the Ultimate Checklist for Lean User Tests

Never again forget anything necessary to run a professional user test: just download this proven checklist PDF for free, that was fine-tuned over 2 years in the trenches of continuous user testing.

If you do so, I would appreciate your feedback on how this worked out for you. (Please note that downloading the PDF will also subscribe you to our awesome weekly, completely hand-curated newsletter, if you haven’t yet signed up already. You can unsubscribe at any time.)

⬇︎Download here: The Ultimate Checklist for Lean User Tests⬇︎

We do respect your email privacy at all times.

The Application Form for Interview Participants

To make the selection of the best interviewees as simple as possible, the application form should provide the organizer with information to help you pick the right candidate. Technically, I suggest collecting this data via a Google form. Your benefit is obvious: It’s a tested, reliable solution free of charge and you don’t need to program anything.

Forms are a good way to separate the wheat from the chaff: The amount of information, as well as the effort the applicant puts in, are both relevant for choosing the right interviewees. Those, who put more effort into their answers during the application process are most of the time also more helpful during the interviews.

The application form should be introduced with a text that explains the nature of the user test. Let me give you an example:

Help us to optimize our website for mobile devices…

Through a permanent dialog with our customers, we aim to optimize our website continuously and make it as user-friendly as possible. We are excited, that you are considering spending 45 minutes of your time with us, to take part in our user testing.

Please note, that naturally we are receiving a lot of interest for these user tests. So many in fact, that unfortunately we cannot always consider all applications. We are looking for applicants, who will be most suitable in helping us with certain questions we may have.

If you are chosen to become one of these participants, we will invite you personally with a private email.

The application form intentionally doesn’t address two things:

The location of the interviews: In the beginning, when the location of those lean user tests had been openly communicated in the application form, we experienced a few situations, where applicants simply turned up without a prior invitation.
In their opinion, the submission of the application form was enough for them to be considered for participation. To avoid this problem, the text of the application form was changed, so that it would no longer show the actual location of the user tests.
This didn’t, however, result in a decrease in the number of applications. The participants were just as happy to drive from the west of Berlin all the way to former East Berlin to participate in these tests.
The compensation in the form of a €20 voucher: You don’t want the kind of applicant in your user test, that is only in for the money. These participants tend to answer, what they presume the interviewer would like to hear. This regularly leads to false test results and the effect is known as false positives.

The Screening Questionnaire

My screening questionnaire required the following basic information from the applicants:

Salutation: Mrs/Miss, Mr, first name, last name. (By the way, you might encounter applicants named “Donald Duck” in the process. #justsaying)
What is Your Age? Here, we only wish to learn the rough age category. To my experience, most participants cheat about their age by a few years anyway. (Just like on every online dating site.) You shouldn’t ask for the exact birthday either because it may influence the conversion rate of the application form very negatively, as most people don’t actually put in their correct birthdays anyway.
Which email address is best to get in touch with you? This email address should only be used for interview-related communication with the applicant and should not be abused for any marketing activities. You should be aware that some applicants will use special email addresses for certain applications, so they can identify a probable misuse easier.
What day would you like to take part in the user test? Underneath this question, you should present a choice of various interview days. The best days to offer are Tuesdays, Wednesdays, and Thursdays. Although Monday has a higher response rate than Fridays, it still does not compare to the three other days mentioned above.
To my experience, it is best to offer no more than two testing days to choose from. Less choice results in a better conversion rate of the form. This will also keep the organization of the whole task at a manageable level. (I once tried to organize three user testing days at the same time, and it proved to be complex.)
At what time would you like to take the test? It makes sense to schedule the interviews from 2 to 6pm, because during the morning hours most students and working people won’t be available. (The B2B environment might be different than the B2C environment in that respect.)
For possible answers, you should provide the actual interview slots. For example, one for every hour starting at 2pm, if you will be running four slots at 60 minutes each during the afternoon session.
There should also be an option “I am available anytime”. The applicants in this category are particularly valuable, as they are a sort of joker and can be moved around to fill the gaps in the schedule. (For example, the slots at 3 pm always proved to be unpopular.)
What is your occupation? This answer is not obligatory. (Choose an open text field.)
Are you already a customer of XYZ? (XYZ equals your company name.) Choices: a) No, I have never heard of XYZ, b) I am not a client yet, but I do receive the newsletter, c) Not yet, but I am registered, d) Yes, I am a client and have bought things from XYZ, e) The “Not yet, but I am registered” category was of particular interest in this case: Why bother signing up for a service, but never use it?
Why do you wish to take part in a user test for XYZ? The reasons for taking part in user interviews may vary significantly. Applicants, who don’t give a specific answer or who are after the expected compensation or seem to be bored (typical answer: “Why not?”), are not that well suited for the task ahead.

You may, of course, add task-specific questions to the questionnaire or questions that refer to a particular group of interviewees that you would like to apply. However, I suggest keeping the questionnaire as short as possible to avoid frustrating prospective candidates. The shorter the form, the more likely people will submit their application, which also is the reason why I like to use pick-lists for answers, thus avoiding too many free text fields.

More to this subject can be found in the following blog post by Usertesting.com: “Recruiting Your Target Audience for UX Testing with Screener Questions”.

Such an application form based on a Google form can easily be incorporated into any HTML page via its iFrame. This resulting landing page will be used for the interviewee marketing campaigns.

Google forms also provide a generic HTML page each that already contains the application form. The URL of this page can be sent directly to your target group via email. This way, you do not have to bother setting up a blog post, for example. However, due to legal or marketing/branding reasons, this should be avoided. This form page, for example, doesn’t contain an imprint and may, therefore, be at risk to cause legal action by a competitor.

Next Post: Inviting Interviewees

See you in part 3, when we cover who is a suitable candidate, how to invite them and why you will need replacement candidates.

Part 1: Lean User Testing — How to Run User Tests Successfully: The ‘Why’ Question

Four Lessons Learned From Making Customer Value Your Priority