As it becomes more and more difficult to generate acceptable participation rates with online surveys and focus groups, panels are gaining widespread popularity. Online panels are pre-screened and selected communities of people that have applied to take surveys on an ongoing basis in return for some form of compensation.

Many large companies, such as Microsoft and Disney, build and maintain their own custom online panels of pre-qualified research participants. Other companies and private research firms turn to vendors that market online panels. Some offer access to consumer panels, while others offer B2B panels for targeting professionals, including CEOs, CFOs, CIOs, managers, directors, and other business decision makers.

Accessing your online research audience via pre-selected panels offers some significant advantages. Read the rest of this entry »

I recently attended a webcast hosted by the American Marketing Association entitled Assessing the Accuracy of Online Panel Research.  

Overall, online panels may seem ideal since they typically give you access to a large list of people from your target market.  However, some panels often have a large number of “professional respondents” (those who participate in panels or surveys for the purpose of financial gain or rewards).  Since these people usually care very little (if at all) about answering questions truthfully and completely, this could strongly affect the accuracy of your survey data.

To avoid this common pitfall, the next time you’re conducting panel research, make sure to ask your research partner how people were selected for the panel you’re using, how often the panel list is cleaned, and what safeguards they have to protect their lists from professional respondents.

With the summer season upon us, I like to retire to the beach for a little while to read, relax, and recharge. And, like most people, I always have a stack of reading material that I haven’t gotten to over the year. Recently, I did pass on James Patterson and catching up on the new Harry Potter and instead looked at the summer 2007 issue of the QRCA, Views from the Qualitative Research Consultants Association. I know, maybe not your cup of tea, but it has always been important to me, and the company, to stay on top of what is new and different.

What caught my eye was the article Intimate Strangers, by Laurie Tema-Lyn. Tema-Lyn talked about that magic moment in focus groups when a set of strangers coalesces into an intimate group and really begins to talk about their personal experiences. Now, Tema-Lyn conducts groups on some very personal issues relating to disease and other life issues, but I see the same things happen in the business groups that we conduct on a regular basis.

This is the truly rewarding moment of the job: when the natural defenses of participants drop and people really start talking about their problems and issues – sharing their personal frustrations with colleagues who feel their pain. This is insight for a company that a sales rep will rarely ever hear because of the nature of the customer-vendor relationship. When a group of business people reach that moment of sharing, of becoming “intimate strangers”, I know I have done my job well.

This is what keeps me going back to the office, when the ocean calls. That said – where is my sun screen…..?

As online survey software becomes more sophisticated, organizations are increasingly turning to web-based research as a viable business intelligence tool. That can be a wise decision, as under the right circumstances it can deliver timely and cost-effective results. However, carefully consider all the factors before deciding if an online methodology will effectively meet your research objectives.

Can we access the audience for an online quantitative study?
In many industries, it can be very difficult to rent reliable email lists as compared to phone lists. In addition, the available lists may not be representative of the entire population you need to survey. Furthermore, given that many people won’t respond to the survey request, the number of available email addresses may not be large enough to deliver a statistically valid sample. However, if you’re conducting a customer or employee survey and have an in-house database with email addresses, an online survey may indeed be the best approach.
�
Is a blind survey the best approach?
For many types of research, such as brand awareness and competitive analysis studies, disclosing the research sponsor upfront can negatively impact people’s willingness to participate or, equally detrimental, bias their responses to questions. At the same time, response rates to unsolicited blind email surveys can be extremely low, as the email must first make it past spam filters and then compete with many other emails for the recipients’ attention. As a result, online surveys are more successful when the sponsor’s name can be revealed and when they’re targeted at customers, employees, or other groups that are already familiar with that company.
�
What’s our best option for obtaining qualitative data?
If your research objectives call for qualitative exploration and conducting in-person focus groups is a viable option, by all means do. Important insights are often gleaned from the immediate reactions and interactions, eye contact, facial expressions, tone of voice and other cues that only a live focus group can deliver. However, in some cases it’s not possible to recruit for and conduct focus groups with people spread across the country. Or, for example, if the audience is made up of high-level executives, they may be unwilling to travel to a focus group facility at a specific time. In these cases, the next best choice is in-depth interviews (IDIs) conducted either in person or via telephone. Though IDIs don’t offer the interactivity of a focus group, they still provide rich qualitative data.

Certainly, the technology exists to conduct online focus groups, which typically take place over several days in a dedicated chat room with a professional moderator. However, virtual focus groups aren’t optimal in business-to-government and business-to-business markets, as most professionals are unwilling to log in multiple times over numerous days to participate in a discussion.

Is it an either/or decision between phone and online?
No, sometimes a hybrid approach is optimal. For example, phone is often mandated by the fact that the study requires sample and quota control to meet the objectives (i.e. perhaps the marketer needs to talk to people within a certain spending range). In these instances, researchers can give people who decline to participate on the phone the option of receiving the survey via email to complete at their convenience. This combined approach often lifts response rates. Or, by directing willing participants to a website while on the phone with them, researchers can measure effectiveness of advertising and direct marketing creative treatments with a statistically valid sample. Finally, because some people tend to not be thorough when completing an online survey, researchers can improve the data quality by calling those individuals for clarifications or further input.
Keep in mind that “online” doesn’t mean “do it yourself.” Making important decisions based on the findings of poorly designed and/or executed research - regardless of the methodology - can have costly consequences. So, even if you’ve bought that online survey software that sounds so easy to use, always consult an experienced research professional for guidance before conducting your study in-house.

Preparing and conducting effective market research programs requires expertise in many different areas. But, without a doubt, the most critical key to research success is the survey instrument. While we can’t even begin to address the complexities of developing survey instruments in one newsletter article, we can offer the following tips for evaluating the questionnaire your research provider develops for your next study.

Start out easy: If possible within the context of the research objectives, pose general, easier questions first in order to set the stage and put the respondents at ease with the survey. Then, lead into the more specific and/or complex questions that require more thought and consideration.

Be crystal clear: All questions must be succinct and easy to understand, or there is a high risk of respondents interpreting them differently or even stopping the interview. Though you’ll likely have a research provider draft the questionnaire, they naturally won’t know your company and market circumstances as well as you and other internal stakeholders. Hence, your internal team should carefully review questions for clarity.

Keep it simple: Sometimes the information you’re seeking just isn’t easily accessible. For example, respondents may not be able to recall the data required to answer or may need the assistance of another source to answer. Put yourself in their shoes. If it’s unrealistic that they could respond without looking up the answer somewhere or asking another person, simplify or eliminate the question.

Don’t bounce around: Whenever possible, complete questions about one topic before moving on to the next in order to help respondents focus their train of thought.

Carefully consider sensitive questions: Even though you want to learn as much as possible, be very discerning about sensitive questions. Completely eliminate those that you feel most respondents aren’t able or willing to answer. (Some will provide less-than-truthful responses to such questions, which is more detrimental to your business intelligence than not getting answers at all.) For those sensitive questions you do decide to include, save them for the end whenever possible in order to minimize the risk of alienating respondents from the entire interview process.

Stay on track: In last month’s Research IT, we talked about the importance of the research firm and client collaborating upfront to thoroughly define the research objectives, data gaps, and the ways in which the research results will be used. This process provides a clear understanding of the content, depth, and data views that need to be delivered in the final analysis report. As you review the survey instrument, it’s critical to keep these objectives and actionable results top of mind. In addition to ensuring the questionnaire is designed to deliver the types of analyses required, this helps you and your research provider keep the survey to a manageable length by discerning between “must have” and “would be nice to have” information. Remember – if the survey has too many questions, respondents are likely to end the interview before it’s completed.

With the increasing difficulty in generating statistically valid response rates for certain types of market research programs, research experts frequently recommend offering an incentive as a means of boosting the number of survey participants.

This incentive decision is based on whether or not the survey subject matter on its own will be enough to motivate the target audience’s participation. Keep in mind that some target audiences may be easier to motivate than others. For example, citizens who are asked to participate in government-sponsored surveys may feel a sense of civic duty to cooperate. Members of trade associations may be motivated by the new benefits that could result from the survey.

Conversely, it’s often much more challenging to gain survey participation from corporate CIOs and other IT-related titles as well as management-level government employees, making incentives a very useful tool for these audiences. Researchers also use incentives with smaller populations that require a higher response rate for an acceptable sample size. Additionally, incentives typically help expedite research projects with tight deadlines as well as those involving lengthier surveys that, for example, require 20 to 30 minutes to complete.

Equally important, numerous studies have shown that incentives don’t create biases that can hurt the quality of information collected. Rather, in addition to reducing undesirable answers to survey questions, such as “don’t know” and “no answer”, incentives can result in lengthier answers to open-ended questions. In short, incentives create a feeling of obligation that motivates participants to be more thoughtful and thorough, resulting in higher data quality.

In the likelihood that your organization will need to include an incentive to boost your survey participation rate, keep these guidelines in mind:

• Align the incentive and research methodology: Pre-completion incentives – often a quarter, a dollar, or a small premium item such as a key chain – are used primarily with mail surveys. Post-completion incentives that are sent to the participants after the survey is conducted are effective for telephone and web-based methodologies. While some researchers use non-monetary gifts such as premium items, a charity donation, or entry in a prize drawing, monetary incentives such as cash cards and gift cards have proven to generate significantly higher participation rates. Not surprisingly, the higher the monetary value, the higher the response rate.

• Select an incentive that’s relevant to your total population: Use a strong dose of common sense when selecting your incentive so as not to skew your research results. For example, an online music gift card may not appeal to older participants. The same goes for charitable donations – choose a non-profit organization with very broad appeal instead of one that could be viewed as niche or controversial in any way. Some researchers even offer incentive options of similar monetary value so participants can select their favorite.

• Offer the incentive regardless of survey completion: Do not tell the participants that they must complete the survey to receive the incentive. Rather, let them know they’ll receive it even if they have to end the survey early or choose to not answer certain questions. Otherwise, you’re at risk that they’ll be less-than-honest or incomplete in some responses just to ensure they get the incentive.

Though an incentive is an added cost to a research program, the increased participation rate, faster fielding timeline, and higher quality data typically make it a very wise investment.

Gleaning performance-relevant insights from the opinions, motivations, frustrations, and priorities of your target audience requires qualitative research. At Market Connections, clients often ask how we choose between the two most effective qualitative research methods — focus groups and in-depth interviews.

In an in-depth interview, which is scheduled in advance and usually runs 20 minutes to an hour, a research specialist conducts a one-on-one session with the participant via phone or in person. In a focus group, 6 - 12 participants gather at the same time at a research facility or in an online chat room (preferably the former if possible). A professional moderator leads a group discussion based on a pre-determined set of questions.

Certainly, each method has its pros and cons. A focus group enables idea sharing and open discussion among peers, which can be very insightful. But, with in-depth interviews, interviewers spend more time with each person and get opinions that aren’t affected by a focus group’s peer influences, such as a domineering participant or a popular idea proposed by others in the group.

Typically the specific circumstances of your research project will dictate whether focus groups, in-depth interviews, or perhaps even a combination, is most appropriate. For example, in the following situations, in-depth interviews are usually more effective than focus groups:

• The audience is limited and geographically dispersed.
• The research requires the input of decision makers from competing firms, since they are often unwilling to talk openly in front of one another.
• The target audience is comprised of very high-level and/or extremely busy professionals, such as corporate executives, doctors, and attorneys, as they are usually unwilling to travel to a research facility at a specific time.
• The discussion topics are sensitive or personal in nature.

A study based on in-depth interviews takes longer to complete than one that uses focus groups because of the amount of time required to prepare for and conduct each interview. � But, regardless, it’s important you and your research firm weigh each option carefully when determining the most effective qualitative method for your specific circumstances.

Keeping your customers coming back for more requires knowing both the good and the bad about their interactions with and perceptions of your company. The nature of those interactions dictates whether your customer satisfaction strategy is supported by transactional surveys, relationship surveys, or a combination of both.

Evaluating the Ongoing Customer Relationship
Relationship customer satisfaction surveys, typically conducted once or twice a year, are appropriate when interactions with the customer are ongoing or very frequent, making this type of study common among many business-to-government and business-to-business companies. They measure satisfaction and performance levels in areas such as price, value, quality, service, innovativeness, and responsiveness as well as the company’s ability to meet customer expectations and needs.

Furthermore, they may include the input of several people at the customer organization. For example, a government contractor needs to know not only the opinions and perceptions of its daily points of contact at the agency, but also senior officers who would approve or disapprove the contract renewal.

Evaluating the Specific Customer Transaction
On the other hand, transactional surveys are appropriate when customer interactions are straight forward or only slightly complex. They are utilized for measuring infrequent, event-driven interactions, such as installing a network or buying a car. It’s best to conduct these surveys immediately after the transaction, when the experience is still fresh in the customer’s mind.

Transactional surveys are highly effective for specifically measuring the quality of customer service. In fact, some call them customer service surveys. They differ from relationship surveys in design in order to measure a contained customer experience all the way down to the specific employee involved.

Utilizing Both for a Complete Customer Satisfaction Picture
Some types of companies may require both survey types in order to implement a comprehensive customer satisfaction strategy. For example, a large federal solutions provider that has consulting, integration, and/or program management engagements with its agency customers certainly requires annual relationship surveys. But, that same company may also sell servers or other COTS products that result in infrequent, event-driven interactions with other customer points of contact. In this example, the company also requires transactional surveys in order to measure customer satisfaction in all areas of the business.

When it’s time to pursue your next research project, you don’t have to choose between a purely quantitative or qualitative approach. You may be able to reap the benefits of both.

To refresh your memories, quantitative research entails a statistically valid size of the target audience and gleans objective, structured, numeric data that provides insights into the “what.” Conversely, qualitative research is subjective in nature and reveals the “why” by capturing insights about feelings, attitudes, opinions, behavior drivers, and so forth.

A good research firm can often combine techniques from both types of research into one study. For example, a customer satisfaction phone survey can ask for scores in various areas of performance. This close-ended approach enables structured responses for a quantitative analysis. But, the survey can also include open-ended questions to reveal the reasons behind specific scores, thoughts on current and future best practices, needs unique to each customer, and more. A good research firm will analyze and report on qualitative data separately, as it often drives different follow-up actions than quantitative data.

Similarly, with qualitative research such as focus groups for concept or product testing, some structured questions can save the moderator time and help quickly gather consensus from participants.

However, even if you utilize some quantitative techniques in those focus groups or other qualitative studies, don’t make the common mistake of treating the study results as quantitative. In qualitative studies, the sample size is typically too small and not randomly selected, so the findings cannot be projected onto the target audience as a whole. It’s not even safe to combine the results of several focus groups in hopes of gaining a quantitative view, as numerous variables are different from group to group, thereby affecting data collection.

Your best approach is to clearly define your research objectives, knowledge gaps, and budget. From there, a strong research firm can help you refine those objectives and determine the most effective means of meeting them. This may indeed include mixing qualitative and quantitative techniques to generate richer market and/or customer knowledge.

In this internet-centric business environment, where online survey software packages are a dime a dozen, some organizations are opting to plan, design, execute, and analyze e-mail-based research initiatives in-house. However, unless those organizations have experienced research professionals on staff, many become frustrated during the process and, in the end, are highly disappointed with the results.

Certainly, when your target audience is likely to be online and e-mail addresses are available, an online survey may be a viable alternative to the telephone methodology. In addition to accommodating longer questionnaires, online research enables faster data collection and is less expensive to execute. It can also result in more precise feedback, since participants can see the questions and take time answering them at their own convenience.

But, it’s important to remember that an online survey does not imply “do it yourself” research. Some of the risks associated with conducting online surveys without research experts at your side include:

• A biased survey. Because the company is highly vested in the success of a product or program, they often inadvertently decide upon questions that they like most, causing misleading, incomplete or inaccurate results.
• An ineffective questionnaire design. Problems with clarity, specific wording, and survey structure are very common among inexperienced researchers. For example, one company asked customers to identify their top-two most-preferred methods of receiving information from a list of options. But, because of unclear wording, most customers picked more than one option for both their most-preferred and second-most preferred methods.
• An inaccurate or incomplete analysis. Experienced research professionals are trained in the numerous techniques of statistical analysis. Without this expertise, it’s very unlikely the research report will be completely accurate or reveal the insights that can truly drive measurable improvements. Top analysis mistakes revolve around sample size and misdirected causality.

Of course, organizations face these risks when they attempt any type of market research on their own – not just online surveys. But, most aren’t even tempted to take on hundreds of calls for a phone survey or to recruit and conduct a focus group.

The bottom line is effective research requires specialized expertise – regardless of whether it’s implemented via e-mail, telephone, mail, or face-to-face.