Highly Significant Findings Can Still Be Unimportant
Thursday, May 15th, 2008
You’ve likely heard researchers use the term “statistical significance” when talking about market study findings. Unfortunately, the term is not always clearly defined, and some clients are inadvertently mislead.
While “significant” in normal conversation means “important”, in the world of statistics it means “probably true” rather than due to chance. It conveys how likely it is that a relationship between two data points under study really exists. Additionally, the higher the level of significance, the more likely it is that the statistic is reliable. Equally notable, when researching a large sample size, very small differences will surface as significant.
However, just because a research finding is statistically significant does not mean that it’s necessarily important. For example, a study may reveal a significant decrease in the decision-making influence of tradeshows on the target audience. However, despite this decline, tradeshows remain the single most important source of information for the target audience, and tradeshow attendance rates have remained steady. If a large percentage of the company’s revenue is produced from leads generated at these tradeshows, the finding is really not important enough to merit a dramatic shift in strategy.
The most commonly accepted level of significance is 95%, meaning that the finding has a 95% likelihood of being true. This level is widely accepted as an industry standard. Depending on the sample size, industry, and type of research being conducted, some researchers are comfortable going as low as a 90% significance level.
So, the next time your research firm conducts a study for you, expect them to identify and present statistically significant data. But, don’t assume all of that data is important to the overall research project goals or to your company’s business intelligence. Rather, plan on assessing the true importance of significant findings within the context of your business objectives and strategies. And, if your research firm is a true partner, they’ll be more than happy to help with that assessment.
