We can derive meaningful statistics from the smaller study samples typical of user
in General Discussion
Mon Jul 31, 2017 10:31 am
• 3 Posts
We can derive meaningful statistics from the smaller study samples typical of user research, and we shouldn’t be afraid to use these statistics to confidently guide business decisions!
Statistical analysis is most useful when evaluated alongside other data.
It’s not that we can’t get meaningful data from small data sets. It’s just that we should never look at statistical significance in isolation from other qualitative data. Jeff’s practical view is that there are no absolute thresholds that make quantitative data valid or invalid, just levels of confidence that the sample data represent real population parameters. In business, taking action on these numbers translates to taking risks and placing bets.
We often see that one participant does something notable in a usability test that involved 25 participants. Statistically, we can say that as many as 20 percent of the entire user population would make the same mistake (though it is far more likely to be more like 4 percent).
We would rarely decide to do stats after the fact. Discussing whether a client wants or could benefit from statistical results (beyond confidence intervals) in advance of the study is key to designing an appropriate study. With large validation studies, it may be appropriate and possible to dig a little deeper, to go beyond qualitatively explaining failures with medical devices. If we know qualitatively that there’s some obvious effect going on, we can gather data about it. If the difference or the relationship is big, we’ll see it, even with a small sample. So there are some big opportunities for our clients with Term Papers on Statistical Analysis