If you have ever questioned the research results touted in an advertisement, magazine article, presented at a meeting or by a salesperson, you are not alone. As a dairy producer or nutritionist, you are probably not a researcher by training. But you do make decisions every day that impact the health, productivity and longevity of your cows. A little bit of healthy skepticism can help you do your due diligence before purchasing a new feed additive, medication or adopting a new management protocol.
Ask to see the original research paper to see how data was collected and analyzed, don’t just take results in a chart at face value. Moreover, the simple step of examining P-values helps determine statistical significance of the findings. This is a good first step to help separate the wheat from the chaff when it comes to research.
At the 2020 ADSA Discover Conference last fall, Nora Bello, professor of statistics at Kansas State University and an animal scientist and veterinarian by training, addressed the issue of interpretation of P-values as one component of the statistical toolbox for reproducible research in the animal sciences. Bello explained that a key tenet of research is repeatability; that is, that the major conclusions should withstand both close interrogation and independent validation. If the results cannot be replicated, or several studies show a wide range of results, that means that the problem has not yet been fully understood and solutions are still to be found. “We have to remember that research is a journey of scientific discovery,” she explained.
An important number to look at when interpreting research results is the P-value. A P-value of 0.05 or below is generally considered the threshold to declare statistical significance. However, one has to be careful when interpreting P-values because misinterpretations are all too common. Despite popular hear-say, a P-value is NOT the probability of having made a mistake. Instead, P-values should be interpreted as “a measure of surprise of the results obtained relative to a set of assumptions,” says Bello. These assumptions are critical in the interpretation of P-values. A P-value assumes that the experiment will be repeated an infinite number of times; and that the treatment studied is truly ineffective. A small P-value ≤ 0.05 provides evidence to cast doubt on the latter assumption leading to the conclusion that the treatment did cause a change in outcome. By contrast, large P-values (P > 0.05) do not allow one to differentiate between potentially real treatment effects for which data might be insufficiently informative and a chance numerical difference that is, by definition, not repeatable. For this reason, “claiming ‘practical significance’ or ‘numerical differences’ in the absence of statistical significance is bogus,” explained Bello. When it comes to interpreting nonsignificant P-values, results are inconclusive at best and should be reported as “no evidence of treatment differences.”
In addition, wording of “trend” or “tendency toward significance” are sometimes used incorrectly to describe P-values just above 0.05. This is misleading, as the wording inappropriately implies directionality based on a single point, says Bello.
Using this understanding of P-values, let’s make an example of lactating cows fed diets A and B, for which milk production averaged 85 and 89 pounds, respectively. If the P-value associated with the corresponding test statistic was, say 0.15, the findings are not statistically significant and therefore, one cannot claim that cows fed diet B produced more milk than cows fed diet A. Instead, if a P-value of 0.05 or below was found for the same example, one could conclude on proof-beyond-a-reasonable-doubt that diet B did cause an increase in milk production.
To learn more on this topic please see the open access invited review by Bello and collaborators in the Journal of Dairy Science.
