Learn to critically analyze statistical claims by examining data sources, study designs, and potential biases that affect the validity of conclusions.
Understanding Statistical Claims
Statistical claims are assertions based on data analysis. To evaluate them properly, consider:
Critical Evaluation Checklist
Source: Who collected the data and why?
Sample: How was data collected? Random or biased?
Measurement: How were variables defined and measured?
Context: What time period and population does it cover?
Alternatives: Are other explanations possible?
Common Issues in Statistical Claims
Issue
Description
Example
Correlation ≠ Causation
Assuming a relationship means one causes the other
Ice cream sales correlate with drownings (both increase in summer)
Selection Bias
Sample isn't representative of population
Phone survey missing people without landlines
Small Sample Size
Results may not be statistically significant
Surveying 5 people about movie preferences
Missing Baseline
No comparison group or before/after data
"Our pill works!" without placebo group
Case Study: Evaluating a Claim
"A study found that students who eat breakfast score 15% higher on tests."
Questions to ask:
Was this a controlled experiment or observational study?
How was breakfast defined and measured?
Were other factors (like socioeconomic status) considered?
What was the sample size and selection method?
Is 15% difference statistically significant?
Red Flags in Statistical Claims
Claims of "proof" (science deals in evidence, not proof)
Overgeneralizations ("All people...")
Emotional language instead of data
Hidden or unclear data sources
Conflicts of interest not disclosed
Key Takeaways
Always consider how the data was collected and by whom
Look for potential confounding variables
Be skeptical of causal claims without experimental design
Check if results are statistically significant and practically important
Remember that good statistics should be reproducible
Practice Question
A news article reports: "People who take Supplement X have 20% fewer colds! Based on a survey of 50 customers." Which of the following is the strongest reason to be skeptical of this claim?
A) The study didn't account for other factors like hand washing habits
B) The sample size was only 50 people
C) The study only surveyed existing customers who chose to buy the supplement
D) 20% seems like an unusually large effect
Consider which issue most fundamentally undermines the validity of the claim:
Sample size affects precision but isn't the biggest issue here
Not accounting for other factors is problematic but expected in surveys
The effect size being large doesn't necessarily make it wrong
Surveying only customers who chose the product introduces what type of bias?
Analysis of each option:
A) While this is a valid concern, it's common for surveys not to account for all variables
B) A sample of 50 could be reasonable depending on the population size
C) Correct - This shows self-selection bias; people who choose to buy the supplement may be healthier to begin with
D) The effect size alone isn't reason for skepticism without other context
The strongest reason is C because it identifies a fundamental sampling bias that undermines the claim's validity. People who choose to buy supplements may differ systematically from those who don't in ways that affect cold frequency.