When a study is not strong proof

Introduction

When reading quickly, words such as study, research, survey, trial, data, and evidence can create an impression of reliability before a reader has evaluated how strong that evidence actually is. This is a common skimming error. The presence of evidence is not the same as the presence of strong evidence.

For readers trying to increase reading speed without sacrificing comprehension, one of the most valuable habits is to read evidence words together with nearby limitation words. A claim supported by a study may still rest on a small sample, an observational design, wide uncertainty, missing measurements, or findings that cannot be generalised broadly. Research methods experts consistently note that limitations affect how confidently conclusions can be drawn and how widely results can be applied. [PMC]pmc.ncbi.nlm.nih.govPMCLimited by our limitationsby PT Ross · 2019 · Cited by 2339 — Study limitations represent weaknesses within a research design that may influence outcomes and co…

The goal is not to become sceptical of every study. It is to avoid treating the word evidence as a shortcut for proof.

Evidence markers that deserve attention

Evidence markers signal that an author is attempting to support a claim. They often appear in phrases such as:

These signals matter because they distinguish evidence-based claims from pure opinion. However, skimmers often stop reading after noticing the evidence marker itself.

Consider two summaries:

• “A study found that faster reading techniques improved comprehension.”
• “A study of 24 volunteers found that faster reading techniques improved comprehension during a short laboratory task.”

The second statement contains information that changes how much confidence a reader should place in the finding. The evidence is still real, but its scope is narrower.

Researchers who discuss study limitations emphasise that weaknesses in design, sampling, measurement, or analysis can influence conclusions. A study may contribute useful information while still providing only limited support for a broader claim. [PMC+2AJE]pmc.ncbi.nlm.nih.govPMCLimited by our limitationsby PT Ross · 2019 · Cited by 2339 — Study limitations represent weaknesses within a research design that may influence outcomes and co…

For fast readers, the danger is not ignoring evidence. It is noticing only the existence of evidence and missing its boundaries.

Limitation words that reduce confidence

Certain words deserve nearly as much attention as the evidence marker itself. They often appear immediately before or after the description of a study.

Small samples and restricted populations

Words and phrases such as:

suggest that the findings may not represent a wider population.

Sample size directly affects uncertainty and the reliability of estimates. Very small samples can weaken both internal and external validity, making it harder to know whether results would hold elsewhere. [PMC]pmc.ncbi.nlm.nih.govPMCHow sample size influences research outcomesVery small samples undermine the internal and external validity of a study.Read more…

A skimmer who remembers only “a study found” may miss that the study examined a highly specific group.

Correlation rather than causation

One of the most frequently misunderstood limitation signals appears when studies report associations.

Words and phrases such as:

do not automatically mean that one factor caused another.

Methodologists repeatedly warn that correlation alone cannot establish causation because other explanations may account for the relationship. Observational studies can reveal important patterns, but determining causal relationships requires additional evidence and careful analysis. [PMC+2PMC]pmc.ncbi.nlm.nih.govPMCObservational Research Opportunities and LimitationsCausal associations will always involve correlation, but the presence of a correlation does not imply causation. The…Read more…

This distinction matters because readers often convert an association into a cause while skimming. Research on language interpretation shows that people frequently infer causation even from carefully worded correlational statements. [PMC]pmc.ncbi.nlm.nih.govCausal implicatures from correlational statements - PMC - NIHby SJ Gershman · 2023 · Cited by 59 — Correlation does not imply causatio…

Uncertainty and imprecision

Evidence can also be weakened by uncertainty.

Look for terms such as:

Scientific communication often includes caveats because the available evidence does not justify stronger claims. Experts on uncertainty communication note that these caveats express limitations in knowledge or confidence. [PMC]pmc.ncbi.nlm.nih.govCommunicating uncertainty about facts, numbers and scienceby AM van der Bles · 2019 · Cited by 562 — Limitations in scientific underst…

Similarly, confidence intervals help show how precise an estimate is. Wide intervals indicate greater uncertainty about the true effect than narrow ones. [Cochrane]cochrane.orgChapter 15: Interpreting results and drawing conclusionsThe confidence interval describes the uncertainty inherent in any estimat…

When reading quickly, uncertainty words are easy to skip because they are short and visually unremarkable. Yet they often carry more meaning than the headline finding.

Missing measures and incomplete evidence

Another common limitation appears when important factors were not measured.

Authors may note that:

Such statements indicate that the evidence may answer only part of the question.

A study can be well conducted and still leave major uncertainties because some outcomes, populations, or time periods were not examined. Recognising these gaps helps prevent overconfident interpretations. [USC Library Guides+2Wordvice]libguides.usc.eduUSC Library GuidesOrganizing Your Social Sciences Research Paper: Limitations…3 days ago — Limitations of the study describe the const…

How to combine evidence and limits while skimming

Fast reading works best when evidence signals and limitation signals are treated as a pair.

Instead of asking only:

“Is there evidence?”

ask:

[> “What limits the evidence?”]pubrica.comsources, a limited sample size, or faulty methodology.Read more…

A practical scanning method is:

For example:

“A survey found that frequent readers reported higher comprehension scores. However, the survey measured self-reported behaviour and cannot establish causation.”

A rushed reader may retain only the first clause. A careful skimmer notices that the limitation substantially changes what can be concluded.

Another example:

“Researchers found a positive effect in a pilot study involving 18 participants.”

The evidence marker is researchers found. The limitation markers are pilot study and 18 participants. Together they suggest an interesting finding, but not strong proof.

This approach allows readers to move quickly through text while still preserving nuance. Rather than treating every study as equally persuasive, they learn to judge evidence in proportion to its constraints.

Why this matters for faster reading

Increasing reading speed often requires deciding where to slow down. Evidence claims are one of those places.

The highest-value information is frequently not the finding itself but the sentence that narrows it. A study may sound impressive until the reader notices that it was observational, based on a small sample, limited to a narrow population, or accompanied by substantial uncertainty. [PMC+2PMC]pmc.ncbi.nlm.nih.govPMCObservational Research Opportunities and LimitationsCausal associations will always involve correlation, but the presence of a correlation does not imply causation. The…Read more…

Readers who train themselves to spot limitation words alongside evidence words avoid a common skimming trap: mistaking the existence of evidence for the strength of evidence. That single adjustment preserves nuance, improves comprehension, and reduces the risk of carrying away a stronger conclusion than the text actually supports.

Amazon book picks

Further Reading

Books and field guides related to When a study is not strong proof. Use these as the next step if you want deeper reading beyond the article.

Book

Calling Bullshit

By Carl T. Bergstrom, Jevin D. West

Directly teaches readers not to mistake evidence language for proof.

See on Amazon

Book

How to Lie with Statistics

By Darrell Huff

Shows how studies and data can be misunderstood or overstated.

See on Amazon

Book

Thinking, Fast and Slow

By Daniel Kahneman

Covers biases that lead people to overtrust weak evidence.

See on Amazon

Book

How to Read a Book

By Mortimer J. Adler, Charles Van Doren

Rating: 4.0/5 from 41 Google Books ratings

Teaches readers to evaluate claims rather than merely absorb them.

See on Amazon

Browse more on Amazon: Calling Bullshit How to Lie with Statistics Thinking, Fast and Slow

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Example marketplace items related to this page. Use the search link to explore similar finds on eBay.

Shop location

Using USA USA USAUKCanadaAustraliaIreland

Example eBay listing

Wooden Office Study Desk Pen Pencil Holder Storage Organizer Box Tidy Case Stand

Search eBay.co.uk: desk study organizer

Browse similar on eBay.co.uk

Example eBay listing

Under Desk Drawers Study Table Drawer Tray Pencil Storage Organizer Box Holder

Search eBay.co.uk: desk study organizer

Browse similar on eBay.co.uk

Example eBay listing

Office Study Desk Organizer Table Box Tidy Case Display Stationaries Pen Holder

Search eBay.co.uk: desk study organizer

Browse similar on eBay.co.uk

Browse more on eBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

1. Source: pmc.ncbi.nlm.nih.gov
   Title: PMCLimited by our limitations
   Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC6684501/
   Source snippet

   by PT Ross · 2019 · Cited by 2339 — Study limitations represent weaknesses within a research design that may influence outcomes and co...

2. Source: libguides.usc.edu
   Link: https://libguides.usc.edu/writingguide/limitations
   Source snippet

   USC Library GuidesOrganizing Your Social Sciences Research Paper: Limitations...3 days ago — Limitations of the study describe the const...

3. Source: aje.com
   Title: how to write limitations of the study
   Link: https://www.aje.com/arc/how-to-write-limitations-of-the-study
   Source snippet

   (with examples)24 Aug 2023 — It discusses the types of limitations, their significance, and provides guidelines for writing about them, h...

4. Source: pmc.ncbi.nlm.nih.gov
   Title: PMCHow sample size influences research outcomes
   Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC4296634/
   Source snippet

   Very small samples undermine the internal and external validity of a study.Read more...

5. Source: pmc.ncbi.nlm.nih.gov
   Title: PMCObservational Research Opportunities and Limitations
   Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC3818421/
   Source snippet

   Causal associations will always involve correlation, but the presence of a correlation does not imply causation. The...Read more...

6. Source: pmc.ncbi.nlm.nih.gov
   Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC10010939/
   Source snippet

   How to Distinguish Correlation from Causation in Orthopaedic...by I Zaniletti · 2022 · Cited by 23 — Correlation does not imply causa...

7. Source: pmc.ncbi.nlm.nih.gov
   Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC10194916/
   Source snippet

   Causal implicatures from correlational statements - PMC - NIHby SJ Gershman · 2023 · Cited by 59 — Correlation does not imply causatio...

8. Source: pmc.ncbi.nlm.nih.gov
   Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC6549952/
   Source snippet

   Communicating uncertainty about facts, numbers and scienceby AM van der Bles · 2019 · Cited by 562 — Limitations in scientific underst...

9. Source: cochrane.org
   Link: https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-15
   Source snippet

   Chapter 15: Interpreting results and drawing conclusionsThe confidence interval describes the uncertainty inherent in any estimat...

10. Source: wordvice.com
    Title: Limitations of the Study
    Link: https://wordvice.com/blog/how-to-present-study-limitations-and-alternatives/
    Source snippet

    How to Write & Examples13 Jul 2024 — The two main categories of limitations are those that result from the methodology and those that res...

11. Source: pmc.ncbi.nlm.nih.gov
    Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC7081045/
    Source snippet

    for Evaluating Causality in Observational Studiesby E ALGianicolo · 2020 · Cited by 1 — The fact that correlation does not imply causalit...

12. Source: pubmed.ncbi.nlm.nih.gov
    Link: https://pubmed.ncbi.nlm.nih.gov/35231933/
    Source snippet

    Cross-Sectional Studies Contribute to Causal...by DA Savitz · 2023 · Cited by 385 — A cross-sectional study may provide insights into th...

13. Source: pmc.ncbi.nlm.nih.gov
    Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC8572982/
    Source snippet

    correlation analysis: important limitations and pitfallsby RJ Janse · 2021 · Cited by 462 — 'Correlation is not causation': a saying not...

14. Source: abs.gov.au
    Title: correlation and causation
    Link: https://www.abs.gov.au/statistics/understanding-statistics/statistical-terms-and-concepts/correlation-and-causation
    Source snippet

    2 Feb 2023 — A correlation between variables, however, does not automatically mean that the change in one variable is the cause of the ch...

Additional References

1. Source: researchgate.net
   Link: https://www.researchgate.net/publication/322778777Thinking_Clearly_About_Correlations_and_Causation_Graphical_Causal[Models
   Source snippet

   (PDF) Thinking Clearly About Correlations and CausationCorrelation does not imply causation; but often, observational data are the only o...

2. Source: scilit.com
   Link: https://www.scilit.com/publications/ead33c45eea2c0c76ec1d798f13a1d3e
   Source snippet

   Communicating the uncertainty in research findingsCommunicating the uncertainty in research findings: confidence intervals. Using confide...

3. Source: researchgate.net
   Link: https://www.researchgate.net/publication/236658638_Reporting_Confidence_Intervals_and_Effect_Sizes_Collecting_the_Evidence
   Source snippet

   (PDF) Reporting Confidence Intervals and Effect SizesConfidence intervals (CIs) and effect sizes are essential to encourage meta-analytic...

4. Source: thenewatlantis.com
   Link: https://www.thenewatlantis.com/publications/correlation-causation-and-confusion

5. Source: stat.uiowa.edu
   Title: Some Practical Guidelines Effective Sample Size Determination
   Link: https://stat.uiowa.edu/sites/stat.uiowa.edu/files/2024-04/Some-Practical-Guidelines-Effective-Sample-Size-Determination.pdf
   Source snippet

   Practical Guidelines for Effective Sample-Size...There are several approaches to sample size. For example, one can specify the desired w...

6. Source: lrdc.pitt.edu
   Title: 2022 Willett CorrelationCausationDissertation
   Link: https://www.lrdc.pitt.edu/rottman/pubs/2022a/2022%20Willett%20CorrelationCausationDissertation.pdf
   Source snippet

   Correlation Doesn't Imply Causation: Improving...by CL Willett · 2022 · Cited by 10 — “Researchers at the Sleep Research Society have fo...

7. Source: reddit.com
   Link: https://www.reddit.com/r/PhilosophyofScience/comments/1fhimhe/how_does_science_cope_with_correlation_does_not/
   Source snippet

   l Nino and Australian rainfall, or of Milankovic wobbles and ice ages...

8. Source: reddit.com
   Link: https://www.reddit.com/r/statistics/comments/6ylvo8/if_correlation_doesnt_imply_causation_how_can_you/
   Source snippet

   e must be some direct or indirect causal chain linking them.Read more...

9. Source: ndi.org
   Link: https://www.ndi.org/sites/default/files/samplesizecalculation.pdf
   Source snippet

   A confidence interval gives you a maximum and minimum...Read mor...

10. Source: pubrica.com
    Link: https://pubrica.com/academy/case-reporting/what-are-the-study-limitations-and-how-should-they-be-stated/
    Source snippet

    sources, a limited sample size, or faulty methodology.Read more...