Systematic reviews have become the gold standard in evidence-based research. They are guiding decisions in fields like healthcare and education among others. In this, you synthesise findings from multiple studies to get a more detailed view of the evidence related to a particular research question. Systematic reviews provide a clear overview of what is known and unknown and help to guide future studies and fill knowledge gaps.
However, like any research method, the limitations of systematic reviews must also be considered to make an informed decision. This blog guides you on the different weaknesses of a systematic review that researchers must be aware of to conduct successful research. But, before that, let’s briefly discuss what a systematic review is.
What Is A Systematic Review?
We might imagine casually browsing articles or literature on a particular subject when we think of a review. However, a systematic review is far more elaborate and methodical. As the name suggests, it follows a systematic approach to ensure that all relevant research is considered. It helps reduce research bias and provides a detailed overview of the evidence. Here are the top steps on how to conduct a systemic review.
- Define your research questions to ensure that the review remains targeted and relevant.
- Develop systematic review protocols for transparency throughout the process.
- Use different databases to conduct research
- Apply the inclusion and exclusion criteria to ensure consistency.
- Perform data extraction from selected studies
- Evaluate the quality of the study
- Analyse and combine results from individual studies to draw conclusions
- Present, interpret, discuss, and report your findings
What Are The Limitations Of A Systematic Review & How To Avoid Them
While systematic reviews are extremely important in the research field, they have their limitations. Some of these limitations of systematic reviews are discussed below, along with how you can mitigate each of them.
- Publication Bias
This type of research bias occurs when studies with statistically significant results are more likely to be published than studies with non-significant ones. Publication bias can skew the overall findings of a systematic review, as the review may disproportionately include studies that report positive outcomes while excluding those that report no effect or negative outcomes.
When you exclude unpublished or non-significant studies, this leads to an overestimation of treatment effects or the efficacy of interventions. Since systematic reviews rely on published literature, they are inherently vulnerable to this bias unless steps are taken to include unpublished studies and grey literature.
How To Avoid Publication Bias In Systematic Reviews
To avoid this, you can conduct detailed searches that include grey literature (e.g., conference proceedings and dissertations) and contact researchers for unpublished data. Additionally, funnel plots and statistical methods like Egger’s test can help detect and adjust for publication bias in meta-analyses. Look for an example of systematic review with limited findings to see how badly it may affect your research.
- Selection Bias
In systematic reviews, selection bias can occur when the selection of studies for inclusion is influenced by factors other than the predefined eligibility criteria. This can happen if your search strategy is not thorough enough or if studies are inappropriately excluded or included based on subjective judgments.
The studies included may not actually represent the broader body of evidence on the topic and lead to skewed results. This can also reduce the validity and reliability of the findings.
How To Avoid Selection Bias In Systematic Reviews
If you want to reduce selection bias, you should develop a comprehensive and transparent search strategy and adhere strictly to predefined inclusion and exclusion criteria. The use of multiple reviewers to independently screen studies for inclusion can also help reduce the risk of bias, as discrepancies between reviewers can be resolved through discussion or with the help of a third reviewer.
- Heterogeneity
Heterogeneity refers to the variability or differences among the studies included in a systematic review, such as differences in study populations or methodologies. If your study has high levels of heterogeneity, it would be difficult to combine studies in a meaningful way. If the studies are too different in terms of design or execution, there would be misleading conclusions.
Moreover, it becomes challenging to generalise the findings of a systematic review if the included studies represent very different populations or contexts.
How To Avoid Heterogeneity In Systematic Reviews
Systematic reviewers can use subgroup analyses or meta-regression techniques to explore sources of heterogeneity. In cases where heterogeneity is too high, a narrative synthesis (rather than a quantitative meta-analysis) may be more appropriate. Another way to mitigate this is to have clear inclusion and exclusion criteria during the selection of studies.
- Language Bias
Language bias occurs when systematic reviews exclude studies published in languages other than English. This can result in the exclusion of relevant studies conducted in non-English-speaking countries and limits the generalisability of the findings.
Excluding non-English studies can introduce bias. This is particularly problematic in fields where research is conducted globally, as excluding studies from non-English-speaking countries may lead to biased conclusions about the effectiveness or generalisability of interventions or treatments. Systematic reviews that focus only on English-language publications may miss important findings from other cultural or geographic contexts, which could lead to incomplete or skewed results.
How To Avoid Language Bias In Systematic Reviews
To reduce the risk of language bias, systematic reviewers should strive to include studies published in languages other than English by broadening their search strategy. Translators can be employed to help with non-English articles, and translation software can assist in screening abstracts. Additionally, reviewers can limit language restrictions only when it is absolutely necessary.
- Time & Resource Intensive
Systematic reviews are known for being labour-intensive and time-consuming. If you wish to conduct these reviews, you have to perform extensive literature searches, screening studies, data extraction, quality assessment, and synthesis of findings. The lengthy process may delay the availability of up-to-date evidence, especially in fields like healthcare and technology, where changes happen rapidly. This does not leave enough time for decision-makers to inform policy or practice.
How To Avoid
One potential solution to this limitation is using “rapid reviews,” which are systematic reviews conducted within a shorter timeframe by streamlining certain processes. However, this approach may involve trade-offs in terms of rigour. Advances in automation and machine learning also promise to reduce the time and effort required for literature searches and data extraction.
- Overreliance On Quantitative Data
Systematic reviews often work with quantitative data not qualitative, particularly when meta-analyses are used to pool results from multiple studies. However, this focus on quantitative data can overlook important qualitative findings that provide insights into the context associated with the intervention. With a focus only on quantitative data, systematic reviews often miss important parts, mostly in areas where qualitative data can shed light on how interventions work, succeed, or fail.
How To Avoid Overreliance On Quantitative Data In Systematic Reviews
To minimise this, your systematic review should incorporate both quantitative and qualitative data through mixed-method reviews. These reviews will combine the strengths of both approaches. Qualitative evidence synthesis techniques, such as thematic analysis or framework synthesis, can be used to explore the qualitative findings.
- Lack Of Transparency In Reporting
Transparency means the extent to which systematic reviews can provide clear and detailed information about their methodology, like data extraction processes. If systematic reviews lack transparency, it becomes difficult for readers to assess the validity and reliability of the findings, which undermines the credibility of the review and hinders replication by other researchers.
How To Avoid Lack Of Transparency In Systematic Reviews
You have to make sure that your review is transparent, and to do this, adhere to reporting guidelines such as PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). These guidelines provide a structured approach to reporting.
- Inability To Establish Causality
Systematic reviews are often based on observational studies or non-randomised trials, which can provide valuable evidence but are limited in their ability to establish causality. Unlike randomised controlled trials (RCTs), observational studies are more susceptible to confounding factors that may influence the results.
When systematic reviews include non-randomised studies, they may identify associations between variables but cannot definitively establish cause-and-effect relationships. This limitation is particularly important in fields such as public health, where interventions may have multifactorial causes and effects.
How To Avoid
Systematic reviews should carefully interpret findings from non-randomised studies. You should also include RCTs where possible to adjust for confounding variables and strengthen the evidence base.
- Influence Of Reviewer Bias
Reviewer bias can occur when the individuals conducting the systematic review have preconceived notions or preferences that influence the selection of the evidence. This bias can affect decisions about which studies to include.
How To Avoid Reviewer Bias In Systematic Reviews
To avoid reviewer bias, systematic reviews should adhere to predefined protocols, such as those outlined in the PRISMA or Cochrane guidelines.
- Difficulty In Updating Systematic Reviews
Since systematic reviews take time, new studies may be published during or after the review process that could alter the conclusions. Keeping systematic reviews up to date is a challenge.
An outdated systematic review may fail to reflect the current state of evidence, especially if important new studies have emerged since the review was conducted. This is problematic when systematic reviews guide decision-making, as decisions may be based on incomplete or outdated information.
How To Avoid
Regularly updating systematic reviews is crucial to ensure they remain relevant and reflect the latest research. Some databases and organisations, such as the Cochrane Collaboration, emphasise the importance of updating reviews periodically.
Advances in technology, including automated search tools and AI-driven data extraction methods, can help make updating systematic reviews more efficient. Furthermore, reviewers can adopt “living systematic reviews,” where evidence is continually updated as new studies are published.
Conclusion
This blog should help answer, what are the limitations of systematic reviews. It is important to evaluate both the strengths and limitations of systematic reviews before making your final decision. Systematic reviews have gained popularity and play an important role in research.
Frequently Asked Questions
Yes, PRISMA guidelines help reduce the risk of bias in systematic reviews by promoting transparency.
- Publication bias
- Overreliance On Quantitative Data
- Language Bias
- Heterogeneity
- Selection Bias
No, a systematic review is not primary research. Instead, it synthesises and evaluates existing primary research studies on a specific topic using a rigorous and structured approach.
