Discussion
We aimed to evaluate the methodological quality of SRs comparing CA and conventional arthroplasty using the AMSTAR 2 tool and summarize and examine the consistency of the evidence generated by these SRs. Our umbrella review identified 42 SRs. We found low confidence in the evidence provided by 3 SRs, and very low confidence in the evidence provided by the remaining 39 SRs. Most SRs concluded that CA procedures had generally better radiological and similar functional outcomes compared with conventional procedures. However, depending on the outcome, discrepancy in the conclusions of the SRs varied significantly. Patient safety related outcomes were infrequently reported in the included SRs. Over-representation of the primary studies was shown by the moderate overlap among TKA SRs, and high overlap among THA SRs. These conclusions have implications for policy makers evaluating and adopting this technology, and for journals considering future SRs for publication.
We found that most of the included SRs showed that CA procedures are equivalent or better than conventional ones, which may have been used to support the increase in utilization of CA THA and TKA.7 However, given that these SRs are inflicted by the very low confidence in their conclusions, we caution that these findings should not be used to support further adoption of this technology. Moreover, the published SRs included little data on patient related safety outcomes, which creates a major gap in the assessment of the technology, especially knowing that THA and TKA are among the top seven orthopaedic procedures with the highest complication rates.75 While the US Food and Drug Administration approved the use of navigation systems, postmarket surveillance is still needed to minimize unintended consequences, as is the case with metal-on-metal hip resurfacing, which proved costly and unsafe.76 77
There is a plethora of outcome measures and inconsistency in reporting outcomes in SRs. This finding highlights the need to standardize the outcomes reported by both the primary studies and SRs,78–80 in order to synthesize the evidence more comprehensively and meaningfully for technology assessment and guidelines development. To address this, core domains have been developed for clinical trials of TKA and THA developed by the Outcome Measures in Rheumatology Trials (OMERCAT) Total Joint Replacement Working Group;81 82 however, those core domains are not yet fully represented in trials and SRs of CA TKA and THA. For example, the included SRs in this review did not report measures related to patient satisfaction, revision, and death domains and only few reported on adverse events domains.
By synthesizing evidence from RCTs and other comparative non-randomized studies,83 SRs provide much needed data for the evaluation of medical devices. The Idea Development Evaluation Assessment and Long-term (IDEAL) framework, allows robust evaluation of surgical innovations based on its stage of development.84 85 Although 35 of the 42 SRs were published after the publication of the IDEAL framework in 2009, none reported the IDEAL stage of the primary studies. We attribute the under-reporting of the IDEAL framework in the included SRs mainly to the lack of awareness of its existence and its value, but partly also because SRs are perceived as outside of the scope of the framework.86 We suggest including SRs in the IDEAL framework as they have the potential to inform the evaluation and assessment phases depending on the robustness of the SRs and the quality of the primary studies.
Our findings also have important implications for journals considering SRs, in general, and on this topic in particular. Since the availability of the Quality of Reporting of Meta-analyses (QUOROM) statement in 1999 and the PRISMA statement in 2009,25 87 most journals require adherence to these guidelines to improve the reporting quality of SRs. Despite the enforcement of these reporting requirements, confidence was low in all included SRs in our study. Therefore, to enhance the confidence in the evidence synthesized by SRs, journals may consider requiring authors to abide by a methodological assessment tool (eg, AMSTAR 2) in addition to PRISMA guidelines. We suspect that many of these additional requirements will not be burdensome to authors. In our umbrella review, many of the unfulfilled requirements for AMSTAR 2 were administrative in nature (eg, presence of a protocol, availability of a list of excluded studies and reasons for exclusion) and can easily be addressed to increase the transparency and raise the confidence level in future SRs.
To our knowledge, this is the first umbrella review evaluating all published SRs comparing CA and conventional total joint arthroplasty. We followed the umbrella review guidelines strictly, and conducted the study selection, data extraction, and quality appraisal in duplicates. We then summarized the evidence in a structured way. We also assessed the overlap bias, an important step usually under-reported by umbrella reviews.24 Nevertheless, we must mention the limitations of our study. First, although we developed a protocol to help plan for our review, we did not register it, a step that would have provided more methodological strength for our review. Second, despite extensive efforts to identify all relevant SRs without language restrictions, it is still possible that we missed some SRs. Third, due to the absence of a reliable method of quantitatively synthesizing the evidence from multiple meta-analyses, we narratively summarized the evidence. Fourth, our extraction and assessment relied on the available manuscripts and supplemental materials. While we tried to contact the journals and authors inquiring for specific missing information, not all of them responded with clarification or additional information. Therefore, we cannot eliminate the possibility of underestimating the methodological quality for some studies because of the lack of access to relevant information.
Based on the findings of this review, we call for high quality SRs that can be used with great confidence to inform the decision on using CA TKA and THA. In addition, we encourage journals publishing SRs to use a methodological assessment tool to assess the quality of SRs. Finally, we advocate for standardization of the reported outcome measures for CA TKA and THA to facilitate evidence synthesis and outcome research.