1.  Assessor's screening questions

These initial questions allow the appraiser to assess quickly whether the review contains sufficient detail to enable a detailed appraisal and, if not, to justify the extra effort involved in the appraisal of a less comprehensible review.

• On first reading is there sufficient information to make a detailed appraisal?   Yes/No
•  Is there no obvious alternative review of better quality available?   Yes/No
  
  

  ---

2.  In relation to what question is this review being appraised (target question)?

In the context of an ARIF request, the target question is the question posed by the requester and should be defined in terms of the population, intervention and outcomes to which it refers and the type of question it is. If there is no such underlying request, the appraisal is conducted as though the specific review question, detailed below, is also the target question.

3. Has a clear question been defined (review question)?

The review question is the question posed by the review and again should be defined in terms of the question type and the population, intervention and outcome to which it refers. Comments should be made as to whether the question is sufficiently well defined to allow the review to be executed systematically (internal validity) and, when appropriate, whether it achieves a good fit with the target question (external validity). Also at this point it is useful to note any additional effects, related issues and perspectives which have not been accounted for.

4. What are the implications for the validity of the review of the type and range of study designs included?

The types of study designs which were included should be stated and the implications for the overall validity of the review commented on in the context of an appropriate hierarchy of evidence.

5. Were inclusion/exclusion criteria clearly stated?

The inclusion and exclusion criteria should be listed and comments made on whether they are clearly stated and consistent with the review question.

6. Was the search strategy adopted likely to have missed many potentially relevant studies?

The search strategy should be recorded and comments made on its comprehensiveness i.e.

• databases searched
• dates searches undertaken
• language restrictions
• reference lists searched
• ongoing trials registers searched
• contact made with authors or relevant manufacturers

Also any elements of the search strategy that might potentially introduce bias should be noted.

7. How were inclusion/exclusion criteria applied?

This is not an assessment criterion, simply useful information for an assessor of a review to note. The objective of most systematic reviews is to be as comprehensive as possible in the identification of all the "included studies" and as a result the search strategy defined must always be highly sensitive.  However a balance needs to be struck between the sensitivity and specificity of the search.  A sensitive search is less likely to miss relevant studies but may yield a vast amount of irrelevant material and a large amount of effort will then be needed to apply the inclusion/exclusion criteria, with potential for error due to reviewer fatigue.  A search with high specificity may yield smaller reference lists but a search that is too specific may not identify relevant studies that have been mis-catalogued within the databases.  To judge the accuracy of the search a QUORUM flow diagram is useful.  The results section may also contain a funnel plot which gives an estimate of the number and size of missing studies.

It is also useful at this point to note whether a list of excluded studies was available and whether any excluded articles were examined.

Back to Top

8. Was the validity of included studies assessed?

Although the importance of this step is generally recognised, consensus on exactly what should be done has, as yet, not been achieved. The way in which the validity of the included studies was taken into account and how information on the validity of the included studies was used should be noted i.e.

• Was an assessment of quality using a recorded checklist (e.g. cochrane checklist) or scoring system (e.g. Jadad scale) made?
• Was there more than one assessor of quality?
• did the review authors report the results of their quality assessment?
• Were any quality issues discussed?
• Were there any quality issues that could impact on the results of the review?
  
  

  ---

9. Was the process of data abstraction adequate?

The way in which the relevant data items were abstracted should be recorded. Comments should be made on whether this was consistent with the review question and what the implications were for the validity of the review.

10. Were the important steps in the review reproducible and bias free?

The repeatability of the following steps should be considered:

• searching for all potentially relevant studies
• applying study inclusion/exclusion criteria
• assessment of validity of included studies
• data abstraction

Also comment should be made on the impact of these processes on the overall validity of the review.

11. What was(were) the relevant and justifiable review bottom line(s) - as stated in the review?

The summary estimate of effect or key results for each outcome, subgroup or comparison examined by the review should be stated. Comments should be made on whether meta-analysis was used; if so whether this was appropriate in relation to the homogeneity or heterogeneity of the results; and the role of chance, as indicated by the confidence intervals or equivalent. Any other conclusions reported which seem reasonable given the data presented, should also be noted.

12. Was the review up-to-date?

The date cut-offs used for ascertainment of relevant literature should be identified.

13. General comments

Comment should be made on the general quality of the review including the elements it incorporates which might make it systematic. The review and its potential uses should be categorised.

14. Recommendation For implementation of this review by purchasers

Suggestions should be made on how a purchaser of health care might best use the review.

Back to Top

Back to ARIF Critical Appraisal Process