Best Evidence Medical Education
Tay Park House, 484 Perth Road, Dundee DD2 1LR, UK
BEME Coordinator: Mrs Pat Lilley
Tel: +44 (0)1382 381967; Fax: +44 (0)1382 381987
Email: p.m.lilley@dundee.ac.uk http://www.bemecollaboration.org
Guide for Topic Review Groups
on carrying out BEME Systematic Reviews
Version 2 – updated February 2003
The BEME Collaboration welcomes approaches from groups interested in conducting a BEME systematic review in a
specific area. These notes are provided to help familiarise prospective groups with the BEME review process. The
notes will be regularly updated to incorporate experience gained from the initial pilot reviews now under way. BEME
welcomes feedback from Topic Review Groups as to how the review process may be improved.
The BEME Collaboration is available to give advice and guidance to Topic Review Groups throughout the course of the
review, and can offer specialist help from the BEME Information Scientist. Groups already working are accumulating
valuable experience which they are pleased to pass on to new groups.
The BEME website contains information on the BEME Collaboration, its structure and its activities to date. The site will
be updated with a variety of information including progress made by the initial review groups.
Comments and questions may be sent to the BEME Collaboration (contact details above).
BEME Steering Group
February 2003
1
Contents
Page
A
What is a BEME Systematic Review?
3
B
Steps in the BEME Review Process
1
Selection of a topic/question for a systematic review
3
2
Forming a Topic Review Group (TRG)
4
3
Registration of a topic/review question
5
4 The
literature
search 5
5
Preparation of a review protocol
6
6
Coding sheets and procedures for extracting evidence
and data
7
7
The draft report and the consultation process
7
8
Final report
7
9
Updating the review
8
10
Prototype timetable for the review process
8
Figure I: Steps in the BEME Review Process
9
C
Training and support for Topic Review Groups
10
Appendix I Responsibilities of the BEME Infrastructure
11
and Topic Review Groups
Appendix II
How to Prepare a BEME Review Protocol
15
Appendix IIIA
Prototype BEME Coding Sheet and Notes
Separate
Appendix IIIB
Coding Sheet Description and Guide
documents
2
A
What is a BEME Systematic Review?
1
A BEME systematic review is a logical, explicit and comprehensive appraisal of available information to
determine the best evidence relating to an issue in medical education.
2
A BEME systematic review is designed to assist individual teachers, institutions and national bodies to make
informed decisions about their educational practice.
3
A BEME systematic review is a description and evaluation of evidence pertinent to a clearly formulated
topic/question that uses explicit methods to systematically identify, assemble, critical y analyse and
synthesise information relevant to the topic of the review.
4
The BEME strategy for review is inclusive and open to a wide range of investigative methodology.
5
Each summary, pertinent to a clearly formulated topic/question, wil be user-friendly, enabling the
practitioner to judge and employ evidence according to his or her individual criteria and context.
6
A BEME systematic review must be registered with and approved by the BEME Collaboration and is produced
by a BEME Topic Review Group (TRG) following accepted BEME review procedures.
B
Steps in the review process
1
Selection of a topic/question for a systematic review
The principal emphasis in the selection of a topic/question for systematic review is a ‘bottom-up’ approach, with
proposals coming from prospective reviewers. The review group will work with the BEME information scientist to define
the precise scope and nature of the topic/question being considered for review. It is necessary to describe the
topic/question as accurately as possible to ensure that the best evidence is retrieved and considered for the review.
Points to consider are:
1.1
A BEME review topic/question should focus on and inform on practical issues or problems faced by the
teacher or institution in their day-to-day practice. For example, the research topic/question should be
phrased in such a way as to provide information on how the teacher or institution should respond to the
adoption of a new teaching approach in their own context.
1.2
BEME review questions may be most helpful where they illuminate the topic area. A valuable question will
usually explore aspects of the review topic and will not necessarily be answered with a simple yes or no.
1.3
The review topic/question should be precisely defined. Typically, a review topic/question will identify:
• population/participants e.g. undergraduate or postgraduate students
• the activity under investigation e.g. the timing of feedback in assessment
• outcomes e.g. change of attitudes or knowledge
1.4
Examples of research topics/questions for systematic reviews currently under way are:
• Systematic Review of High Fidelity Simulation in Medical Education (Miami Group)
3
• What conditions are necessary or required for assessment and feedback to enhance physician
improvement? (ABIM Group)
• Predictive values of assessment measurements obtained in medical schools and future performance in
medical practice (AGU Group)
The terms used in the question should be defined: eg, What is a high-fidelity simulator?
1.5
The BEME Collaboration Information Scientist can assist by running a scoping search. These broad, simplified
searches are conducted to determine the size of the body of literature relevant to the topic as initially
defined. The scoping search is not comprehensive, nor does it consult all sources.
1.6
Following the scoping search, it may be advisable either to broaden or narrow the scope of the proposed
review topic/question in order to ensure it is manageable whilst at the same time generating sufficient
studies for the review. Several iterations may be necessary before the final topic/question is defined.
2
Forming a Topic Review Group (TRG)
2.1
The TRG wil largely propose its own composition. The BEME Review Editorial Group (REG) strongly
recommends that the following are taken into account:
The TRG through its members should have:
• knowledge of the specific area being reviewed
• appropriate knowledge of medical education
• appropriate knowledge of research methods
• appropriate knowledge of the process of conducting systematic reviews
It is helpful if the TRG has access to:
• information technology skills (see 4 below, and Appendix I)
• clerical/database handling skil s
• a research librarian
In general it is recommended, although not essential, that the TRG has international composition.
2.2
The recommended number of TRG members will depend on the size of the review, the location of proposed
group members and the resources available. It is recommended that between 6-12 is an optimum number.
3
Registration of a topic/review question
3.1
Once accepted by BEME, the review topic/question and the TRG will be entered into the BEME Register of
Reviews on the website. This will alert all interested parties to the proposed review. The TRG will be asked to
complete a topic registration form giving:
• names and affiliations of review group members;
• name and full contact details of lead reviewer;
• summary of the proposed review
3.2
The review question will be registered exclusively to the TRG for an initial period of one year. Within six
months, BEME expects to receive a draft protocol (see 5 below) including a proposed timetable for
4
completing the review (see 10 below). If a TRG does not submit a draft protocol within the first year, and is
unable to demonstrate adequate progress with a registered review, the review topic may be released to
another group.
3.3
If a prospective group expresses an interest in a topic already registered and on which a TRG is actively
working, the registered TRG will be asked to discuss the topic with the proposed new group to see if
collaboration is feasible. If collaboration is not appropriate the proposed group will be asked to consider
resubmitting a review topic/question in a related or different area.
4 The
literature
search
4.1
The TRG should devise a robust search strategy appropriate to the review topic/question, with advice from
the BEME Information Scientist:
4.1.1
The literature searches should be sensitive (broad and inclusive) in design so that no evidence is
overlooked.
4.1.2
Access to the search process should be readily available and easy to verify They should be detailed
in the protocol (see 5 below) and appended to the draft and final reviews (see 7 and 8 below).
4.1.3
Searches should be run across the various databases, specialist registers and the web; hand-
searching should be carried out across selected journals, the grey literature (eg, conference
proceedings, newsletters) should be consulted and decisions taken on the methods the TRG plans
to use to contact experts in the area [see BEME Guide on Systematic searching for evidence in
medical education, to be published in April 2003).
4.1.4
The TRG should determine the years to be covered by the search, depending on the topic area to
be reviewed. Whilst a ten-year period is suggested by the BEME Collaboration, this may vary
according to the number of studies in the topic area, the relevance of papers published outside the
ten-year limit, etc. The TRG should justify its decision in relation to the years to be covered. It is
strongly advised that a cut-off date for studies to be considered in the review should be set. The
time period covered should be clearly stated in the protocol and report (see 4.1.7 below).
4.1.5
The BEME Collaboration recommends that as far as is reasonably possible, studies should not be
excluded on the grounds of either geography or language. Access to and translation of studies not
published in English is the responsibility of the TRG and will depend on its resources. The BEME
listserv could be used to try to locate a potential translator, or to trace a difficult-to-obtain
article.
4.1.6
The search process will involve determining the final individual search syntax for each source,
executing the searches, and collating the results (removing duplicates, etc.)
4.1.7
Depending on the time taken to carry out the review, searches may have to be rerun to locate any
newly published studies that may be relevant to the review.
4.2
Initial appraisal of literature search: The TRG should assess citations generated by the search and decide, on
the basis of the title/abstract, whether the citation is relevant to their topic. An evidence-based search will
tend to have high sensitivity to reduce the possibility of missing evidence, therefore it will produce clearly
irrelevant items that can easily be identified and eliminated. If there is any doubt as to the relevance, items
should not be eliminated to ensure comprehensiveness. The fulltext article for all potentially relevant
citations should be retrieved by the TRG. If it is unclear whether a citation is relevant (eg, if no abstract is
provided, or if relevance is unclear from the abstract), the fulltext article must be retrieved to be properly
5
assessed. Note: TRGs have reported instances of poor quality abstracts which make it difficult to determine
the relevance of the study to the review topic/question.
4.3
The TRG should decide on its policy for review of citations and coding of articles (see 6 below). BEME
recommends that there should be two-person agreement on each aspect coded. In the event of difference of
opinion, a third person should decide. A record of differences of opinion should be kept. One TRG has
reported poorer inter-rater agreement on qualitative data than on quantitative data.
4.4
The final inclusion and exclusion criteria should be agreed by the TRG and stated in the protocol (see 5
below).
5
Preparation of the review protocol
The protocol describes the review's context and objectives, defines the review's topic/question and remit, and outlines
the methodology to be employed to critically appraise and analyse evidence from the selected studies (see Appendix II).
5.1
Within an agreed period, usually a maximum of 6 months from registration of the review question, the TRG
should submit a draft review protocol.
5.2
The draft protocol will be reviewed by the REG, who may submit it for peer review as appropriate.
5.3
The draft protocol will be approved by the REG, possibly following revisions by the TRG if required.
5.4
The approved protocol will be posted on the BEME website.
6
Coding sheets and procedures for extracting evidence and data
The coding process identifies those studies that will ultimately be used in the review, and those that will be excluded if
they do not meet the scope of the review or quality standards, according to the criteria set out in the protocol (see
Appendix II). Excluded studies and the reasons for their exclusion should be recorded and this information should be
appended to the final review.
6.1
A prototype coding sheet is available to all groups for adaptation to meet the needs of their review in terms
of the data to be extracted (Appendix III). Adaptations made to the prototype coding sheet by the TRG should
be documented. Initial groups have reported the need for extensive changes to the BEME prototype coding
sheet to meet their needs. A BEME working group will be set up to look at this aspect, based on input from
TRGs.
6.2
When the TRG finalises its coding sheet an electronic form will be provided by the BEME Collaboration for use
by the TRG, linked to a customised database specific to the TRG and provided by the BEME Collaboration. If
TRGs wish to use their own database they should check its compatibility with the Dundee-designed database
to avoid future problems of consolidation of data.
6.3
All studies retrieved by the systematic searches wil ultimately be stored in the TIMELIT database which will
form a repository for studies generated through BEME reviews.
7
The draft report and the consultation process
7.1
Following a rigorous and objective synthesis of the evidence base the TRG wil produce a draft report with its
preliminary conclusions, which will be submitted to the REG for review.
6
7.2
The draft report will be posted on the BEME website for wider comment.
7.3
Following approval by the REG, the draft report will be presented for discussion at either a specially
convened working group or at a session at one or more international meetings (eg AMEE or AAMC, and in
some cases at medical specialty meetings) and feedback relating to possible modifications will be requested.
The aim is to identify any relevant information that may have been missed from the review, to provide input
on the interpretation of the evidence and to comment on the user-friendliness of the review.
8 Final
report
8.1
Following the consultation process detailed in (7) above, the final report will be prepared by the TRG and
submitted to the REG for approval.
8.2
The approved review will be published on the BEME website, under the authorship of TRG members.
8.3
The TRG is encouraged to prepare papers based on the review’s findings for submission by the authors to
relevant journals. The BEME Collaboration must be acknowledged in all papers submitted for publication.
8.4
The database of studies relevant to the review will be available via the BEME website.
9
Updating the review
9.1
Reviews should normally be updated within three years of the date of the review. The TRG will have first
option of updating the review.
9.2
If the TRG is unable to provide resources for an update, any other group may request permission from BEME
to update the review.
10
Prototype timetable for the review process
The steps in the BEME Review Process are given in Figure 1. Whilst each review will vary, the following is the anticipated
duration and recommended maximum for each stage:
Anticipated Recommended
duration
maximum
Forming TRG and defining question:
4-6 weeks )
Registration of topic:
2-4 weeks )
6 months from
Literature
search:
2-4
weeks
) registration
Preparing
protocol:
1-2
months )
Data abstraction:
2-4 months
)
8 months from
Draft
report:
2-3
months ) protocol
Final
report: 1-3
months ) 4
months
from
draft report
Updating review:
3 years from
7
final report
C
Training and support for Topic Review Groups
BEME aims that Topic Review Groups should become self-sufficient in terms of developing the expertise needed to
conduct BEME systematic reviews. Support and advice is readily available as follows:
• the BEME Steering Group will advise on matters relating to BEME policy generally;
• the BEME Review Editorial Group should be consulted on matters of policy specifically relating to conducting
systematic reviews;
• the BEME Information Scientist may be contacted for advice on issues relating to literature searches;
• TRGs already operating are available to pass on the valuable experience they have gained to date.
Printed resource materials
A BEME Guide Systematic Searching for Evidence in Medical Education by Alex Haig is scheduled for completion in April
2003 and will then be made available to Topic Review Groups. The guide outlines the chal enges presented by searching
for evidence in medical education and comprehensively covers relevant databases, websites and internet searching,
and grey literature.
Workshops
Workshops are run at major medical education meetings (eg, AAMC, AMEE, ASME) on practical aspects relating to
conducting systematic reviews. A one-day pre-conference workshop ‘Finding and appraising evidence in medical
education’ is being run by Alex Haig and Marshal Dozier at the AMEE Bern Conference (www.amee.org).
Training workshops may also be run either by the BEME Information Scientist or other members of the BEME Steering
Group at the specific request of a Topic Review Group at a location of their choice. Because of limited resources, Topic
Review Groups must allocate funding for a Training Workshop.
8
Figure 1
Steps in the BEME Review Process
Submission of Outline Proposal by
Topic Review Group
Acceptance by Review Editorial Group
Entered in BEME Register of Reviews on BEME
website
Submission of Protocol
including
Review Topic/Question
Approval by Review Editorial Group
Entered in BEME list of
approved Protocols on website
Draft Report of Systematic Review
Consultation Process
Final Report of Systematic Review
Entered in BEME Register of Reviews on BEME
website
Publications
Updating of Review
9
Appendix 1
Best Evidence Medical Education (BEME)
Responsibilities of BEME Collaboration infrastructure
and Topic Review Groups (TRGs) in relation to systematic reviews
A
Responsibilities of BEME Collaboration infrastructure
1 General
1.1
Identification of role of the systematic review within overall context and philosophy of BEME
1.2
Consideration of proposed methodology for systematic reviews by the BEME Review Editorial Group (REG)
1.3
Preparation and updating of BEME documentation:
• Information for TRGs on how to conduct a BEME systematic review
• How to construct a BEME protocol
• Responsibilities of BEME infrastructure and TRGs
• Prototype coding sheet
2
Specific to each review
2.1
Recommendation on composition of TRG
2.2
Recommendation on choice of topic and review question
2.3
Assistance from Information Scientist with scoping search
2.4
Registration of topic as official ‘BEME’ review, and group as an official BEME Topic Review Group
2.5
Provision of training to groups, where requested and as funding permits
2.6
Provision of advice on budget for proposed review
2.7
Advise on coding sheet for review, to be developed from BEME prototype
2.8
Advise on defining protocol and approval of protocol
2.9
Provision of BEME database for TRGs to enter data from all studies identified in literature search
2.10
Review draft report prepared by TRG and provide comments where appropriate
2.11 Agree
final
report.
10
3
Publicity and publication
3.1
Disseminate information about BEME review topics and TRGs as appropriate
3.2
Liaise with TRG with regard to publication of the review in appropriate formats, including on the BEME
website.
B
Responsibilities of Topic Review Group
1
Prior to conducting systematic review
1.1
Identify local members and leader for TRG
1.2
Collaborate with BEME REG to identify international reviewers if appropriate
1.3
Formulate the research question for approval by the REG
1.4
Establish communications process between group members and agree requirements for face-to-face or at-
a-distance meetings for duration of review
1.5
Identify funding for the review, taking into account the issues listed in Appendix A.
2
Implementation of systematic review
2.1
Refine question and revise coding sheet on basis of scoping search of titles and abstracts of articles
provided by Information Scientist;
2.2
Prepare review protocol for submission to REG for approval
2.3
Access full-text of articles selected by TRG for review
2.4
Code articles as per protocol and enter information into database provided
2.5
Synthesise evidence and prepare draft report for approval
2.6
Prepare final report for publication on BEME website.
11
Appendix A
(appended to ‘Responsibilities’ document)
Issues for Topic Review Groups (TRGs) to consider in estimating budget for BEME
systematic review
1
Staffing costs and overheads:
• Funding for review group members, information officer, secretarial, administrative and data input as
appropriate, for the duration of the review;
• Consultancy costs for expertise not available within the TRG, eg statistician
• Overheads relating to services provided by institution, eg heating, communications network, space,
central administration.
2
Database searching/document acquisition/translation costs:
• Cost of accessing databases: whilst access to some databases is free, others impose a charge.
• Accessing full-text articles: TRGs will need to access the full text of articles selected for review.
Depending on the extent of the review question this could result in a considerable number of articles to
acquire through inter-library loans.
• Articles published in foreign languages: translation costs may be incurred if articles in languages not
spoken by review group members are identified in the search.
3 Communication
costs:
This will depend on
• the number of group members and their location
• the duration of the review
• the perceived need for face-to-face meetings
• other means of communication. eg teleconferences, mailing costs
4
Equipment and consumables:
• Computers and software
• Printing and photocopying
• Office supplies
(continued)
12
5
Group training costs:
• Funds for training workshop, if required.
6
Contribution to BEME infrastructure costs:
• Funding for staff support is at present provided by NHS Education for Scotland (computing and
information services), and the Association for Medical Education in Europe (administration). A
contribution to central funding from the American Board of Internal Medicine is gratefully
acknowledged. Additional funding is required for further development of BEME.
• It is hoped that review groups, where funding permits, will make some contribution to infrastructure
costs.
13
Appendix II
Draft
BEME Review Protocol
Rationale for BEME Review Protocol
The protocol details the procedures for identifying, evaluating and amalgamating the evidence for a BEME review. The
protocol describes the review's context and objectives, defines the review's question and remit, and outlines the
methodology to be employed to critically appraise and analyse evidence from the selected studies. Setting out an
explicit and thorough protocol before the review commences defines the review process and will help ensure that the
evidence to be considered is systematic and comprehensive and that the possibility of bias is diminished.
When a topic for a systematic review is proposed by a Topic Review Group (TRG) and accepted provisionally by the
BEME Review Editorial Group (REG), the proposed title, summary and group contact details are posted on the BEME
website. The aim is to prevent duplication of effort, and also to identify others who may wish to collaborate in the
review. The TRG is then asked to produce a protocol, set out as detailed below. A draft protocol is submitted to the REG
for comment. The approved protocol, following revisions if requested by the REG, is posted on the BEME website for
wider comment.
Note: This document should be read in conjunction with Guide for Topic Review Groups, and Responsibilities of
BEME Topic Review Groups and the BEME Infrastructure (Appendix I)
Components of a BEME protocol
The protocol comprises the following:
1 Cover
sheet
2
Background to the topic
3
Review question(s) and objectives
4
Search sources and strategies
5
Study selection criteria
6
Procedure for extracting data
7
Synthesis of extracted evidence
8 Project
timetable
9
Conflict of interest statement
10
Plans for updating the review
11
Changes to the protocol
Details of each component are given below.
1 Cover
sheet
The cover sheet should include the following information:
• Review title
• Names and affiliations of group members, with lead reviewer highlighted
14
• Contact details for the lead reviewer
• Sources of support
2
Background to the topic
The review topic should be explained and placed in context to make the rationale and need for the review explicit. It
should contain:
• a brief overview of the research question
• a conceptual discussion relating to the research question
• a statement of the significance of the research question
Reference should be made to any previous reviews of the research topic where these exist and reasons for the need
for the current systematic review.
3
Review topic/question(s) and objectives
A BEME review topic/question should focus on and inform on practical issues or problems faced by the teacher or
institution in their day-to-day practice. For example, the research topic/question should be phrased in such a way as to
provide information on how the teacher or institution should respond to the adoption of a new teaching approach in
their own context.
The review topic/question should be precisely defined. Typically, a review topic/question will identify:
• population/participants e.g. undergraduate or postgraduate students
• the activity under investigation e.g. the timing of feedback in assessment
• outcomes e.g. change of attitudes or knowledge
BEME review questions may be most helpful where they illuminate the topic area.
A valuable question will usually explore aspects of the review topic and will not necessarily be answered with a simple
yes or no eg. What are the aspects of high fidelity simulators that promote effective learning? The terms used in the
question should be defined: eg, What is a high-fidelity simulator? Having defined the question it may be necessary to
broaden or narrow the search (see 4 below).
4
Search sources and strategies
BEME reviews require systematic searches for evidence across a comprehensive range of sources relevant to the
topic. Systematic searching in medical education is particularly challenging given that the evidence is widely dispersed
and very often poorly indexed. An appropriate search strategy should be constructed to ensure the remit of the review
is met and this will be published as an appendix to the completed review.
The TRG should identify the ful range of sources to be consulted including databases and specialist registers to be
accessed, journals to be hand-searched, the grey literature to be accessed and methods that will be used to contact
experts in the field. The TRG should indicate the time limit for the search (see Guide for Topic Review Groups, Section
4). Neither language nor geography should restrict searches.
Depending upon the timetable for the review (see 8 below), it may be necessary to include in the protocol the provision
to conduct updating searches during the review.
15
5
Study selection criteria
The TRG should make explicit the criteria it plans to use for selecting or excluding studies in order that the results are
objective and reproducible, bearing in mind the following:
5.1
As part of the BEME approach, studies should not be excluded solely on the basis of type of study design;
5.2
Where studies are to be excluded for reasons of methodology, these reasons should be made explicit;
5.3
If TRGs intend to set different criteria with regard to time limit, geography or language from the
recommendations given in the document Guide for Topic Review Groups, Section 4, these should be detailed
here.
6
Procedure for extracting data
The prototype BEME coding sheet (Appendix IIIA) is available to TRGs to adapt to meet the needs of the group, and
adaptations made should be documented. The TRG should include a statement of how many group members will code
each article, how the TRG intends to check for inter-rater reliability and the proposed procedure to resolve differences
in coding of studies. Changes made to the recommendations given in the document Guide for Topic Review Groups,
Section 6, should be detailed.
7
Synthesis of extracted evidence
The methodology used in BEME systematic reviews will inevitably evolve during the initial pilot reviews. TRGs should
make explicit the methodological procedures they will use to synthesise the evidence.
8 Project
timetable
TRGs should submit a proposed timetable for the stages involved in producing the systematic review. A prototype
timetable is available for consultation ( see Guide for Topic Review Groups, Section 10). Timetables will vary between
BEME groups, depending on an individual group's resources, its size, geographic distribution of members and
complexity of the topic, etc.
9
Conflict of interest statement
The protocol must include a conflict of interest statement, relating to all TRG members. These include academic,
institutional, political, personal or other conflicts. Special concern must be given to financial conflict of interest, with
the acknowledgement of any financial support, including money, hospitality or subsidy.
10
Plans for updating the review
TRGs should indicate if they intend to update the systematic review after completion and if so, include brief details and
a potential timeline (see Guide for Topic Review Groups, Section 9).
11
Changes to the Protocol
16
Whilst a wel thought out protocol should reduce the need for changes, unanticipated issues may arise and
modifications to the review topic/question, coding sheets and/or protocol may become necessary as the TRG becomes
more familiar with the evidence. Any subsequent changes to the protocol must be carefully recorded, with the reasons
for the changes and the date the changes took place. Significant changes to the protocol must be submitted to the REG
for approval.
BEME Collaboration
March 2002
17
Document Outline
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿð·
- þÿð·
- þÿð·
- þÿð·
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ
- þÿ