|
Home
Guidelines for Conducting a Practice Analysis for Post-Graduate
Clinical Residency and Fellowship Programs
Introduction
The purpose of this document is to assist developing
post-graduate clinical residency and fellowship programs in validating
the content of their curriculum. Curriculum content validation is
required for a program to become credentialed by the APTA, and is done
via a practice analysis. A practice analysis is a systematic plan of
study of the professional practice behaviors and knowledge that comprise
a specialty area of practice. The purpose of the practice analysis
is to collect data that reliably and accurately describes what knowledge
and skills are necessary to practice in a given area of
specialization.
The following guidelines were amended by the
Committee for Credentialing Postprofessional Clinical Residency &
Fellowship Programs from a document that was developed by the American
Board of Physical Therapy Specialties (ABPTS) to assist specialty
councils in validating or revalidating the specialist certification
examinations, through a practice analysis.
Steps in the Process of Developing a Practice
Analysis
Each residency or fellowship program should complete
the following steps when conducting a practice analysis.
All documents related to the implementation of the
practice analysis, including all data collected, should be carefully
archived for the life of the curriculum from which it is drawn.
These data will serve as the rationale and substance of the residency or
fellowship program's curricular content.
1.
Identify a practice analysis coordinator.
The role of practice analysis coordinator may be assumed by a
subject matter expert (SME) in the specialty area or an individual with
expertise in the conduct of practice analysis. The practice analysis
coordinator will serve as the project manager who will direct the
practice analysis activities so that it can be completed within the
specified time frame.
NB: ABPTS maintains a roster of individuals
who are qualified and willing to serve as consultants who
provide assistance with practice analysis activities.
The responsibilities of the practice analysis
coordinator is as follows:
A. The practice
analysis coordinator forms a project team, which includes subject matter
experts (SMEs). SMEs are individuals who have been identified as having
recognized expertise regarding the knowledge, skills, and abilities
required for practice in the specialty area. While there is no
minimum for the number of members in the SME group, the group must
represent the spectrum of the specialty area with diverse origins of
practice, practice setting, geographic area, gender, and race.
B. The project
team develops the content of the pilot survey instrument, reviews the
data from the pilot survey instrument, develops the practice analysis
survey, and interprets the practice analysis survey results, as
described in the subsequent sections.
C. The practice
analysis coordinator works with the project team to develop competency
statements that describe
(a) the practice process,
(b) current best practice and knowledge, and
(c) skills specific to the specialty
practice.
For residency or fellowship programs that are
validating their curriculum via a practice analysis, many of these
competency statements will mirror the behavioral objectives of the
curriculum. These competency statements become the basis for the
practice analysis survey. See section "b."
D. The practice
analysis coordinator together with the other members of the project team
provide input for the pilot survey. The APTA's Specialist Certification
and Research Departments may be of assistance in planning activities
related to the formatting and printing of the pilot survey, mailing the
survey, and data analysis. Interpretation of the survey results
are the responsibility of the project team.
E. The practice
analysis coordinator and the project team prepare the pilot and final
practice analysis surveys.
F. The practice
analysis coordinator assists the project team to analyze and interpret
the survey results by developing and applying consistent decision rules.
The practice analysis coordinator assists the project team to interpret
the survey data. The results of the survey analysis are used to
determine which knowledge, skills, and abilities (KSAs) or competencies
are to be included in the Specialty Practice Content
As might be expected, the need for additional
support will vary for each practice analysis depending on a number of
factors, including the expertise of practice analysis coordinator and
subject matter experts in research and practice analysis.
2.
Develop the pilot survey
The pilot survey is the first draft of the practice
analysis survey. The purpose of conducting a pilot survey is to insure
clarity of the survey questions prior to distributing the full practice
analysis survey to the entire sample population. In addition, the
practice analysis coordinator and the SMEs may use information collected
from the pilot survey to determine whether any new competencies should
be incorporated into the practice analysis survey, and whether the
survey should be subdivided in order to reduce the time required to
complete it.
The pilot survey should be developed considering the
following elements:
(1) purpose
of the pilot study
(2) sample
size for the pilot study
(3) plan
for data entry and analysis
(4)
plan for achieving the desired survey return rate
The pilot survey instrument must assess existing
competencies (knowledge, skills, abilities) in order to determine if
they are important to specialty practice. For residency or fellowship
programs, these competencies are the statements developed from the
curriculum?s behavioral objectives (see section 1.C above). In
addition, the survey should include any new competencies identified by
processes such as SME groups. New competencies can be identified from
the pilot survey respondents by including open-ended questions asking
for additional knowledge, skills or abilities than those listed in the
survey.
The survey must include an assessment of the
importance of each competency and an assessment of the frequency with
which practitioners perform each activity. An assessment of the
criticality of each task/activity must also be included. ABPTS has
developed standard wording for importance, frequency, and criticality
scales that specialty council?s use for their pilot and practice
analysis surveys. This information can be obtained from the appropriate
Specialty Council or from the ABPTS.
The language of the survey questions and Specialty
Practice Content should be consistent with the terminology of the
Guide to Physical Therapist Practice.
Consideration should be given to developing a survey
that could be divided into sections such that the most rapidly changing
knowledge, skills and abilities could potentially be revalidated.
3.
Field test the pilot survey.
The size of the sample for the pilot surveys for
practice analyses conducted within the last five years range from six
(6) to forty-five (45). ABPTS recommends that the pilot survey be
field tested with no fewer than twenty-five (25) individuals from varied
geographic and demographic populations.
4.
Analyze pilot data.
Data should be analyzed descriptively by computing
means, standard deviations, and frequency distributions for the three
rating scales (frequency, importance, and level of criticality) for each
of the competencies.
5.
Revise the survey, if necessary.
Examples of revisions that might be required include
changes to improve clarity or the addition of new items based on
responses to open-ended questions.
6.
Submit the revised survey to the Committee.
The Committee must approve the revised survey prior
to fielding.
7.
Conduct the practice analysis survey.
For practice analyses conducted by the Specialty
Councils to validate the specialization exam, the ABPTS requires
representative random samples of both board-certified specialists and
section members. In most cases, surveys should be fielded to 95%
of certified specialists and an equivalent number of non-certified
section members. For the larger specialty areas, ABPTS
recommends that the survey be fielded to no fewer than eight hundred
(800) individuals from varied geographic and demographic populations.
These numbers can be used as guidelines when a residency or fellowship
program is planning and fielding the survey.
A follow-up mailing to individuals who have not
responded to the survey, either in the form of a duplicate copy of the
survey or a reminder post card, is required to increase the response
rate. Lengthy questionnaires should be subdivided into "stand- alone"
portions such that the individual respondents can complete their task
within 60
minutes.
ABPTS recommends a minimum response rate of
50%.
8.
Analyze practice analysis survey results.
Data should be analyzed descriptively by computing
means, standard deviations, and frequency distributions for the three
rating scales (frequency, importance, and level of criticality) for each
of the competencies for the total sample and any appropriate subgroups
(e.g. certification status, gender, age, race). Data should be
analyzed to determine if there are significant differences between
subgroups.
9.
Interpret practice analysis survey results.
The survey results will be interpreted by applying
consistent decision rules to identify the competencies that define
specialty practice. The practice analysis coordinator is
responsible for working with the project team to derive the decision
rules for defining specialty practice. The practice analysis
coordinator may wish to review the technical reports or Description of
Specialty Practice (DSP) of recently conducted practice analyses for an
overview of the development of decision rules.
The results of the survey analysis are used to
determine which knowledge, skills, and abilities (KSAs) or competencies
are to be included in the program?s curriculum. The justification
for inclusion or exclusion of competencies in the curriculum must be
documented.
10. Submit
the full technical report to the Committee as an addendum to the
application for credentialing, Section 3: Curriculum.
The full technical report must provide a description
of every step of the practice analysis. It represents the
permanent record of the practice analysis, which can be used as a
resource for defense of the process, future analyses, etc. The
components described below must be included in the technical report:
(1)
description of all project team members including names, addresses,
credentials, and delineation of their specific involvement
(2) description of the sampling strategy, groups
surveyed, number surveyed, return rate, follow-up procedure for
non-respondents, and any demographic data depicting the respondents
(3) copy of the pilot survey instrument
(4) description of responses to the pilot survey
(5) description of changes made to the pilot survey
with a rationale for the changes
(6) copy of the practice analysis survey instrument,
including instructions to the respondents and cover letters
(7) description of the rationale for the choice of
measurement scales (frequency, importance, criticality.)
(8) copy of raw data
(9) description of data analysis including tables
and/or graphs, and any sub-sample analysis (eg, ratings of certified
specialists vs. non-certified specialists)
(10) explanation
of how the results of data analysis were used to determine which
competencies were included in the curriculum
(11) conclusions
with statements about the committee?s confidence in the practice
analysis process highlighting the strengths of the practice analysis,
problems with any portion of the analysis, and recommendations for
future practice analyses.
For more information on the establishment of a
clinical residency or fellowship program, contact the Professional
Development Department at 703.706.8514 or Prof_Dev@apta.org.
|
