How We Review and Rate a Practice From Start to Finish
To be included on
CrimeSolutions.gov, practices undergo a seven-step review and evidence-rating
process. On this page you'll find details of each step in this process. You also will find information on how and when we determine to re-review a Practice.
1. Preliminary practice identification
identified for potential inclusion on CrimeSolutions.gov through:
- Literature searches of relevant databases, journals and publications, including:
- Social science databases using keywords identified in the areas of criminal justice, juvenile justice, and victims of crime;
- Journals (including peer-reviewed journals) and other relevant resources; and
- Other web-based
databases of effective programs and meta-analyses of evaluated programs.
- Nominations from the field: See how to nominate a
practice here: Nominate a Program or Practice for CrimeSolutions.gov.
2. Initial practice screening
After practices are
identified, research staff review practice materials to determine whether the
goals of the practice fall within the scope of CrimeSolutions.gov. To fall
within the scope, the practice must:
to prevent or reduce crime, delinquency or related problem behaviors (such as
aggression, gang involvement or school attachment);
to prevent, intervene or respond to victimization;
to improve justice systems or processes; and/or
an offender population or an at-risk population (that is, individuals who have
the potential to become involved in the justice system).
Prevention practices not
explicitly aimed at reducing or preventing a problem behavior must apply to a
population at risk for developing problem behaviors.
3. Practice definition and literature search
Based on the
identified meta-analyses, CrimeSolutions.gov Lead Researchers develop a clear
definition that includes definite boundaries for what does and does not
constitute the practice. Meta-analyses vary in scope such that some
meta-analyses may address multiple practices or multiple populations. For
example, CrimeSolutions.gov often summarizes results separately for juveniles
and adults, but both may be addressed within the same meta-analysis. The
practice definition is used to develop search terms to identify additional
meta-analyses, systematic reviews, and reviews in the practice area.
CrimeSolutions.gov does not limit the number of eligible meta-analyses that may
be reviewed for any given practice.
4. Initial evidence screening
All meta-analyses included in the evidence base
for practices must meet the following minimum criteria:
- Intervention. The meta-analysis must include at least two studies of the practice of interest.
- Aggregation. The meta-analysis must aggregate the results from at least two studies.
- Primary aim of the intervention. The programs included in the meta-analysis must aim to address the goals identified in the initial screening stage (Step 2 above).
- Literature search. The literature
search that guided the inclusion of primary studies in the meta-analysis
must include at least two sources and must provide evidence that
unpublished literature was sought in the search.
- Primary outcomes. The
meta-analysis must report on at least one eligible outcome related to
crime, delinquency, overt problem behaviors (e.g., aggression, gang
involvement, substance abuse), crime victimization, justice system
practices or policies, or risk factors for crime and delinquency.
- Control groups. All studies
included in the meta-analysis must include an appropriate control,
comparison or counterfactual condition, or the meta-analysis must analyze
these studies separately from those that appropriate counterfactuals.
- Reporting of results. The meta-analysis must report effect sizes that represent the magnitude of the treatment effect.
- Combining effect sizes. When an average effect size is reported for multiple studies, all effect sizes in the combination must address the same type of relationship.
- Publication date. At least 50 percent of the studies included in the meta-analysis must be published or otherwise available on or after 1980.
- Age of samples. Samples included in the meta-analysis must be restricted to either adults or juveniles, or mean effect sizes for adults and juveniles must be reported separately.
5. Expert review
Once the Lead Researcher selects
the eligible meta-analyses that will comprise the practice’s evidence base,
Study Reviewers begin the practice evidence review using the practice scoring
instrument to assess the quality, strength and extent to which the evidence
indicates that the practice achieves its goals. Each meta-analysis within the
evidence base is assessed by at least two Study Reviewers. The scoring
instrument indicates the overall rating for each outcome that is reviewed.
See the Practices
Scoring Instrument (PDF)
The scoring for practices consists of two parts looking at overall quality and internal validity:
Part 1: Overall Quality is assessed for each eligible meta-analysis included within the evidence base for a practice. The Study Reviewers make this assessment based on information about the methods and procedures used in each meta-analysis. Multiple items are scored to determine overall quality and these items are weighted differently based on their importance.
|Meta-analysis' Overall Quality
||Assesses the extent to which meta-analysis authors were attentive to the methodological quality of the primary studies included in the meta-analysis.
||Assesses multiple elements related to the quality of the statistical analysis used to calculate and report effect size estimates.
- Handling dependent effect sizes
- Effect size reporting
- Weighting of results
- Analysis model
- Heterogeneity attentiveness
|Eligibility and Search
||Assesses the degree to which the meta-analysis provides a clear statement of the inclusion and exclusion criteria for selecting studies to be included, and whether the literature search was comprehensive and not limited to commercial publishers.
- Eligibility criteria
- Comprehensive literature search
- Grey literature search
|Reliability, Outliers, and Publication Bias
||Assesses methods used in the meta-analysis to extract data from primary studies, account for extreme scores, and account for biases towards large and statistically significant effects in published findings.
- Coder Reliability
- Outlier Analysis
- Publication Bias
1 Scores for elements in each dimension are combined and then weighted according to their importance to arrive at an Overall Quality score.
Study Reviewers assign numerical values to each element in the practice scoring instrument. The elements include a definition and other guidance reviewers consider when rating the elements. See the Practice Scoring Instrument for guidance on each element.
Part 2: Internal Validity is
assessed for each outcome within each eligible meta-analysis for a practice.
Internal validity refers to the extent to which changes in the outcome can be
attributed to the intervention, program or practice. Stated differently,
internal validity refers to the extent that the research design is free from
threats that may bias the estimated effect. There may be multiple outcomes that
are independently assessed within each meta-analysis.
Randomized controlled trials
(RCTs) have the strongest inherent internal validity. CrimeSolutions.gov uses
the extent to which average effect sizes are based on results from RCTs to
assess internal validity. Meta-analyses based on a higher proportion of RCTs
generally receive higher internal validity scores. Quasi-experimental designs
may also be included in eligible meta-analyses for CrimeSolutions.gov, but the
internal validity ratings will generally be lower as the proportion of
quasi-experimental designs increases. See the Practice
Scoring Instrument for guidance on how internal validity
6. Outcome classification
To arrive at the outcome rating, the internal validity score for each outcome coded above is combined with the overall quality rating for the meta-analysis from which it was extracted. This information, along with information about the direction and statistical significance of the mean effect size, is used to categorize each outcome into one of following:
- Class 1 – Strong evidence of a positive effect
Highest quality evidence with statistically significant average effect size favoring the practice
- Class 2 – Moderate evidence of a positive effect
Moderate quality evidence with statistically significant average effect size favoring the practice
- Class 3 – Negative effect
Moderate to high quality evidence with statistically significant average effect size in the opposite direction of the intended effect for the practice
- Class 4 – Non-significant or null effect
Moderate to high quality evidence and the average effect size is not statistically significant from the comparison group
- Class 5 – Inconclusive evidence
Limitations in the study design preclude CrimeSolutions.gov from reporting further on these average effect sizes regardless of their direction or statistical significance.
Discrepancies among Reviewers: In the event that there is a classification discrepancy between the Study Reviewers, the Lead Researcher will work to achieve a consensus classification. If necessary, the Lead Researcher will also review the study and make a final determination on the classification.
For more information about scoring and classifying outcomes, see the Practice Scoring Instrument.
7. Outcome evidence rating
In cases where there is only one eligible meta-analysis for each outcome in a practice, the conversion from outcome classification described above to final outcome evidence rating is straightforward.
|Outcome Evidence Rating
||Determination of Outcome Evidence Rating
||Outcomes in Class 1
||Outcomes in Class 2
||Outcomes in Class 3 or 4
||Outcomes in Class 5 – does not receive an icon
In some cases, multiple meta-analyses provide information about the same outcome. For each meta-analysis, that outcome is first rated separately for quality and validity as described above and then classified according to the five classes in Step 5 above. An additional step is needed to arrive at a final evidence rating for that outcome: Outcome findings are combined from multiple meta-analyses using the process below. CrimeSolutions.gov does not limit the number of eligible meta-analyses that may be included.
Each class of outcomes is weighted as follows:
- Class 1 = 3
- Class 2 = 1
- Class 3 = -3
- Class 4 = 0
- Class 5 is not counted
Ratings for the same outcome from multiple meta-analyses are summed and averaged. The final evidence rating for the outcome is assigned based on this score.
- If the averaged points ≥ 1.50, then final outcome rating is Effective.
- If the averaged points are ≥ 0.50 and ≤ 1.49, the final outcome rating is Promising.
- If the averaged points ≤ 0.49, the final outcome rating is No Effects.
|Outcome Evidence Rating
||Determination of Outcome Evidence Rating
||The averaged points ≥ 1.50
||The averaged points ≥ 0.50 and ≤ 1.49
||The averaged points ≤ 0.49
*A single study icon () is used to identify practice outcomes that have been subjected to one meta-analysis. A multiple studies icon () is used to represent a greater extent of evidence supporting the evidence rating. The icon depicts practice outcomes that have more than one meta-analysis in the evidence base demonstrating effects in a consistent direction.
Read more about: the Practices Scoring Instrument.
a Practice and Updating a Rating
consider re-reviewing a practice's rating when:
meta-analyses, or meta-analyses not previously identified, are found that
meet the CrimeSolutions.gov criteria. This may include meta-analyses that have
been updated since the original review.
new materials may or may not be sufficient to warrant a new evidence rating. If
a Lead Researcher determines that there is sufficient evidence in the new
materials to warrant another review, then the new information is sent to Practice
Reviewers for assessment. Even if the outcome rating does not change, the new evidence
and materials may be included or referenced on the profile page.
more information: Inquiring
About or Appealing an Evidence Rating.