FAQs

Following are some frequently asked questions associated with CrimeSolutions.gov:

• Are programs implemented and evaluated outside the United States eligible for CrimeSolutions.gov?
  

  Yes. Programs from other countries are included in CrimeSolutions.gov. These programs must have English language citations and are otherwise subject to the same criteria used for programs located within the United States.

• Can I cite the CrimeSolutions.gov review of my program in my program materials?
  
  Intervention materials, including Web sites, may state that a program has been reviewed and posted on CrimeSolutions.gov. After a program profile has been posted on CrimeSolutions.gov, the program developer/evaluator may Submit Feedback to request the CrimeSolutions.gov logo graphic. The logo may be used as a link to CrimeSolutions.gov or in the program materials. However, the posting of a program summary on CrimeSolutions.gov does not constitute an endorsement, promotion, or approval of the intervention by CrimeSolutions.gov or OJP. Please refer to the What is the purpose of CrimeSolutions.gov? FAQ for additional information.
• Can I share information with the Office of Justice Programs (OJP) about how I was able to successfully use program or other content from the CrimeSolutions.gov site in my community?
  
  Yes! We are very interested in receiving success stories related to the use of CrimeSolutions.gov program evaluations or other content from the site. Using our Submit Feedback form, please provide us with the content you found useful (for example, a specific program record), how it was used, what you found useful or beneficial about the information, and what the successful outcome was related to your use of the content. The information you provide will be shared with personnel at OJP. Upon review of your submission, OJP personnel may reach out to you directly to gather further information and feedback about your experience.
• How can I become a CrimeSolutions.gov Study Reviewer?
  

  CrimeSolutions.gov Study Reviewers are subject-matter and research methodology experts who must complete training and certification prior to reviewing studies. Those interested in becoming Study Reviewers must first meet the minimum qualifications: (1) Study Reviewers must possess a Ph.D., preferably in a social science-related field of study such as sociology or criminal justice, but other fields of study will be considered; (2) Study Reviewers must be subject-matter experts and have substantial knowledge of research in at least one or more of the following seven topic areas: corrections and reentry, courts, crime and crime prevention, drugs and substance abuse, juvenile justice, law enforcement, and/or crime victims and victimization; (3) Study Reviewers must have extensive knowledge about research methodology. Potential Reviewers must be available to complete reviews up to about 5 days per year. Practitioners in the field are welcome to apply, as long as they meet the requisite education and background requirements.

   

  To learn more about the qualifications and responsibilities of Study Reviewers, visit the CrimeSolutions.gov Researchers and Reviewers page. To submit your request to become a CrimeSolutions.gov Study Reviewer, please Contact Us. Upon receipt of your request, your message will be shared with the appropriate personnel at the Office of Justice Programs and Development Services Group, Inc., who will contact you for more information.

• How can I reference the CrimeSolutions.gov review of my program?
  

  Please use the following text: This program received the rating of ['Effective' or 'Promising'] by CrimeSolutions.gov. For more information, visit [URL of CrimeSolutions.gov program profile].

• How do I provide feedback or express concerns about an evidence rating or any other aspect of CrimeSolutions.gov?
  

  The program information and evidence ratings included on CrimeSolutions.gov are not static. As additional programs are identified and new research becomes available, CrimeSolutions.gov content will be updated and supplemented to reflect the most current programmatic and research information available. We also rely on users to provide us with critical feedback about the CrimesSolutions.gov Web site, itself. What is useful and what is not? What additional features would you like to see on the site in the future? CrimeSolutions.gov users are welcome to Submit Feedback.

  Specific concerns about programs’ evidence ratings or information contained within CrimeSolutions.gov may also be submitted via the Submit Feedback online form. If necessary, changes to the information presented on the site will be made, per the Updating an Evidence Rating process. Rationale and documentation explaining any rating changes will be posted on the Web site.

• How does CrimeSolutions.gov determine whether a program works?
  

  Study Reviewers analyze the most rigorous evaluation research available to determine whether there is evidence that the program achieves its goal(s). Up to three studies, representing the most rigorous evaluation research available, are selected to comprise the program’s evidence base. In some cases, a fourth study may be included in the review if Study Reviewers feel it is necessary to accurately assess a program. The reviewers use a standard Scoring Instrument for each study and assign scores across four dimensions:

  ·         Program’s Conceptual Framework

  ·         Study Design Quality

  ·         Study Outcomes

  ·         Program Fidelity

  The individual study scores are then aggregated to determine the final evidence rating.

  Read more about the: Program Review and Rating from Start to Finish or see the Scoring Instrument.

• How does CrimeSolutions.gov identify programs to be rated?
  

  Potential justice-related programs are identified through:

   

  ·         Literature searches of relevant databases, journals, and publications; and

  ·         Nominations from experts, practitioners, or others.

   

  Read more about how to Nominate a Program for CrimeSolutions.gov.

• How does CrimeSolutions.gov treat programs for which there is insufficient evidence about whether they work?
  

  Programs with evidence that is insufficient to determine whether they achieve their intended outcomes do not receive an evidence rating on CrimeSolutions.gov. However, CrimeSolutions.gov periodically updates a static list of programs that have been reviewed by Study Reviewers, but not assigned an evidence rating due to lack of evidence. A program is placed on the insufficient evidence list if the study (or studies) reviewed received only Class 5 study ratings indicating that there were significant limitations in the study design such that it was not possible to establish a causal relationship to the program’s justice-related outcomes.

  See the List of Programs with Insufficient Evidence or Program Review and Rating from Start to Finish.

• If a program is rated as “Effective” on CrimeSolutions.gov, does that mean the Department of Justice (DOJ) or the Office of Justice Programs (OJP) is endorsing this program for use by others?
  

  No. It is important to note the CrimeSolutions.gov Web site does not constitute an endorsement of particular programs, nor does it conduct original research.   The programs reported upon favorably are being recognized for their accomplishments in support of the mission of the Office of Justice Programs (OJP). Furthermore, it is not intended to replace or supersede informed judgment and/or innovation. CrimeSolutions.gov recognizes that rigorous evaluation evidence is one of several factors to consider in justice programming, policy, and funding decisions. OJP also recognizes the importance of encouraging and supporting innovative approaches that may not yet have extensive evidence of effectiveness. Please refer to the What is the purpose of CrimeSolutions.gov? FAQ for additional information.

• Is there a recommended format for citing information from program profiles and summaries listed on this site?
  

  CrimeSolutions.gov recommends the following citation format: Office of Justice Programs. Name of program. Retrieved [month, date, year document was accessed], from CrimeSolutions.gov, [URL of summary].

• What do the evidence ratings used on CrimeSolutions.gov mean?
  

  CrimeSolutions.gov classifies programs in three levels: “Effective,” “Promising” and “No Effects” based on the strength of the evaluation research that indicates a program achieves its goals (i.e., its intended outcomes).

  Evidence Ratings
  Evidence Rating	Icon*		Description
  	One Study	More than One Study
  Effective			Programs have strong evidence indicating they achieve their intended outcomes when implemented with fidelity.
  Promising			Programs have some evidence  indicating they achieve their intended outcomes. Additional research is recommended.
  No Effects			Programs have strong evidence indicating that they did not achieve their intended outcomes when implemented with fidelity.

  * A single study icon is used to identify programs that have been evaluated with only one study.

  A multiple studies icon is used to represent a greater extent of evidence supporting the evidence rating. The icon depicts programs that have more than one study in the evidence base demonstrating effects in a consistent direction.   

  Read more about evidence ratings at About CrimeSolutions.gov.

• What is considered before a program is rated on CrimeSolutions.gov?
  

  Before a program’s evidence is reviewed and rated, the program is screened to determine whether it meets CrimeSolutions.gov’s criteria for inclusion on the Web site. This includes a thorough review of the program (does it fall within the scope of CrimeSolutions.gov?) and the strength of the available research about the program (can we determine with confidence whether it works?). The final determination as to whether a program’s evidence is reviewed is made by CrimeSolutions.gov Lead Researchers. Read more about: Program Review and Rating from Start to Finish or Nominate a Program for CrimeSolutions.gov.

• What is the purpose of CrimeSolutions.gov?
  

  CrimeSolutions.gov is intended to be a central, reliable, and credible resource to help practitioners and policymakers understand what works in justice-related programs and practices. Its purpose is to assist in practical decision making and program implementation by gathering information on specific justice-related programs and reviewing the existing evaluation research against standard criteria. 

  It is important to note the CrimeSolutions.gov Web site does not constitute an endorsement of particular programs, nor does it conduct original research. The programs reported upon favorably are being recognized for their accomplishments in support of the mission of the Office of Justice Programs (OJP). Furthermore, it is not intended to replace or supersede informed judgment and/or innovation. CrimeSolutions.gov recognizes that rigorous evaluation evidence is one of several factors to consider in justice programming, policy, and funding decisions. OJP also recognizes the importance of encouraging and supporting innovative approaches that may not yet have extensive evidence of effectiveness.

  Important Notes about How to Use the Information Provided on CrimeSolutions.gov: 

  ·        CrimeSolutions.gov rates the quality of evaluation research supporting a program’s outcomes. Consequently, its ratings do not necessarily reflect a program’s effectiveness in every possible scenario or situation. Users should be careful to review the program record to understand each program’s target population, demographics, setting, and the research results for each program outcome.

  ·         Review of programs and their posting on the CrimeSolutions.gov does not constitute an endorsement, promotion, or approval of these programs by CrimeSolutions.gov, the Office of Justice Programs, or the U.S. Department of Justice.

  ·         It is inappropriate to use CrimeSolutions.gov as an exhaustive list of justice-related programs, since CrimeSolutions.gov has not reviewed all programs in existence.

  ·         Policymakers and funders are discouraged from limiting their selection of potential contractors and/or grantees to only those programs that appear on CrimeSolutions.gov.

  ·         Users are permitted to download publicly available articles and research via the National Criminal Justice Reference Service (NCJRS) or order copies for a nominal fee. However, users are asked to contact individual study authors for permission to duplicate original studies, as these studies are not the property of CrimeSolutions.gov.

  ·         CrimeSolutions.gov is funded by the U.S. Department of Justice's Office of Justice Programs and its evidence reviews are managed by Development Services Group, Inc. The views expressed on this Web site do not necessarily reflect the views of the U.S. Department of Justice.

  Read more about Tips for Using CrimeSolutions.gov.

• What is the relationship between CrimeSolutions.gov and the Office of Juvenile Justice and Delinquency Prevention (OJJDP)’s Model Programs Guide?
  

  OJJDP is a component of the Office of Justice Programs (OJP), and OJJDP’s Model Programs Guide (MPG) predates CrimeSolutions.gov. Although MPG programs fall within the broader scope of CrimeSolutions.gov, it was necessary to reassess these programs because the criteria and ratings used by the two sites differ.

  The following information outlines how MPG’s programs were reviewed for possible inclusion on CrimeSolutions.gov:  

  ·         MPG programs were examined to ensure that they met the CrimeSolutions.gov screening criteria.

  ·         New studies were identified and reviewed to determine whether they should be included in the evidence base used to rate a particular MPG program. (This process is being completed first for MPG’s "Exemplary" rated programs. MPG’s "Effective" and "Promising" programs will be re-assessed at a later date.)

  ·         Using the CrimeSolutions.gov Scoring Instrument, the studies were reviewed and re-rated, if necessary.

  ·         New evaluation methodology and outcomes sections were written to reflect any new studies included in the evidence base.

  Read more about: Program Review and Rating from Start to Finish.

• Who reviews the evidence used to rate the programs on CrimeSolutions.gov?
  

  At least two Study Reviewers independently assess the evaluation research for each program under consideration by CrimeSolutions.gov. All Study Reviewers are subject matter and research methodology experts who complete training and certification prior to reviewing studies. Lead Researchers select the evidence base, coordinate the review process, match programs with Study Reviewers based on expertise, and resolve any scoring discrepancies prior to a program being assigned a final evidence rating. Study Reviewers use a standard Scoring Instrument to determine whether a program achieves its goals.

  Read more about CrimeSolutions.gov Researchers and Reviewers or the Program Review from Start to Finish.

• Who sponsors and operates CrimeSolutions.gov?
  

  CrimeSolutions.gov is an initiative of the Office of Justice Programs (OJP) within the U.S. Department of Justice. OJP contracts with the Development Services Group, Inc (DSG) to coordinate the program and evidence review process and provide program content and with Lockheed Martin to provide Web site and technical support.   

  Read more information about the Office of Justice Programs.

• Why are outcomes scored separately on the CrimeSolutions.gov scoring instrument?
  

  Each outcome is scored separately because programs may impact multiple outcomes, and the evidence supporting the different outcomes may vary. CrimeSolutions.gov also distinguishes primary from secondary outcomes. Primary outcomes are the central intended effects that a program is designed to achieve. Within the scope of CrimeSolutions.gov, those primary outcomes must also relate to criminal justice, juvenile justice, or victimization prevention, intervention, or response. Secondary outcomes are the ancillary effects of a program. When studies are reviewed, multiple outcomes are aggregated with more weight given to primary outcomes than secondary outcomes.

  Read more about the: Program Review and Rating from Start to Finish or see the Scoring Instrument.

• Why does CrimeSolutions.gov include “No Effects” programs?
  

  "No Effects” programs have strong evidence to indicate they do not achieve their intended outcomes when implemented with fidelity. CrimeSolutions.gov includes “No Effects” programs to inform policy makers and practitioners about the current status of available evaluation evidence before planning or implementing similar programs. However, CrimeSolutions.gov recognizes that rigorous evaluation evidence may be one of several factors (such as cost or implementation) to consider in justice programming, policy, and funding decisions.

  Read more about: Program Review and Rating from Start to Finish or Tips for using CrimeSolutions.gov.

• Why does CrimeSolutions.gov use a maximum of three evaluations to determine an evidence rating for a program?
  

  CrimeSolutions.gov Lead Researchers review the available research about a program and select up to three of the most rigorous studies to determine the evidence rating. In many cases, there are fewer than three evaluations available for a particular program. The studies selected are those with the most rigorous methods and study designs and are expected to produce the most reliable and credible results. Study Reviewers are subject matter experts familiar with the body of research in the topic area, so they may, in consultation with the Lead Researcher, request to review additional studies if they can demonstrate a compelling need. Study Reviewers also use additional studies, program materials, process evaluations, and curricula outside the evidence-base as supplemental material to inform the overall assessment process.

  To see more about the entire review process: Program Review and Rating from Start to Finish.

• Why doesn’t CrimeSolutions.gov make more use of findings from meta-analysis?
  

  CrimeSolutions.gov uses a well-developed, systematic method for reviewing and rating the evidence from program evaluations, including multiple evaluations of the same program. Meta-analysis is a different systematic method that statistically combines the results of multiple studies that all share a similar research hypothesis. Currently, CrimeSolutions.gov uses meta-analytic studies, when available, to inform evidence ratings and program profiles. OJP plans to add more information to CrimeSolutions.gov about relevant results from meta-analytic studies in criminal justice, juvenile justice, and victim services.

   

  Incorporating results from meta-analyses into CrimeSolutions.gov will require different strategies for study review and information presentation. There are important methodological differences to address, and there are cases where meta-analysis may produce different outcomes than the current CrimeSolutions.gov approach. For example, a meta-analysis of Scared Straight and related programs found that they fail to deter crime and actually lead to more offending behavior (Petrosino, et. al., 2004). CrimeSolutions.gov screened all the same studies and selected the three most rigorous studies of the Scared Straight model for a review. That review resulted in a finding of “insufficient evidence” due in large part to a lack of fidelity information that would confirm that all studies were evaluating the original conceptual framework.

   

  Petrosino A, Turpin-Petrosino C, Buehler J. 2004. “Scared Straight” and other juvenile awareness programs for preventing juvenile delinquency. Campbell Systematic Reviews 2004.

• Will the Office of Justice Programs (OJP) only fund programs that appear on this site?
  

  No! CrimeSolutions.gov is a resource to assist justice practitioners and policy makers in using evaluation evidence for practical decision making and program implementation. CrimeSolutions.gov is not intended to be an exhaustive list of worthy and unworthy investments. OJP and its components also recognize the importance of supporting innovative approaches and practices that may not yet have extensive evidence of effectiveness. Please refer to the What is the purpose of CrimeSolutions.gov? FAQ for additional information.

  Read more information about the Office of Justice Programs.