Inquiring About or Appealing an Evidence Rating
CrimeSolutions.gov is committed to principles of quality, fairness, and transparency in its program reviews. As part of that commitment, any party, such as the program developer or researcher, may submit an inquiry regarding an evidence rating (i.e., Effective, Promising, or No Effects) or the inclusion of a program in the Programs with Insufficient Evidence section of CrimeSolutions.gov. The inquiry should be grounded in 1) a fair reading of the evidence (please see the program’s Evidence-Base) and 2) an understanding of the rating procedures and criteria established for CrimeSolutions.gov (please see Program Review and Rating from Start to Finish). Inquiries should be submitted using the Contact Us form found on CrimeSolutions.gov.
A written response will be made to each formal inquiry. The goal of the response is to provide additional explanation for the evidence rating that was assigned. The response may include paraphrased or excerpted comments from the Reviewers’ scoring instruments in order to explain why certain sections of the instrument (e.g., Design Quality or Outcome Evidence) received the scores given. (Under no circumstances will the identity of the Reviewers be provided.) If applicable, additional information, such as information regarding the resolution of disparate scores among reviewers, may also be included in the response to an inquiry.
Step 1. Following the inquiry and response described above, an individual may elect to re-contact CrimeSolutions.gov using the Contact Us form to request a formal appeal of an evidence rating. Individuals who request a formal appeal will be provided additional guidance by CrimeSolutions.gov that will require the submitter to provide pertinent, specific information that would indicate the necessity for an appellate review.
Step 2. The information submitted will be turned over to the CrimeSolutions.gov Appeals Board, which will work toward providing a systematic response that addresses all concerns. (The Appeals Board is comprised of CrimeSolutions.gov Study Reviewers, supplemented with additional expert reviewers if needed.) Two members of the Appeals Board, selected by CrimeSolutions.gov based on expertise pertinent to the appeal and the absence of any conflicts of interest, will conduct the appellate review.
Step 3. Each appellate reviewer will conduct an independent review of the appeal, the studies in the program’s Evidence-Base, and copies of the original Reviewers’ scoring instruments. (The identities of the original Reviewers will remain anonymous until they are required to participate in the process.)
Step 4. After the separate reviews, the appellate reviewers will participate in a conference call to discuss the original review, the program’s final rating, and the concerns raised by the inquirer. The goal of the conference call is to reach consensus about the program’s evidence rating. If the appellate reviewers agree that the program’s original rating should not be changed, they will provide a written explanation documenting the discussion and the reason for their final decision. If they believe a new rating is warranted, they will provide a written explanation describing how and why their scores differ from the scores of the original Reviewers. In this situation, a final conference call will be held between the Appeals Board members and the original Reviewers, to discuss the disagreement on the scoring instruments and come to a consensus on a final program rating.
Step 5. Once a final consensus rating is reached, a written response describing the final decision from the Appeals Board will be provided to the individual who initiated the appeal. The goal is to complete the entire appeal process within 60 days.
Step 6. If warranted as a result of the appeal, changes to a program’s evidence rating or additions to information in the program profile will be made on CrimeSolutions.gov. Read more about: Updating an Evidence Rating.