Note: For process evaluation see the "Process Evaluation" section below. After the process evaluation section is a short section on impact evaluation.

Chapter 8

Evaluation

Without a systematic evaluation, an agency may not be able to assess whether its gang suppression strategy is successful. A careful evaluation will suggest ways to improve or expand operations, point out strengths and weaknesses, and help other departments adapt similar strategies to meet their communities' needs. Also, the public, elected officials, and potential funding sources will want to see evidence of success.

The evaluation design need not be overly complex to produce meaningful results. This chapter presents concepts to consider in conducting evaluations. Local colleges and universities also may be of assistance either by helping with evaluation design or by providing student interns who can collect and analyze data.

The police department should prepare for the evaluation during the project's early planning stages. An early analysis will describe the environment in which the department is operating and may reveal areas in which new or revised data collection procedures are needed. When the department establishes its gang suppression objectives, planners also should determine the indicators for measuring the achievement of those objectives.

In general, the evaluation seeks to answer several key questions:

• Was the strategy or program implemented as intended?
• What specific activities were implemented?
• Did these activities lead to the attainment of specific objectives?
• Were the program's overall goals achieved?
• To what extent were successes or failures a result of factors other than the strategy?

To answer these questions, a two-stage approach to evaluation is used. This approach is described in the following sections.

Process Evaluation

A process evaluation documents and analyzes the early development and actual implementation of the strategy or program, assessing whether strategies were implemented as planned and whether expected output was actually produced. Examples of output include:

• Number of gang-related crimes reported.
• Number and category (for example, leaders, hardcore members, and associates) of gang members arrested.
• Number and types of weapons seized from gang members and associates.
• Types and amounts of drugs seized from gang members and associates.
• Number of gang members and associates convicted, along with charges and sentences.

Other output measures are listed on a quarterly reporting form completed by departments funded under BJA's Urban Street Gang Program. A copy of this form is included in Exhibit 4 which is not attached.

Detailed information about the program as it was actually implemented is invaluable for determining what worked and what did not. A thorough process evaluation should include the following elements:

• Description of the program environment and supplying data.
• Description of the process used to design and implement the program.
• Description of program operations, including any changes in the program.
• Identification and description of intervening events that may have affected implementation and outcomes.
• Documentation such as meeting minutes, reports, memorandums, newsletters, and forms.

Describing the Program Environment

Before a program's effectiveness can be judged, it is important to understand its operating environment. The program analysis stage (see Chapter 3) produces a detailed description of the environment at the time of planning. Significant changes in the environment should be documented during program implementation to help determine whether similar results may be expected in other communities or whether the results are site specific. That is, if the environment is unique, then the results achieved in that setting may not be replicable elsewhere.

Describing the Process

Good ideas do not always yield good results. Therefore, to understand the tasks to be performed and the scope of effort, a clear description of the implementation process is required. This step also will aid in replicating the effort in other environments. Describing the implementation process would involve elements such as:

• Interaction among participants (for example, gang specialists or gang unit, patrol, investigations, crime analysis, other local criminal justice agencies, Federal enforcement agencies, and citizen groups).
• The extent of participation (for example, by various agencies, units, and individuals).
• Any training provided to officers or other participants.
• Interaction among participants and others in the community who were not involved in planning and implementing the strategy.

Describing and Measuring Program Operations

The process evaluation must describe the way the gang suppression strategy worked, or failed to work, using quantitative and qualitative data. Questions to consider in assessing the process evaluation include the following:

• What problems were encountered in implementing objectives? How were they resolved?
• Have all planned activities been implemented? If not, what remains to be done? Were they accomplished on schedule?
• If objectives, plans, or timetables were revised, why was this necessary?
• What new objectives were added and why?
• What changes occurred in leadership or personnel? What effect did these changes have?
• What costs were incurred? Did they exceed initial projections?
• What was the level of resident support in targeted neighborhoods? How did this affect the overall enforcement effort?
• What lessons have been learned that might be useful to other jurisdictions?

Identifying Intervening Events

Local programs operate in continually changing environments. To identify the effects of various intervening external factors on program outcomes, an intervening variable analysis is employed. The intervening variables that may affect a program are numerous, complex, and varied and may or may not be able to be controlled or eliminated. These variables are discussed later in this chapter in the Impact Evaluation section. To assess outcomes correctly, these variables will need to be identified, interpreted, and described.

Collecting Process Data

The process evaluation should begin during the program planning phase and continue through program implementation. Two main categories of data should be collected. The first data source includes official police, city/county, census, and other data gathered during the initial program planning and analysis phase. These data will help determine whether program outcomes may be expected in similar jurisdictions.

The second data source includes interviews with and observations of participants. Observations should begin with early program development and continue throughout program implementation. Major planning activities as well as enforcement activities are of interest. Observers will answer questions such as the following:

• Does the process proceed smoothly, or are communications and relations difficult and strained?
• Do participants work together to identify a range of potential strategies?
• Do the status and hierarchy of involved personnel interfere with communications?

Interviews with key participants also should be conducted to complement information attained from observation. These interviews should reveal the reactions of patrol officers, investigators, and others to the program's development, noting their observations about difficulties encountered and associated explanations, as well as suggested solutions. An open-ended format for observations is suggested so that observers are not limited in their focus. However, the protocol for interviews should be more structured to ensure consistency and validity.

Impact Evaluation

An impact evaluation measures the program's effects and the extent to which its goals were attained. Although evaluation designs may produce useful information about a program's effectiveness, some may produce more useful information than others. For example, designs that track effects over extended time periods (time series designs) are generally superior to those that simply compare periods before and after intervention (pre-post designs); comparison group designs are superior to those that lack any basis for comparison; and designs that use true control groups (experimental designs) have the greatest potential for producing authoritative results. At a minimum, gang suppression programs should use pre-post designs. Even better is the use of longer time series analyses and comparison or control group designs.

Dependent Variables

The major limiting factor on evaluations is the availability of baseline data on the dependent variables of interest. Examples of dependent variables are gang-related crime, citizens' fear of crime, and weapons offenses. Although data on arrests, reported crime, and calls for service for prior time periods are available in most jurisdictions, it may be impossible to determine from these data which events are gang related. Some jurisdictions find it informative to examine pre- and post-data surrounding events that frequently involve gang members (for example, all weapons offenses, including shots fired; homicides; drug trafficking; and driveby shootings).

Less commonly found are data on other variables, such as fear of crime, levels of disorder, or satisfaction with the police.

If no prior data exist for an important dependent variable, the department must gather the data prior to the start of a new program. This effort may involve producing special computer runs on selected types of incidents or geographic areas, conducting a special community survey, taking photographs that show gang influence in targeted neighborhoods, or hand tabulating data from existing records.

Comparison or Control Groups

Whenever possible, it is desirable to identify a comparison or control group. This may be a group of persons or an area in the community that does not receive the intervention but has characteristics similar to the group that does. Another possible comparison group is a similar, nearby community. The same pre- and post-intervention measurements should be made of the comparison group as of the group receiving the intervention.

Basic Evaluation Procedures

Regardless of the evaluation design, certain fundamental procedures should be followed, the most important of which are outlined in the eight steps that follow.

Carefully State the Hypothesized Effects.  In the project planning phase, the department should carefully state its expectations of a gang enforcement strategy that is implemented correctly. There may be one anticipated effect or many, examples of which are:

• The strategy will reduce gang-related crime by 30 percent.
• The strategy will eliminate drug trafficking by the 86th Street Crew.
• The strategy will reduce fear of crime among residents in the southwest quadrant.
• The strategy will result in conviction of two hardcore gang leaders under Federal Triggerlock statutes.

In some programs, the effects may be expected to occur in stages: for example, the program will reduce street-level drug sales by gang members, which will reduce fear of crime. In this case, one stated effect is anticipated to result in another.

Identify Possible Unintended Effects.  Most programs have potential associated risks that also should be identified early in the planning phase to allow the department an opportunity to address them better. The evaluation can assess whether the unintended effects occurred. Examples of unintended but anticipated effects are:

• Officers will object to the organizational changes required for implementation.
• New gang leaders will quickly assume control after leaders targeted by the strategy are incarcerated.
• Neighborhood residents will react negatively to increased enforcement efforts.
• Key personnel will be transferred or will retire.

Define Measurement Criteria.  The indicators that will be measured (for example, gang-related crime and community support) must be clearly defined to obtain consistent, reliable measurements. What is meant by gang related crime? Should a gang motive be present before crimes are so classified? Are measures of some crimes more important than others? How are satisfaction with the police, fear of crime, or community support defined? Terms should be clarified so the department and evaluators understand exactly what is to be measured and how.

Determine Appropriate Time Periods.  Data collection can be costly, and information about results is important to people with investments in the program. For practical reasons, evaluators often must compromise ideal time periods. This step responds to issues such as how far back in time baseline data should be collected and how long the program should operate to give it a fair opportunity to show results.

Generally, the BJA Urban Street Gang Program demonstration sites obtained data on key dependent variables (for example, drug arrests, driveby shootings, and gang-related homicides) for a 5-year period before project commencement. This method helped identify trends and aberrations. For example, an unusual problem or a unique special operation could have resulted in an unusually high arrest rate in a given year.

The demonstration sites were expected to devote the first 3 months to the needs assessment and planning processes, with program activities occurring for the next 12 to 15 months. The impact evaluation should be based on data accumulated for at least 12 months of program activities. The demonstration sites were required to submit their evaluation reports in their 18th month.

Although not a steadfast rule, most jurisdictions should be able to make some assessments about program impact after 12 months of activities. However, the appropriate period may vary considerably, depending on the nature of the problem, the type and complexity of the response, and other factors.

Monitor Program Implementation.  Systematic program monitoring is required to aid the site in correcting and overcoming problems, to effect management improvements that may reduce future implementation failures, and to interpret program results correctly.

Collect Data Systematically.  The data collected on program implementation, hypothesized effects, and unintended effects must be as accurate as possible. If more than one person collects data, each must follow the same rules and use the same definitions. If data are collected over a long period, the same rules and definitions must be used throughout.

Analyze Data.  Data analysis should produce a description of the program as it was implemented. If the evaluation design is strong enough, analysis can go beyond describing what happened and provide convincing explanations of why it happened. The analysis should present the evidence that helps determine whether the program had its hypothesized effects and whether it resulted in any unintended effects.

In some situations, it may be useful to determine whether differences between target and control groups are statistically significant. Various statistical techniques may be used to determine whether observed differences are likely due to the program design or whether they occurred by chance.

For most jurisdictions, statistical significance will be less important than other considerations, such as if the program effect was large enough to make a substantial difference and if enough benefits were derived to justify program costs. These judgments must be made by the department, city or county administrators, and residents, and the evaluation results should assist in making informed decisions.

Replicate the Program.  Replication of a program ensures that the documented program effects were not a chance occurrence, the result of unobserved intervening factors, or limited to one place or one time period. Once a program component is found to have consistent effects in several applications, the strategy may be used confidently with predictable results. This feature underscores the importance of evaluations.