PA 555 Professor Brian Stipak
Spring 2010 email: stipakb@pdx.edu.
Thur. 4:00-6:30 http://web.pdx.edu/~stipakb
COURSE SYLLABUS: PA 555
Program Evaluation
- Read this syllabus carefully if you are taking this course. It defines some of your responsibilities and some of my responsibilities. -
My version of the PA555 course provides a broad overview of the field of program evaluation. It includes some coverage of qualitative evaluation methods, performance measurement, performance auditing, and benefit‑cost analysis, but the biggest concentration is on the classic approach to thinking about research design. The course serves best those students desiring a broad overview of program evaluation, but may disappoint those students wanting in‑depth coverage of particular methods.
Course requirements include a project paper, a course "journal" in which you answer questions weekly, and usually one or two very short in‑class quizzes on specific topics. Class sessions require substantial student participation, often in response to questions I ask based on the weekly journal assignments; also, students will give presentations on their projects during the last two classes.
Course Materials
Here are the major required course reading materials:
1) On-line materials available through my web site
2) Packet of course materials available from Clean Copy, 1704 SW Broadway, phone 221‑1876
3) Michael Q. Patton, How to Use Qualitative Methods in Evaluation, ISBN 0-8039-3129-8
Other useful references:
Peter H. Rossi et al., Evaluation: A Systematic Approach, ISBN 0-76190-893-5, a comprehensive textbook
Emil J. Posavac and Raymond G. Carey, Program Evaluation: Methods and Case Studies, ISBN 0-13-227560-0, a comprehensive textbook, especially good on research design
Joseph S. Wholey et al., eds., Handbook of Practical Program Evaluation, Jossey‑Bass Pub., ISBN 0787967130, articles cover the range of evaluation topics
C. Fitz-Gibbon and L. Morris, How to Design a Program Evaluation, Sage Pub., ISBN 0-8039-3128-X, a simple presentation of research design
Lawrence Mohr, Impact Analysis for Program Evaluation, ISBN 0-8039-5936-2, a more technical treatment of research design and the role of statistics
Donald T. Campbell and David A. Kenny, A Primer on Regression Artifacts, ISBN 1‑57230‑859‑1, an excellent technical treatment of the statistical issues that occur in non‑experimental impact evaluations.
Stewart I. Donaldson and Michael Scriven, Evaluating Social Programs and Problems: Visions for the New Millennium, ISBN 0-8058-4185-7, a provocative collection of essays
Ronald D. Sylvia et al., Program Planning and Evaluation for the Public Manager, ISBN 0-88133-920-2, good sections on evaluation designs and benefit cost analysis
On-Line Course Materials
Course materials and links to relevant web sites are on my web site. Go to the page "Resources / help for my students and for others". Carefully look over:
1) PA555 help page
2) links under "Program Evaluation", "Performance Measurement", and "Performance Auditing"
3) files in the "PA555" folder and the "PerfMeasures" folder
The readings listed in the course outline that are available on‑line through my web site have the designation "(web)" in the course outline.
Since information available on the web is growing and dynamic, please let me know when you find other on‑line materials that are helpful for this course. You can email me and/or you can post to the course listserv.
Course Listserv
I have subscribed all students enrolled in the course who have an email address in the PSU Information System to the course listserv. If you do not have an email address in the PSU information system then I did not subscribe you and you will need to subscribe yourself.
You must subscribe to the course listserv within two days after the first class meeting, if you are not already subscribed or if you want to subscribe using another email address. I will use this listserv to send messages to the class members. You can also send messages to the class members‑-for example, to solicit help and to help other students requesting help.
To subscribe to the listserv go to the information page for this listserv, which you can link to via my web site, or you can just go directly to:
"https://www.lists.pdx.edu/lists/listinfo/pa555"
Course Journal
A course requirement is to update weekly a "journal" in which you write out assignments listed under the sections in the course outline.[1] Since a purpose of this requirement is to prepare you to participate in, and to get the most out of, the up-coming class meeting, you need to do this each week before the class meets. At the end of the term you will turn in your entire journal.
Some of the journal assignments involve answering questions in the course outline. If you feel these questions are too limiting, you may substitute questions of your own for the course outline questions. In that case, when I read your journal I will assign points based on your questions as well as your answers.
When writing your journal entries, make sure to include abundant specifics drawing on the readings, reports, and on-line materials that we are covering for that topic. When I score the journals at the end of the term, I will assign points paying special attention to the detail in the journal assignments. Make sure that you draw on concepts from, and refer to, the readings. For example, for the journal assignment (Section 2) discussing the Portland SEA report, cite specifics to support the statements you make about the content and utility of the SEA report to make it clear that you have examined the report in detail. My fall-back question when I score the journals will be, "Could this section have been written without doing the required reading?"; if yes, the assigned score will be low.
Project Paper and Project Presentation
A course requirement is to write a project paper.
Suggested project paper length: 8-20 pages.
Please write your email address, and your home and work telephone numbers, on the front of your project paper in case I need to contact you about your paper.
Project Paper Description: Alternative Evaluation Designs
The purpose of the project paper requirement is to learn more about the major methods of program evaluation and the application of those methods. The assignment is to take a specific program or agency and to consider how you could apply program evaluation methods in that setting. The program/agency can be one you know about, have worked in or previously worked in, or even an imaginary program or agency. Your paper will have two main sections that examine the application of 1) qualitative methods, 2) quantitative methods, followed by a third section that assesses the appropriateness of the alternative methods.
For more specific description of the project paper see the on-line document, "pa555Project.doc".
Alternative Course Project Paper Option
The normal course project paper as described above and in the on-line description is intended for most students. However, a few students may strongly desire to do their course project paper on an alternative topic, and the alternative course project paper option provides flexibility to accommodate that desire.
To choose this option, you need to turn in to me a paper proposal that 1) explains why you prefer the alternative paper option, and 2) explains what you propose to do for the alternative project paper. After I have approved your proposal you may proceed with this option.
For further description of the alternative project paper option see the on-line document, "pa555ProjectAlternative.doc".
Project Presentation
Another course requirement is to make a short in-class presentation based on your project paper, accompanied by a summary hand-out for the class.
In-Class Presentation: Choose one aspect of your project paper to use for your in-class presentation. For most of you this will mean that you will focus on one part of the application of program evaluation methods that you discussed in your project paper. You should then strive to provide a concise presentation that instructs the class about the potential application of these methods in your example program/agency.
Suggested time for presentation: 4-6 minutes, with an 6.5 minute absolute maximum. You need to practice and time your presentation.
Summary Hand-Out: Part of the project presentation requirement is to prepare, and to hand out to the class at the time of your presentation, a one-page handout. Try to focus the hand-out on what you are presenting that could be useful as a future reference source for other class members.
Course Grades
I will assign course grades as fairly as I can based on a computerized total points score and calculated grade. I usually allow myself a maximum leeway of one increment (for example, B to B+) from the calculated grade to take into account class preparation, participation, and other special considerations.[2] The total point score is computed from scores on the following criteria (approximate weights in parentheses):
30%1.Course journal, including meeting the due date (Late journals will be severely penalized and may not be accepted, since they present work that the course requires you to do weekly).
30%2.Project paper, including meeting the due date
10%3.Project presentation, accomplishing the presentation's objective, effectiveness of the presentation, meeting the time limit, and quality of the handout
10%4.Research design quiz
5%5.Benefit-cost analysis quiz
15% 6. Other class participation and misc.
I do not give all graduate students "A's". A "B" is also a passing grade at the graduate level. A "C" or lower means I feel the work was below acceptable graduate student standards. In practice, for this type of course I typically assign about half or more of the grades in the A/A- range, half or less in the B+/B/B- range, and occasionally assign a grade of C or lower.
The university policy on assigning incomplete ("I") grades restricts their use to special circumstances (see PSU Bulletin). Missing assignments will usually result in a lower grade or an "X" grade, not an "I". If major assignments are missing a very low grade may result. If you enroll in this course but find you are not able to do all of the assignments, I recommend that you drop the course. If you remain in the course and do not complete all of the work, you need to write me (or email me) to request an "I" grade.
I consider academic dishonesty a serious offense and will penalize offenders to the full extent possible under university policies. Academic dishonesty includes cheating on examinations, copying or stealing other people's work, turning in work done by someone else as one's own, plagiarism, and other kinds of misrepresentation. If you know of any of these problems occurring in this class, please let me know so that I can investigate. The vast majority of you would never do this, and my commitment to you is if I should learn about such cheating I will deal with it severely, regardless of who the student is and what excuses the student offers.
Return of Assignments, Quizzes
You can pick up the course journals and project papers from me after finals week, or you can give me a stamped, self-addressed envelope to mail them to you. I keep for reference copies of any quizzes, but a key quiz will be available for you to see immediately after completing the quizzes, and your scored quizzes will be available for you to examine in my office.
My Availability Outside of Class
I am available to consult with you about course matters by phone and in person outside of class hours. To see me in person, schedule a time with me. Also, feel free to stop by my office if you are in the CUPA Building; if I am in, I will see you then if I can. This combined by-appointment/flexible-office-hours approach provides a more practical way to arrange consultation than limited specific office hours, given the varied work schedules of students in our program.
Course Outline
Note: Readings with the designation "(web)" are available through my web site.
1. Introduction, Basic Concepts, and Overview
How has program evaluation developed and changed over the years?
What are some of the major concerns of people writing about program evaluation?
What are some of the methodologies used in program evaluation?
What topics in the evaluation field appear controversial?
What "guiding principles" does the AEA espouse for evaluators?
What "evaluation standards" does the AEA espouse for evaluators?
What interests you about the field of program evaluation?
What are some of the professional evaluation organizations and what are they like?
What applications does program evaluation have to your work?
Thomas D. Cook, "Lessons Learned in Evaluation Over the Past 25 Years"
Peter H. Rossi and James D. Wright, "Evaluation Research: An Assessment"
Guiding Principles for Evaluators (web)
Program Evaluation Standards (web)
Browse the American Evaluation Association web site and the Oregon Program Evaluators Network (OPEN) web site
Further reference:
Rossi et al., Ch. 1
Journal assignment 1: Answer five of the questions.
Journal assignment 2: Write a commentary, and be prepared to present it briefly orally, about a program evaluation topic from your first week's readings that interests you. I will call on some of you to present your oral commentary, which will aid in introducing the class to the range of topics, interests, methods, and types of evaluations in the program evaluation literature and community(ies). You can draw your comments from any of our course readings or other research. See my web site for some on‑line resources. If you feel like visiting the library try browsing the program evaluation-related materials in the PSU library (4th floor, H section), and perhaps check out: Evaluation Review (H1.E75, 4th floor), the discontinued Evaluation Studies Review Annual (H1.E77, 4th floor), Journal of Evaluation and Program Planning (H62.A1E9, 4th floor), Journal of Policy Analysis and Management (H1.J656, 4th floor), and Evaluation and the Health Professions (RA399.A1 E9, 2nd floor).
2. Performance/Outcome Measurement/Monitoring
How is program evaluation related to performance measurement?
Why does anyone care about performance measurement?
What use is performance measurement?
What is "service efforts and accomplishments reporting?"
Government Performance and Results Act, Public Law 103-62
Richard C. Tracy and Ellen P. Jean, "Measuring Government Performance: Experimenting with Service Efforts and Accomplishments Reporting in Portland, Oregon"
Daniel E. O'Toole and Brian Stipak, "Implementing Service Efforts and Accomplishments Reporting: The Portland Experience"
Alice Galloway report, "Family Nursery Program"
Performance Measures in Portland, GASB case study (web)
Performance Measures in Oregon, GASB case study (web)
Examine the web sites for the local and state benchmarks
Carefully review one of the City of Portland's annual SEA reports (available on-line from the City Auditor's office and also on my web site; hard copies available at the Audit Services Division). Understand it well enough so that you can say what data it uses and what is presented in the report.
SEA Review Guidelines (web)
Further reference:
Performance Measures in Multnomah County, GASB case study (web)
Maria Aristigueta, Managing for Results in State Government
"Managing for Results: A Proposal for the City of Portland", Report No. 288, Office of the City Auditor, Portland, Oregon (December 2002) (web)
Joseph S. Wholey, "Trends in Performance Measurement: Challenges for Evaluators" in Eleanor Chelimsky and William Shadish, Evaluation for the 21st Century
Daniel E. O'Toole and Brian Stipak, "Service Efforts and Accomplishments Reporting in Local Government", Journal of Government Financial Management (Spring 2001)
Harry P. Hatry et al., eds., Service Efforts and Accomplishments Reporting: Its Time Has Come
Harry Hatry, Performance Measurement: Getting Results
Harry Hatry, "Performance Measurement: Fashions and Fallacies", Public Performance and Management Review (June 2002)
Dennis P. Affholter, "Outcome Monitoring", in Wholey et al.
New Directions for Evaluation (Fall, 1997), This issue of the journal is devoted entirely to performance measurement in public and nonprofit agencies.
Oregon Benchmarks, overview (web)
Journal assignment 1: Answer one of the questions.
Journal assignment 2: Write down your assessment of the local and state benchmarks from visiting their web sites and from other readings (1-2 pages).
Journal assignment 3: Take the SEA Review Guidelines (2004 AGA Guidelines) and apply them to a City of Portland SEA report. For each of the review criteria assign a point score, and following your point score explain why you assigned that score. Total the number of points that you assigned overall. (Note: This assignment has double-weighting for journal grading.)
3. Evaluability Assessment and Process Evaluation
What is evaluability assessment and how is it related to impact evaluation?
What is process evaluation?
Evaluability Assessment (web)
Process Evaluation Manual (web)
Process Evaluation overview (web)
Two example process evaluation reports (web):
ExampleProcessEval1.doc
ExampleProcessEval2.doc
Further Reference:
Joseph S. Wholey, "Assessing the Feasibility and Likely Usefulness of Evaluation" in Wholey et al.
Mary Ann Scheirer, "Designing and Using Process Evaluation" in Wholey et al.
Jean A. King et al., How to Assess Program Implementation
Journal assignment: Answer the questions.
4. Qualitative Evaluation Methods
What are "qualitative methods?"
How do qualitative methods differ from quantitative methods?
What are the potential advantages and problems of using qualitative methods?
What are the potential advantages and problems of combining qualitative and quantitative methods?
Michael Q. Patton, How to Use Qualitative Methods in Evaluation
Handout on "Designs for Combining Qualitative and Quantitative Research"
Further Reference:
Gary Shank, Qualitative Research: A Personal Skills Approach
Thomas D. Cook and Charles S. Reichardt, Qualitative and Quantitative Methods in Evaluation Research
R. A. Krueger, Focus Groups: A Practical Guide for Applied Research
G. McCracken, The Long Interview
M. B. Miles and A. M. Huberman, Qualitative Data Analysis
David Morgan, Focus Groups as Qualitative Research
Michael Q. Patton, Qualitative Evaluation Methods
John Brewer and Albert Hunter, Multimethod Research: A Synthesis of Styles
Abbas Tashakkori and Charles Teddlie, Handbook of Mixed Methods in Social and Behavioral Research
Journal assignment: Answer the questions.
5. Quantitative Evaluation Methods: Impact Analysis
What is research (experimental) validity, and what are the major threats to validity?
Describe some different research designs and how they handle the threats to validity.
What purposes do statistics have in evaluation research, and what relationship does research design have to the use of statistics?
Research Design, summary of classic Campbell/Stanley framework (web)
Program Evaluation Guidebook (web), Ch. 2 on program logic model and Ch. 6 on research design
Thomas H. Wonnacott and Ronald J. Wonnacott, Regression, pp. 1‑11.
Alexander W. Astin, "College Retention Rates Are Often Misleading"
Research Methods Knowledge Base, Design Section (web), review topics selectively
Example impact evaluation reports (web):
ExampleImpactEval1.doc
ExampleImpactEval2.doc
Published Example studies:
Annette Jolin and Brian Stipak, "Drug Treatment and Electronically Monitored Home Confinement: An Evaluation of a Community-Based Sentencing Option"
Harry A. Lando et al., "Brief Supportive Telephone Outreach as a Recruitment and Intervention Strategy for Smoking Cessation"
Lawrence W. Sherman and Richard A. Berk, "The Specific Deterrent Effects of Arrest for Domestic Assault"
Further Reference:
Posavac and Carey, Program Evaluation, Ch. 8-10
Ronald D. Sylvia et al., Program Planning and Evaluation for the Public Manager, Ch. 5, 6
Carol T. Fitz-Gibbon and Lynn L. Morris, How to Design a Program Evaluation
Lawrence B. Mohr, Impact Analysis for Program Evaluation
Donald T. Campbell and David A. Kenny, A Primer on Regression Artifacts
Donald T. Campbell and Julian C. Stanley, Experimental and Quasi‑Experimental Designs for Research
Thomas D. Cook and Donald T. Campbell, Quasi-Experimentation: Design and Analysis Issues for Field Settings
Leonard Bickman, ed., Research Design: Donald Campbell's Legacy
R. Windsor et al., Evaluation of Health Promotion and Education Programs, Ch. 5
David de Vaus, Research Design in Social Research
Journal assignment 1: Answer the questions.
Journal assignment 2: Critically review one of the example studies, either one of the three published studies or one of the two unpublished studies (Note: the Jolin/Stipak published and unpublished studies are similar)(suggested length: 3-5 pages). (Note: This assignment has double-weighting for journal grading.)
Quiz: There will be an in-class quiz on research design.
6. Performance Auditing
What is the history of performance auditing?
How do performance audits differ from program evaluations?
What are performance audits supposed to accomplish?
How do performance auditors conduct an audit?
How useful are performance audits to administrators?
U.S. General Accounting Office, Government Auditing Standards (the "little yellow book") (web), review for general content
Brian Stipak and Daniel E. O'Toole, "Performance Auditing in Local Government: Current Use and Future Prospects"
Richard C. Tracy, "Performance Auditing: Catalyst for Change in Portland, Oregon"
Amanda Noble, "Performance Auditing as a Public Administration Career Choice"
Examine a performance audit report of the Portland City Auditor's office of your choosing. Reports are available on-line and also in hard copy at the Audit Services Division. Examine your report closely enough so that you understand and can explain what the report presents and how the audited agency officially replied.
Further Reference:
Eleanor Chelimsky, "Comparing and Contrasting Auditing and Evaluation", Evaluation Review (August 1985)
Stan Divorski, "Differences in the Approaches of Auditors and Evaluators to the Examination of Government Policies and Programs", New Directions for Evaluation (Fall 1996)
Roland Malon et al., Performance Auditing in Local Government
Frederick Mosher, The GAO: The Quest for Accountability in American Government
Journal assignment: Discuss what the performance audit report you read shows about 1) the utility of performance audits for administrators, and 2) the differences between performance auditing and program evaluation (suggested length: 3-5 pages). Cite specifics. (Note: This assignment has double-weighting for journal grading.)
7. Quantitative Evaluation Methods: Benefit‑Cost and Cost‑Effectiveness Analysis
What is the difference between benefit-cost analysis and cost-effectiveness analysis, and what are their relative advantages and disadvantages?
What are the major difficulties in doing benefit-cost analysis?
How can an analyst manipulate the results of a benefit-cost analysis?
Benefit-Cost Analysis, overview (web)
Benefit-Cost Analysis, OMB guidelines (web)
Further Reference:
Ronald D. Sylvia et al., Program Planning and Evaluation for the Public Manager, Ch. 7
Edith Stokey and Richard Zeckhauser, A Primer for Policy Analysis, Ch. 9
Edward M. Gramlich, A Guide To Benefit‑Cost Analysis
Henry M. Levin and Patrick J. McEwan, Cost-Effectiveness Analysis: Methods and Applications
T. F. Nas, Cost-Benefit Analysis
Theodore H. Poister, Public Program Analysis, Ch. 11
Mark S. Thompson, Benefit‑Cost Analysis for Program Evaluation
B. T. Yates, Analyzing Costs, Procedures, Processes, and Outcomes in Human Services
Journal assignment 1: Answer two of the questions.
Journal assignment 2: Do the homework problems. Show all of your work.
Quiz: There will be an in-class quiz on benefit-cost analysis. The quiz will be similar to the homework problems.
8. Assessing The Current and Past Practice of Program Evaluation
How good is the current and past practice of program evaluation?
Why do some program evaluations fail, and how can we improve chances for success?
How useful are evaluations to managers, and how can we make them more useful?
What potential utility does program evaluation have in the organization in which you now work, or in some other organization you have in mind?
Mark W. Lipsey et al., "Evaluation: The State of the Art and the Sorry State of the Science"
Further Reference:
Wholey et al., Ch. 25
Rossi et al., Ch. 12
Journal assignment: Answer one of the questions.
9. The Future of Program Evaluation
What are the possible future directions and prospects for program evaluation?
Further Reference:
Stewart I. Donaldson and Michael Scriven, eds., Evaluating Social Programs and Problems: Visions for a New Millennium
10. Special Topics: These are optional topics that will be covered briefly, at most, in class. These topics may interest some students as areas for possible course projects.
1) Multiattribute Evaluation
Reference:W. Edwards and J. R. Newman, Multiattribute Evaluation
2) Use of Client Surveys in Program Evaluation
Reference:Brian Stipak, "Using Clients to Evaluate Programs," ESRA, Vol. 7
M. Bush and A. C. Gordon, "The Advantages of Client Involvement in Evaluation Research," ESRA, Vol. 3
J. Nowakowksi, ed., The Client Perspective on Evaluation
3) Decision Analysis in Program Evaluation
Reference:G. Pitz and J. McKillip, Decision Analysis for Program Evaluators
4) Regression Discontinuity Evaluation Design
Reference:W. M. Trochim, Research Design for Program Evaluation: The Regression-Discontinuity Approach
L. B. Mohr, Impact Analysis for Program Evaluation, Ch. 6
R. J. Marcantonio and T. D. Cook, "Convincing Quasi-Experiments", Ch. 7 in Wholey et al.
Donald T. Campbell and David A. Kenny, A Primer on Regression Artifacts
PA555 Spring 2010
Approximate Class Schedule
|
Date |
Assignment Due |
Topics / Activities |
|
|
1 |
4/1 |
|
Introduction |
|
2 |
4/8 |
Sect 1 readings, journal assns Prepare in-class presentation |
Student commentaries Performance measurement |
|
3 |
4/15 |
Sect 2 readings, journal assns
|
Performance measurement Eval Assessment / Process Eval Qualitative methods |
|
4 |
4/22 |
Sect 3 readings, journal assn Sect 4 readings, Ch. 1, 2, 4, 5, journal assn |
Eval Assessment / Process Eval Qualitative methods Research design |
|
5 |
4/29 |
Sect 4 readings, Ch. 3, 6, 7 Begin Sect 5 readings Sect 5, journal assn 1 |
Research design |
|
6 |
5/6 |
Sect 5, journal assn 2 |
Research design
|
|
7 |
5/13 |
Sect 6, readings, journal assn |
?Quiz on research design? Performance auditing |
|
8 |
5/20 |
Sect 7 readings, journal assn 1 |
Performance auditing Benefit cost analysis
|
|
9 |
5/27
|
Sect 7 journal assn 2 Sect 8 readings, journal assn Sect 7 journal assn 2 |
Benefit cost analysis Discuss remaining topics
|
|
10 |
6/3 |
Course journals due |
Discuss remaining topics Project presentations
|
|
11 |
6/10 |
Project papers |
Project presentations Course evaluations |
|
|
*Note: This class schedule is approximate . Adjustments will occur during the term. Some topics above appear for more than one date because a topic may take more than one class period, or to allow leeway when the topic is covered. |
||