| |
| |
| Preface | |
| |
| |
| |
| Program Evaluation: An Overview | |
| |
| |
| Evaluation Tasks That Need to Be Done | |
| |
| |
| Verify That Resources Would Be Devoted to Meeting Unmet Needs | |
| |
| |
| Verify That Implemented Programs Do Provide Services | |
| |
| |
| Examine the Outcomes | |
| |
| |
| Determine Which Programs Produce the Most Favorable Outcomes | |
| |
| |
| Select the Programs That Offer the Most Needed Types of Services | |
| |
| |
| Provide Information to Maintain and Improve Quality | |
| |
| |
| Watch for Unplanned Side Effects | |
| |
| |
| Common Types of Program Evaluations | |
| |
| |
| Assess Needs of the Program Participants | |
| |
| |
| Examine the Process of Meeting the Needs | |
| |
| |
| Measure the Outcomes of the Program | |
| |
| |
| Integrate the Needs, Costs, and Outcomes | |
| |
| |
| Activities Often Confused with Program Evaluation | |
| |
| |
| Different Types of Evaluations for Different Kinds of Programs | |
| |
| |
| Organizations Needing Program Evaluations | |
| |
| |
| Time Frames of Needs | |
| |
| |
| Extensiveness of Programs | |
| |
| |
| Purpose of Program Evaluation | |
| |
| |
| The Roles of Evaluators | |
| |
| |
| A Variety of Work Settings | |
| |
| |
| Comparison of Internal and External Evaluators | |
| |
| |
| Evaluation and Service | |
| |
| |
| Evaluation and Related Activities of Organizations | |
| |
| |
| Summary and Preview | |
| |
| |
| Study Questions | |
| |
| |
| Additional Resource | |
| |
| |
| |
| Planning an Evaluation | |
| |
| |
| An Overview of Evaluation Models | |
| |
| |
| The Traditional Model | |
| |
| |
| Social Science Research Model | |
| |
| |
| Industrial Inspection Model | |
| |
| |
| Black Box Evaluation | |
| |
| |
| Objectives-Based Evaluation | |
| |
| |
| Goal-Free Evaluation | |
| |
| |
| Fiscal Evaluation | |
| |
| |
| Accountability Model | |
| |
| |
| Expert Opinion Model | |
| |
| |
| Naturalistic or Qualitative Model | |
| |
| |
| Success Case Method | |
| |
| |
| Empowerment Evaluation | |
| |
| |
| Theory-Driven Evaluation | |
| |
| |
| An Improvement-Focused Approach | |
| |
| |
| Steps in Preparing to Conduct an Evaluation | |
| |
| |
| Identify the Program and Its Stakeholders | |
| |
| |
| Become Familiar with Information Needs | |
| |
| |
| Plan the Evaluation | |
| |
| |
| Dysfunctional Attitudes Toward Program Evaluation | |
| |
| |
| Assume That the Program Is Perfect | |
| |
| |
| Fear That the Evaluation Will Offend the Staff | |
| |
| |
| Fear That the Evaluation Will Inhibit Innovation | |
| |
| |
| Fear That the Program Will Be Terminated | |
| |
| |
| Fear That Information Will Be Misused | |
| |
| |
| Fear That Qualitative Understanding May Be Supplanted | |
| |
| |
| Fear That Evaluation Drains Program Resources | |
| |
| |
| Fear of Losing Control of the Program | |
| |
| |
| Fear That Evaluation Has Little Impact | |
| |
| |
| The Effect of These Attitudes | |
| |
| |
| Summary and Preview | |
| |
| |
| Study Questions | |
| |
| |
| Additional Resource | |
| |
| |
| |
| Selecting Criteria and Setting Standards | |
| |
| |
| Useful Criteria and Standards | |
| |
| |
| Criteria That Reflect a Program's Purposes | |
| |
| |
| Criteria That the Staff Can Influence | |
| |
| |
| Criteria That Can Be Measured Reliably and Validly | |
| |
| |
| Criteria That Stakeholders Participate in Selecting | |
| |
| |
| Developing Goals and Objectives | |
| |
| |
| How Much Agreement on Goals Is Needed? | |
| |
| |
| Different Types of Goals | |
| |
| |
| Goals That Apply to All Programs | |
| |
| |
| Evaluation Criteria and Evaluation Questions | |
| |
| |
| Does the Program or Plan Match the Values of the Stakeholders? | |
| |
| |
| Does the Program or Plan Match the Needs of the People to Be Served? | |
| |
| |
| Does the Program as Implemented Fulfill the Plans? | |
| |
| |
| Do the Outcomes Achieved Match the Goals? | |
| |
| |
| Using Program Theory | |
| |
| |
| Is the Program Accepted? | |
| |
| |
| Are the Resources Devoted to the Program Being Expended Appropriately? | |
| |
| |
| Some Practical Limitations in Selecting Evaluation Criteria | |
| |
| |
| Evaluation Budget | |
| |
| |
| Time Available for the Project | |
| |
| |
| Criteria That Are Credible to the Stakeholders | |
| |
| |
| Summary and Preview | |
| |
| |
| Study Questions | |
| |
| |
| Additional Resource | |
| |
| |
| |
| Developing Measures | |
| |
| |
| Sources of Data for Evaluation | |
| |
| |
| Intended Beneficiaries of the Program | |
| |
| |
| Providers of Services | |
| |
| |
| Observers | |
| |
| |
| Which Sources Should Be Used? | |
| |
| |
| Good Assessment Procedures | |
| |
| |
| Use Multiple Variables | |
| |
| |
| Use Nonreactive Measures | |
| |
| |
| Use Variables Relevant to Information Needs | |
| |
| |
| |
| Using Multiple Measures in an Evaluation of a Summer Community Program for Youth | |
| |
| |
| Use Valid Measures | |
| |
| |
| Use Reliable Measures | |
| |
| |
| Use Measures That Can Detect Change | |
| |
| |
| Use Cost-Effective Measures | |
| |
| |
| Types of Measures of Evaluation Criteria | |
| |
| |
| Written Surveys and Interviews with Program Participants | |
| |
| |
| Checklists, Tests, and Records | |
| |
| |
| Preparing Special Surveys | |
| |
| |
| Format of a Survey | |
| |
| |
| Preparing Survey Items | |
| |
| |
| Instructions and Pretests | |
| |
| |
| Summary and Preview | |
| |
| |
| Study Questions | |
| |
| |
| Additional Resource | |
| |
| |
| |
| Ethics in Program Evaluation | |
| |
| |
| Standards for the Practice of Evaluation | |
| |
| |
| Ethical Issues Involved in the Treatment of People | |
| |
| |
| Compensating for Ineffective, Novel Treatments | |
| |
| |
| Obtaining Informed Consent | |
| |
| |
| Maintaining Confidentiality | |
| |
| |
| Role Conflicts Facing Evaluators | |
| |
| |
| Recognizing the Needs of Different Stakeholders | |
| |
| |
| Program Managers Are Concerned with Efficiency | |
| |
| |
| Staff Members Seek Assistance in Service Delivery | |
| |
| |
| Clients Want Effective and Appropriate Services | |
| |
| |
| Community Members Want Cost-Effective Programs | |
| |
| |
| The Validity of Evaluations | |
| |
| |
| Valid Measurement Instruments | |
| |
| |
| Skilled Data Collectors | |
| |
| |
| Appropriate Research Design | |
| |
| |
| Adequate Descriptions of Program and Procedures | |
| |
| |
| Avoiding Possible Negative Side Effects of Evaluation Procedures | |
| |
| |
| Can Someone Be Hurt by Inaccurate Findings? | |
| |
| |
| Consider Statistical Type II Errors | |
| |
| |
| Pay Attention to Unplanned Effects | |
| |
| |
| Analyze Implicit Values Held by the Evaluator | |
| |
| |
| Institutional Review Boards and Program Evaluation | |
| |
| |
| Ethical Problems Evaluators Report | |
| |
| |
| Summary and Preview | |
| |
| |
| Study Questions | |
| |
| |
| Additional Resource | |
| |
| |
| |
| The Assessment of Need | |
| |
| |
| Definitions of Need | |
| |
| |
| Sources of Information for the Assessment of Need | |
| |
| |
| Describing the Current Situation | |
| |
| |
| Social Indicators of Need | |
| |
| |
| Community Surveys of Need | |
| |
| |
| Services Already Available | |
| |
| |
| Key Informants | |
| |
| |
| Focus Groups and Open Forums | |
| |
| |
| Inadequate Assessment of Need | |
| |
| |
| Failing to Examine Need | |
| |
| |
| Failing to Examine the Context of Need | |
| |
| |
| Failing to Relate Need to Implementation Plans | |
| |
| |
| Failing to Deal with Ignorance of Need | |
| |
| |
| Using Needs Assessments in Program Planning | |
| |
| |
| Summary and Preview | |
| |
| |
| Study Questions | |
| |
| |
| Additional Resource | |
| |
| |
| |
| Monitoring the Operation of Programs | |
| |
| |
| Monitoring Programs as a Means of Evaluating Programs | |
| |
| |
| What to Summarize with Information Systems | |
| |
| |
| Relevant Information | |
| |
| |
| Actual State of Program | |
| |
| |
| Program Participants | |
| |
| |
| Providers of Services | |
| |
| |
| Program Records and Information Systems | |
| |
| |
| Problems with Agency Records | |
| |
| |
| Increasing the Usefulness of Records | |
| |
| |
| How Records Can Be Used to Monitor Programs | |
| |
| |
| Reporting Information Separately for Each Therapist | |
| |
| |
| Developing Information Systems for Agencies | |
| |
| |
| Threatening Uses of Information Systems | |
| |
| |
| Avoiding Common Problems in Implementing an Information System | |
| |
| |
| Guard Against the Misuse of the Information | |
| |
| |
| Avoid Setting Arbitrary Standards | |
| |
| |
| Avoid Serving the Needs of Only One Group | |
| |
| |
| Avoid Duplicating Records | |
| |
| |
| Avoid Adding to the Work of the Staff | |
| |
| |
| Avoid a Focus on Technology | |
| |
| |
| Summary and Preview | |
| |
| |
| Study Questions | |
| |
| |
| Additional Resource | |
| |
| |
| |
| Qualitative Evaluation Methods | |
| |
| |
| Evaluation Settings Best Served by Qualitative Evaluations | |
| |
| |
| Admission to Graduate Studies | |
| |
| |
| Dissatisfaction with a Library Collection | |
| |
| |
| Evaluating a Political Campaign | |
| |
| |
| Gathering Qualitative Information | |
| |
| |
| The Central Importance of the Observer | |
| |
| |
| Observational Methods | |
| |
| |
| |
| Using Qualitative Methods in an Evaluation of a University Library | |
| |
| |
| Interviewing to Obtain Qualitative Information | |
| |
| |
| Carrying Out Naturalistic Evaluations | |
| |
| |
| |
| Making Unrestricted Observations | |
| |
| |
| |
| Integrating Impressions | |
| |
| |
| |
| Sharing Interpretations | |
| |
| |
| |
| Preparing Reports | |
| |
| |
| Are Qualitative Evaluations Subjective? | |
| |
| |
| Coordinating Qualitative and Quantitative Methods | |
| |
| |
| The Substance of the Evaluation | |
| |
| |
| Getting Insights from the Most Successful Participants | |
| |
| |
| Changing Emphases as Understanding Expands | |
| |
| |
| The Evaluation Questions | |
| |
| |
| Cost of Evaluation | |
| |
| |
| Philosophical Assumptions | |
| |
| |
| Summary and Preview | |
| |
| |
| Study Questions | |
| |
| |
| Additional Resource | |
| |
| |
| |
| Single-Group, Nonexperimental Outcome Evaluations | |
| |
| |
| Single-Group Evaluation Designs | |
| |
| |
| Observe Only After the Program | |
| |
| |
| Observe Before and After the Program | |
| |
| |
| Uses of Single-Group, Descriptive Designs | |
| |
| |
| Did the Participants Meet a Criterion? | |
| |
| |
| Did the Participants Improve? | |
| |
| |
| |
| A Pretest-Posttest Design to Evaluate a Peer-Based Program to Prevent Skin Cancer | |
| |
| |
| Did the Participants Improve Enough? | |
| |
| |
| Relating Change to Service Intensity and Participant Characteristics | |
| |
| |
| Threats to Internal Validity | |
| |
| |
| Actual but Nonprogram-Related Changes in the Participants | |
| |
| |
| Apparent Changes Dependent on Who Was Observed | |
| |
| |
| Changes Related to Methods of Obtaining Observations | |
| |
| |
| Effects of Interactions of These Threats | |
| |
| |
| Internal Validity Threats Are Double-Edged Swords | |
| |
| |
| Construct Validity in Pretest-Posttest Designs | |
| |
| |
| Overinterpreting the Results of Single-Group Designs | |
| |
| |
| Usefulness of Single-Group Designs as Initial Approaches to Program Evaluation | |
| |
| |
| Assessing the Usefulness of Further Evaluations | |
| |
| |
| Correlating Improvement with Other Variables | |
| |
| |
| Preparing the Facility for Further Evaluation | |
| |
| |
| Summary and Preview | |
| |
| |
| Study Questions | |
| |
| |
| Additional Resource | |
| |
| |
| |
| Quasi-Experimental Approaches to Outcome Evaluation | |
| |
| |
| Making Numerous Observations | |
| |
| |
| Time-Series Designs | |
| |
| |
| Patterns of Outcomes Over Time Periods | |
| |
| |
| Analysis of Time-Series Designs | |
| |
| |
| Observing Other Groups | |
| |
| |
| Nonequivalent Control Group Designs | |
| |
| |
| Problems in Selecting Comparison Groups | |
| |
| |
| |
| Nonequivalent Control Groups Used to Evaluate an Employee Incentive Plan | |
| |
| |
| Regression-Discontinuity Design | |
| |
| |
| Observing Other Dependent Variables | |
| |
| |
| Combining Designs to Increase Internal Validity | |
| |
| |
| Time-Series and Nonequivalent Control Groups | |
| |
| |
| Selective Control Design | |
| |
| |
| Summary and Preview | |
| |
| |
| Study Questions | |
| |
| |
| Additional Resource | |
| |
| |
| |
| Using Experiments to Evaluate Programs | |
| |
| |
| Experiments in Program Evaluation | |
| |
| |
| Benefits of Experiments | |
| |
| |
| Experimental Designs | |
| |
| |
| Objections to Experimentation | |
| |
| |
| Don't Experiment on Me! | |
| |
| |
| We Already Know What Is Best | |
| |
| |
| I Know What Is Best for My Client | |
| |
| |
| Experiments Are Just Too Much Trouble | |
| |
| |
| The Most Desirable Times to Conduct Experiments | |
| |
| |
| When a New Program Is Introduced | |
| |
| |
| When Stakes Are High | |
| |
| |
| When There Is Controversy About Program Effectiveness | |
| |
| |
| When Policy Change Is Desired | |
| |
| |
| |
| Teaching Doctors Communication Skills: An Evaluation with Random Assignment and Pretests | |
| |
| |
| When Demand Is High | |
| |
| |
| Getting the Most out of an Experimental Design | |
| |
| |
| Take Precautions Before Data Collection | |
| |
| |
| Keep Track of Randomization While the Experiment Is in Progress | |
| |
| |
| Analyze the Data Reflectively | |
| |
| |
| Summary and Preview | |
| |
| |
| Study Questions | |
| |
| |
| Additional Resource | |
| |
| |
| |
| Analyses of Costs and Outcomes | |
| |
| |
| Cost Analyses and Budgets | |
| |
| |
| Types of Costs | |
| |
| |
| An Example Budget | |
| |
| |
| The Necessity of Examining Costs | |
| |
| |
| Comparing Outcomes to Costs | |
| |
| |
| The Essence of Cost-Benefit Analysis | |
| |
| |
| The Essence of Cost-Effectiveness Analysis | |
| |
| |
| When Outcomes Cannot Be Put into the Same Units | |
| |
| |
| Some Details of Cost Analyses | |
| |
| |
| Units of Analysis Must Reflect the Purpose of the Program | |
| |
| |
| Future Costs and Benefits Are Estimated | |
| |
| |
| Who Pays the Costs and Who Reaps the Benefits? | |
| |
| |
| |
| The Value of Providing Smoking Cessation Clinics for Employees on Company Time | |
| |
| |
| Using Cost-Benefit and Cost-Effectiveness Analyses | |
| |
| |
| Major Criticisms of Cost Analyses | |
| |
| |
| The Worth of Psychological Benefits Is Hard to Estimate | |
| |
| |
| Placing a Value on Lives Seems Wrong | |
| |
| |
| Cost-Benefit and Cost-Effectiveness Analyses Require Many Assumptions | |
| |
| |
| Summary and Preview | |
| |
| |
| Study Questions | |
| |
| |
| Additional Resource | |
| |
| |
| |
| Evaluation Reports: Interpreting and Communicating Findings | |
| |
| |
| Developing a Communication Plan | |
| |
| |
| Explore Stakeholder Information Needs | |
| |
| |
| Plan Reporting Meetings | |
| |
| |
| Set a Communication Schedule | |
| |
| |
| Personal Presentations of Findings | |
| |
| |
| Need for Personal Presentations | |
| |
| |
| Content of Personal Presentations | |
| |
| |
| Audience for the Personal Presentations | |
| |
| |
| Distributing Drafts of Reports | |
| |
| |
| Content of Formal Written Evaluation Reports | |
| |
| |
| Remember the Purposes of the Formal Report | |
| |
| |
| Provide an Outline and a Summary | |
| |
| |
| Describe the Context of the Evaluation | |
| |
| |
| Describe the Program Participants | |
| |
| |
| Justify the Criteria Selected | |
| |
| |
| Describe the Data-Gathering Procedures | |
| |
| |
| Provide the Findings | |
| |
| |
| Develop Recommendations | |
| |
| |
| Formal Reports Should Look Attractive | |
| |
| |
| Provide Progress Reports and Press Releases | |
| |
| |
| Summary and Preview | |
| |
| |
| Study Questions | |
| |
| |
| Additional Resources | |
| |
| |
| |
| How to Encourage Utilization | |
| |
| |
| Obstacles to Effective Utilization | |
| |
| |
| Constraints on Managers | |
| |
| |
| Value Conflicts Among Stakeholders | |
| |
| |
| Misapplied Methodology | |
| |
| |
| Evaluating a Program at Arm's Length | |
| |
| |
| Dealing with Mixed Findings | |
| |
| |
| Don't Abdicate Your Responsibility | |
| |
| |
| Don't Take the Easy Way Out | |
| |
| |
| Show How to Use the Evaluation to Improve the Program | |
| |
| |
| Using Evaluations When an Innovative Program Seems No Better than Other Treatments | |
| |
| |
| When Can Evaluators Be Sure Groups Do Not Differ? | |
| |
| |
| Are Evaluations Valuable Even When No Advantage for the Innovation Is Found? | |
| |
| |
| |
| Evaluations of the Outcomes of Boot Camp Prisons: The Value of Finding No Differences Between Program and Comparison Groups | |
| |
| |
| Developing a Learning Culture | |
| |
| |
| Work with Stakeholders | |
| |
| |
| Adopt Developmental Interpretations | |
| |
| |
| Frame Findings in Terms of Improvements | |
| |
| |
| Treat Findings as Tentative Indicators, Not Final Answers | |
| |
| |
| Recognize Service Providers' Needs | |
| |
| |
| Keep Evaluation Findings on the Agency's Agenda | |
| |
| |
| The Evaluation Attitude | |
| |
| |
| Summary and Possible Trends for Program Evaluation | |
| |
| |
| Study Questions | |
| |
| |
| Additional Resource | |
| |
| |
| |
| Illustrative Program Evaluation Report | |
| |
| |
| References | |
| |
| |
| Name Index | |
| |
| |
| Subject Index | |