| |
| |
| List of Exhibits, Figures, and Tables | |
| |
| |
| Preface | |
| |
| |
| Acknowledgments | |
| |
| |
| |
| Building a Comprehensive Evaluation Process | |
| |
| |
| Challenges | |
| |
| |
| Global Evaluation Trends | |
| |
| |
| Measurement and Evaluation Challenges | |
| |
| |
| Benefits of Measurement and Evaluation | |
| |
| |
| The Myths of Measurement and Evaluation | |
| |
| |
| Key Steps and Issues | |
| |
| |
| Stakeholders | |
| |
| |
| Levels and Steps | |
| |
| |
| Chain of Impact | |
| |
| |
| ROI Process Model | |
| |
| |
| Objectives | |
| |
| |
| Evaluation Planning | |
| |
| |
| Data Collection | |
| |
| |
| Analysis | |
| |
| |
| Isolation of the Effects of Learning and Development | |
| |
| |
| Conversion of Data to Monetary Values | |
| |
| |
| The Cost of Programs | |
| |
| |
| The Return on Investment Calculation | |
| |
| |
| Intangible Benefits | |
| |
| |
| Data Reporting | |
| |
| |
| Operating Standards | |
| |
| |
| Implementation Issues | |
| |
| |
| Final Thoughts | |
| |
| |
| |
| Defining Needs and Objectives: Ensuring Business Alignment | |
| |
| |
| The Challenge | |
| |
| |
| Business Alignment Issues | |
| |
| |
| Begin with the End in Mind | |
| |
| |
| Required Discipline | |
| |
| |
| The Needs Analysis Dilemma | |
| |
| |
| Payoff Needs | |
| |
| |
| Key Questions | |
| |
| |
| Obvious vs. Not So Obvious | |
| |
| |
| The Reasons for New Programs | |
| |
| |
| Determining Costs of the Problem | |
| |
| |
| The Value of Opportunity | |
| |
| |
| To Forecast or Not to Forecast | |
| |
| |
| Business Needs | |
| |
| |
| Determining the Opportunity | |
| |
| |
| Defining the Business Measure-Hard Data | |
| |
| |
| Defining the Business Need-Soft Data | |
| |
| |
| Using Tangible vs. Intangible-A Better Approach | |
| |
| |
| Finding Sources of Impact Data | |
| |
| |
| Identifying All the Measures | |
| |
| |
| Exploring "What If...?" | |
| |
| |
| Job Performance Needs | |
| |
| |
| Analysis Techniques | |
| |
| |
| Taking a Sensible Approach | |
| |
| |
| Learning Needs | |
| |
| |
| Subject-Matter Experts | |
| |
| |
| Job and Task Analysis | |
| |
| |
| Observations | |
| |
| |
| Demonstrations | |
| |
| |
| Tests | |
| |
| |
| Management Assessment | |
| |
| |
| Preference Needs | |
| |
| |
| Key Issues | |
| |
| |
| Impact Studies | |
| |
| |
| Levels of Objectives for Programs | |
| |
| |
| Reaction and Planned Action | |
| |
| |
| Learning Objectives | |
| |
| |
| Application and Implementation Objectives | |
| |
| |
| Business Impact Objectives | |
| |
| |
| ROI Objectives | |
| |
| |
| The Importance of Specific Objectives | |
| |
| |
| Final Thoughts | |
| |
| |
| |
| Measuring Inputs and Indicators | |
| |
| |
| Measuring Input and Indicators | |
| |
| |
| Defines the Input | |
| |
| |
| Reflects Commitment | |
| |
| |
| Facilitates Benchmarking | |
| |
| |
| Explains Coverage | |
| |
| |
| Highlights Efficiencies | |
| |
| |
| Provides Cost Data | |
| |
| |
| Tracking Participants | |
| |
| |
| Tracking Hours | |
| |
| |
| Tracking Coverage by Jobs and Functional Areas | |
| |
| |
| Tracking Topics and Programs | |
| |
| |
| Tracking Requests | |
| |
| |
| Tracking Delivery | |
| |
| |
| Tracking Costs | |
| |
| |
| Pressure to Disclose All Costs | |
| |
| |
| The Danger of Costs Without Benefits | |
| |
| |
| Sources of Costs | |
| |
| |
| Learning Program Steps and Costs | |
| |
| |
| Prorated Versus Direct Costs | |
| |
| |
| Employee Benefits Factor | |
| |
| |
| Major Cost Categories | |
| |
| |
| Cost Reporting | |
| |
| |
| Tracking Efficiencies | |
| |
| |
| Tracking Outsourcing | |
| |
| |
| Tracking for the Scorecard | |
| |
| |
| Defining Key Issues | |
| |
| |
| Input Is Not Results | |
| |
| |
| Reports to Executives Should Be Minimized | |
| |
| |
| The Data Represent Operational Concerns | |
| |
| |
| This Data Must Be Automated | |
| |
| |
| Final Thoughts | |
| |
| |
| |
| Measuring Reaction and Planned Action | |
| |
| |
| Why Measure Reaction and Planned Action? | |
| |
| |
| Customer Service | |
| |
| |
| Early Feedback Is Essential | |
| |
| |
| Making Adjustments and Changes | |
| |
| |
| Predictive Capability | |
| |
| |
| For Some, This Is the Most Important Data | |
| |
| |
| Comparing Data with Other Programs | |
| |
| |
| Creating a Macro Scorecard | |
| |
| |
| Sources of Data | |
| |
| |
| Participants | |
| |
| |
| Participants' Managers | |
| |
| |
| Internal Customers | |
| |
| |
| Facilitators | |
| |
| |
| Sponsors/Senior Managers | |
| |
| |
| Areas of Feedback | |
| |
| |
| Content vs. Non-Content | |
| |
| |
| The Deceptive Feedback Cycle | |
| |
| |
| Key Areas for Feedback | |
| |
| |
| Timing of Data Collection | |
| |
| |
| Early, Detailed Feedback | |
| |
| |
| Pre-Assessments | |
| |
| |
| Collecting at Periodic Intervals | |
| |
| |
| For Long Programs with Multiple Parts | |
| |
| |
| Data Collection with Questionnaires and Surveys | |
| |
| |
| Questionnaire/Survey Design | |
| |
| |
| Intensities | |
| |
| |
| Questionnaire/Survey Response Rates | |
| |
| |
| Sample Surveys | |
| |
| |
| Data Collection with Interviews and Focus Groups | |
| |
| |
| Improving Reaction Evaluation | |
| |
| |
| Keep Responses Anonymous | |
| |
| |
| Have a Neutral Person Collect the Forms | |
| |
| |
| Provide a Copy in Advance | |
| |
| |
| Explain the Purpose of the Feedback and How It Will Be Used | |
| |
| |
| Explore an Ongoing Evaluation | |
| |
| |
| Consider Quantifying Course Ratings | |
| |
| |
| Collect Information Related to Improvement | |
| |
| |
| Allow Ample Time for Completing the Form | |
| |
| |
| Delayed Evaluation | |
| |
| |
| Ask for Honest Feedback | |
| |
| |
| Using Data | |
| |
| |
| Building the Macro-Level Scorecard | |
| |
| |
| Shortcut Ways to Measure Reaction and Planned Action | |
| |
| |
| Final Thoughts | |
| |
| |
| |
| Measuring Learning and Confidence | |
| |
| |
| Why Measure Learning and Confidence? | |
| |
| |
| The Importance of Intellectual Capital | |
| |
| |
| The Learning Organization | |
| |
| |
| The Learning Transfer Problem | |
| |
| |
| The Compliance Issue | |
| |
| |
| The Use and Development of Competencies | |
| |
| |
| The Role of Learning in Programs | |
| |
| |
| The Chain of Impact | |
| |
| |
| Certification | |
| |
| |
| Consequences of an Unprepared Workforce | |
| |
| |
| The Challenges and Benefits of Measuring Learning | |
| |
| |
| The Challenges | |
| |
| |
| The Benefits | |
| |
| |
| Measurement Issues | |
| |
| |
| Objectives | |
| |
| |
| Typical Measures | |
| |
| |
| Timing | |
| |
| |
| Cognitive | |
| |
| |
| Levels of Bloom's Taxonomy | |
| |
| |
| Data Collecton Methods | |
| |
| |
| Questionnaires/Surveys | |
| |
| |
| Objective Tests | |
| |
| |
| Criterion-Referenced Tests | |
| |
| |
| Performance Tests | |
| |
| |
| Technology and Task Simulations | |
| |
| |
| Case Studies | |
| |
| |
| Role Playing/Skill Practice | |
| |
| |
| Assessment Center Method | |
| |
| |
| Exercises/Activities | |
| |
| |
| Informal Assessments | |
| |
| |
| Administrative Issues | |
| |
| |
| Reliability and Validity | |
| |
| |
| Consistency | |
| |
| |
| Monitoring | |
| |
| |
| Pilot Testing | |
| |
| |
| Readability | |
| |
| |
| Scoring | |
| |
| |
| Reporting | |
| |
| |
| Confronting Test Failures | |
| |
| |
| Using Learning Data | |
| |
| |
| Final Thoughts | |
| |
| |
| |
| Measuring Application and Implementation | |
| |
| |
| Why Measure Application and Implementation? | |
| |
| |
| The Value of Information | |
| |
| |
| A Key Transition Time | |
| |
| |
| The Key Focus of Many Programs | |
| |
| |
| The Chain of Impact | |
| |
| |
| Barriers and Enablers | |
| |
| |
| Reward Those Who Are Most Effective | |
| |
| |
| Challenges of Measuring Application and Implementation | |
| |
| |
| Linking Application with Learning | |
| |
| |
| Designing Data Collection into Programs | |
| |
| |
| Applying Serious Effort to Level 3 Evaluation | |
| |
| |
| Including Level 3 in the Needs Assessment | |
| |
| |
| Developing ROI with Application Data | |
| |
| |
| Key Issues | |
| |
| |
| Methods | |
| |
| |
| Objectives | |
| |
| |
| Topics to Explore | |
| |
| |
| Sources | |
| |
| |
| Timing | |
| |
| |
| Responsibilities | |
| |
| |
| The Use of Questionnaires | |
| |
| |
| Progress with Objectives | |
| |
| |
| Relevance/Importance of the Program | |
| |
| |
| Knowledge/Skill Use | |
| |
| |
| Changes with Work/Action Items | |
| |
| |
| Improvements/Accomplishments | |
| |
| |
| Define the Measure | |
| |
| |
| Provide the Change | |
| |
| |
| Monetary Value | |
| |
| |
| Total Impact | |
| |
| |
| List of Other Factors | |
| |
| |
| Improvements Linked with the Program | |
| |
| |
| Perceived Value | |
| |
| |
| Links with Output Measures | |
| |
| |
| Success of the Program Team | |
| |
| |
| Barriers and Enablers | |
| |
| |
| Management Support | |
| |
| |
| Appropriateness of Program and Suggestions for Improvement | |
| |
| |
| Checklist | |
| |
| |
| Improving Response Rates | |
| |
| |
| Data Collection with Interviews | |
| |
| |
| Types of Interviews | |
| |
| |
| Interview Guidelines | |
| |
| |
| Data Collection with Focus Groups | |
| |
| |
| Applications for Focus Group Evaluation | |
| |
| |
| Guidelines | |
| |
| |
| On-the-job Observation | |
| |
| |
| Guidelines for Effective Observation | |
| |
| |
| The Use of Action Plans and Follow-Up Assignments | |
| |
| |
| Developing the Action Plan | |
| |
| |
| Successful Use of Action Plans | |
| |
| |
| Action Plan Advantages and Disadvantage | |
| |
| |
| The Use of Performance Contracts | |
| |
| |
| Transfer of Learning | |
| |
| |
| Developing ROI for Level 3 | |
| |
| |
| Data Use | |
| |
| |
| Final Thoughts | |
| |
| |
| |
| Measuring and Isolating the Impact of Programs | |
| |
| |
| Why Measure Business Impact? | |
| |
| |
| Higher-Level Data | |
| |
| |
| Breaking the Chain of Impact | |
| |
| |
| A Business Driver for Many Programs | |
| |
| |
| Show Me the Money Data | |
| |
| |
| Easy to Measure | |
| |
| |
| Common Data Types | |
| |
| |
| Types of Impact Measures | |
| |
| |
| Hard Versus Soft Data | |
| |
| |
| Tangible Versus Intangible | |
| |
| |
| Scorecards | |
| |
| |
| Specific Measures Linked to Programs | |
| |
| |
| Business Performance Monitoring | |
| |
| |
| Identify Appropriate Measures | |
| |
| |
| Convert Current Measures to Usable Ones | |
| |
| |
| Developing New Measures | |
| |
| |
| The Use of Action Plans to Develop Business Impact Data | |
| |
| |
| Set Goals and Targets | |
| |
| |
| Define the Unit of Measure | |
| |
| |
| Place a Monetary Value on Each Improvement | |
| |
| |
| Implement the Action Plan | |
| |
| |
| Provide Specific Improvements | |
| |
| |
| Isolate the Effects of the Program | |
| |
| |
| Provide a Confidence Level for Estimates | |
| |
| |
| Collect Action Plans at Specified Time Intervals | |
| |
| |
| Summarize the Data and Calculate the ROI | |
| |
| |
| Advantages of Action Plans | |
| |
| |
| Use of Performance Contracts to Measure Business Impact | |
| |
| |
| The Use of Questionnaires to Collect Business Impact Data | |
| |
| |
| When You Don't Have a Clue | |
| |
| |
| When the Measure Is a Defined Set | |
| |
| |
| When the Measure Is Known | |
| |
| |
| Response Rates | |
| |
| |
| Selecting the Appropriate Data Collection Method for Each Level | |
| |
| |
| Isolating the Effects of the Program | |
| |
| |
| Identifying Other Factors: A First Step | |
| |
| |
| Using Control Groups | |
| |
| |
| Using Trend-Line Analysis | |
| |
| |
| Forecasting | |
| |
| |
| Using Estimates | |
| |
| |
| Calculating the Impact of Other Factors | |
| |
| |
| Use of the Techniques | |
| |
| |
| Final Thoughts | |
| |
| |
| |
| Identifying Benefits and Costs, and Calculating ROI | |
| |
| |
| Why Calculate Monetary Benefits? | |
| |
| |
| Value Equals Money | |
| |
| |
| Impact Is More Understandable | |
| |
| |
| Money Is Necessary for ROI | |
| |
| |
| Monetary Value Is Needed to Understand Problems | |
| |
| |
| Key Steps to Convert Data to Money | |
| |
| |
| Standard Monetary Values | |
| |
| |
| Converting Output Data to Money | |
| |
| |
| Calculating the Cost of Quality | |
| |
| |
| Converting Employee Time Using Compensation | |
| |
| |
| Finding Standard Values | |
| |
| |
| Data Conversion When Standard Values Are Not Available | |
| |
| |
| Using Historical Costs from Records | |
| |
| |
| Using Input from Experts to Convert Soft Data | |
| |
| |
| Using Values from External Databases | |
| |
| |
| Linking with Other Measures | |
| |
| |
| Using Estimates from Participants | |
| |
| |
| Using Estimates from the Management Team | |
| |
| |
| Using Staff Estimates | |
| |
| |
| Technique Selection and Finalizing the Values | |
| |
| |
| Use the Technique Appropriate for the Type of Data | |
| |
| |
| Move from Most Accurate to Least Accurate | |
| |
| |
| Consider the Resources | |
| |
| |
| When Estimates Are Sought, Use the Source with the | |
| |
| |
| Broadest Perspective on the Issue | |
| |
| |
| Use Multiple Techniques When Feasible | |
| |
| |
| Apply the Credibility Test | |
| |
| |
| Review the Client's Needs | |
| |
| |
| Is This Another Project? | |
| |
| |
| Consider a Potential Management Adjustment | |
| |
| |
| Consider the Short-Term/Long-Term Issue | |
| |
| |
| Consider an Adjustment for the Time Value of Money | |
| |
| |
| Why Monitor Costs? | |
| |
| |
| Why Measure ROI? | |
| |
| |
| Fundamental Cost Issues | |
| |
| |
| Monitor Costs, Even If They Are Not Needed | |
| |
| |
| Cost Will Not Be Precise | |
| |
| |
| Disclose All Costs | |
| |
| |
| Fully Loaded Costs | |
| |
| |
| Reporting Costs Without Benefits | |
| |
| |
| Cost-Tracking Issues | |
| |
| |
| Prorated Versus Direct Costs | |
| |
| |
| Employee Benefits Factor | |
| |
| |
| Major Cost Categories | |
| |
| |
| Initial Analysis and Assessment | |
| |
| |
| Development of Solutions | |
| |
| |
| Acquisition Costs | |
| |
| |
| Application and Implementation Costs | |
| |
| |
| Maintenance and Monitoring | |
| |
| |
| Support and Overhead | |
| |
| |
| Evaluation and Reporting | |
| |
| |
| Cost Accumulation and Estimation | |
| |
| |
| Basic ROI Issues | |
| |
| |
| Definition | |
| |
| |
| Annualized Values: A Fundamental Concept | |
| |
| |
| BCR/ROI Calculations | |
| |
| |
| Benefit/Cost Ratio | |
| |
| |
| ROI Formula | |
| |
| |
| ROI Targets | |
| |
| |
| ROI Is Not for Every Program | |
| |
| |
| Other ROI Measures | |
| |
| |
| Payback Period | |
| |
| |
| Discounted Cash Flow | |
| |
| |
| Final Thoughts | |
| |
| |
| |
| Measuring the Hard to Measure and the Hard to Value: Intangible Benefits | |
| |
| |
| Why Intangibles Are Important | |
| |
| |
| Intangibles Are the Invisible Advantage | |
| |
| |
| We Are Entering the Intangible Economy | |
| |
| |
| More Intangibles Are Converted to Tangibles | |
| |
| |
| Intangibles Drive Programs | |
| |
| |
| Measurement and Analysis of Intangibles | |
| |
| |
| Measuring the Intangibles | |
| |
| |
| Converting to Money | |
| |
| |
| Identifying Intangibles | |
| |
| |
| Analyzing Intangibles | |
| |
| |
| Customer Service | |
| |
| |
| Team Effectiveness | |
| |
| |
| Cooperation/Conflict | |
| |
| |
| Decisiveness/Decision Making | |
| |
| |
| Communication | |
| |
| |
| Innovation and Creativity | |
| |
| |
| Innovation | |
| |
| |
| Creativity | |
| |
| |
| Employee Attitudes | |
| |
| |
| Employee Satisfaction | |
| |
| |
| Organizational Commitment | |
| |
| |
| Employee Engagement | |
| |
| |
| Employee Capability | |
| |
| |
| Experience | |
| |
| |
| Knowledge | |
| |
| |
| Learning | |
| |
| |
| Competencies | |
| |
| |
| Educational Level | |
| |
| |
| Attention | |
| |
| |
| Leadership | |
| |
| |
| 360-Degree Feedback | |
| |
| |
| Leadership Inventories | |
| |
| |
| Leadership Perception | |
| |
| |
| Job Creation and Acquisition | |
| |
| |
| Productivity Versus Job Growth | |
| |
| |
| Importance of Job Creation and Growth | |
| |
| |
| Recruitment Sourcing and Effectiveness | |
| |
| |
| Recruitment Efficiency | |
| |
| |
| Stress | |
| |
| |
| Networking | |
| |
| |
| Final Thoughts | |
| |
| |
| |
| Reporting Results | |
| |
| |
| Why the Concern About Communicating Results? | |
| |
| |
| Communication Is Necessary to Make Improvements | |
| |
| |
| Communication Is Necessary to Explain Contributions | |
| |
| |
| Communication Is a Politically Sensitive Issue | |
| |
| |
| Different Audiences Need Different Information | |
| |
| |
| Principles of Communicating Results | |
| |
| |
| Communication Must Be Timely | |
| |
| |
| Communication Should Be Targeted to Specific Audiences | |
| |
| |
| Media Should Be Carefully Selected | |
| |
| |
| Communication Should Be Unbiased and Modest | |
| |
| |
| Communication Must Be Consistent | |
| |
| |
| Testimonials Are More Effective Coming from Respected Individuals | |
| |
| |
| The Audience's Opinion of the Program Will Influence the Communication Strategy | |
| |
| |
| The Process for Communicating Results | |
| |
| |
| The Need for Communication | |
| |
| |
| Planning the Communications | |
| |
| |
| The Audience for Communications | |
| |
| |
| Basis for Selecting the Audience | |
| |
| |
| Information Development: The Impact Study | |
| |
| |
| Communication Media Selection | |
| |
| |
| Meetings | |
| |
| |
| Interim and Progress Reports | |
| |
| |
| Routine Communication Tools | |
| |
| |
| E-Mail and Electronic Media | |
| |
| |
| Program Brochures and Pamphlets | |
| |
| |
| Case Studies | |
| |
| |
| Presenting Information | |
| |
| |
| Routine Feedback on Program Progress | |
| |
| |
| The Presentation of Results to Senior Management | |
| |
| |
| Streamlining the Communication | |
| |
| |
| Building Scorecards | |
| |
| |
| Reactions to Communication | |
| |
| |
| Using Evaluation Data | |
| |
| |
| Final Thoughts | |
| |
| |
| |
| Implementing and Sustaining a Comprehensive Evaluation System | |
| |
| |
| Why the Concern About Implementing and Sustaining Evaluation? | |
| |
| |
| Resistance Is Always Present | |
| |
| |
| Implementation Is Key | |
| |
| |
| Consistency Is Needed | |
| |
| |
| Efficiency Is Necessary | |
| |
| |
| Implementing the Process: Overcoming Resistance | |
| |
| |
| Assessing the Climate | |
| |
| |
| Developing Roles and Responsibilities | |
| |
| |
| Identifying a Champion | |
| |
| |
| Developing the Evaluation Leader | |
| |
| |
| Establishing a Task Force | |
| |
| |
| Assigning Responsibilities | |
| |
| |
| Establishing Goals and Plans | |
| |
| |
| Setting Evaluation Targets | |
| |
| |
| Developing a Timetable for Implementation | |
| |
| |
| Revising or Developing Policies and Guidelines | |
| |
| |
| Preparing the L&D Team | |
| |
| |
| Involving the L&D Team | |
| |
| |
| Using Measurement and Evaluation as a Learning Tool | |
| |
| |
| Teaching the L&D Team | |
| |
| |
| Initiating Evaluation Studies | |
| |
| |
| Selecting the Initial Program | |
| |
| |
| Developing the Planning Documents | |
| |
| |
| Reporting Progress | |
| |
| |
| Establishing Discussion Groups | |
| |
| |
| Preparing the Sponsors and Management Team | |
| |
| |
| Removing Obstacles | |
| |
| |
| Dispelling Myths | |
| |
| |
| Delivering Bad News | |
| |
| |
| Monitoring Progress | |
| |
| |
| Final Thoughts | |
| |
| |
| |
| How Results-Based Are Your Workplace | |
| |
| |
| Learning and Performance Programs? An Assessment for the L&D Staff | |
| |
| |
| Glossary | |
| |
| |
| Index | |
| |
| |
| About the Authors | |