Level 3 and Level 4 Evaluation of Training
By now, Donald L. Kirkpatrick's four-level model for evaluating training has become a classic tool for HRD practitioners. Briefly, the model clarifies that four outcomes may be measured:
Level 1: Reaction: How satisfied are the trainees with the program?
Level 2: Learning: To what extent has learning occurred?
Level 3: Behavior: To what extent has on-the-job performance improved?
Level 4: Results: To what extent has training had an impact on business results?
Level 3 evaluations are significant because they reveal whether learning in the classroom was transferred to action in the workplace. Executives want to hold training programs accountable, because they want their investment to produce desired changes in behavior, improvements in on-the-job performance. But while Level 1 evaluations are simple, straightforward and inexpensive, Level 3 evaluations have been perceived to be more challenging and costly. As a result, most organizations have not attempted them.
As a matter of fact, assessment technology is available that makes effective Level 3 evaluation both simple and affordable. 20/20 Insight GOLD is an example of a multi-source feedback system that can be quickly customized and configured on-site to conduct practically any kind of survey. If the desired performance that is the focus and content of the training program can be described in behavioral terms, these items can be set up as a brief multi-source feedback survey. Administration in advance of the program establishes a quantified pre-course performance benchmark. Administration of the same survey after the program reveals whether an individual’s on-the-job performance has improved. Repeated administrations can determine whether improvement is continuing. Individual unlimited usage licenses may be used repeatedly for any type of customized assessment at no extra cost.
This evaluation technique is especially effective when used with a leadership development program such as Vital Learning’s Supervision Series, which addresses the kind of interpersonal skills that are otherwise hard to measure. The Supervision Series includes behavior-based job aids, online videos, workplace exercises, performance analysis tools and other media designed for post-course ongoing learning. When 20/20 Insight and the Supervision Series are used this way, they are a learning reinforcement system, letting trainees know how they’re doing as they work on transferring their newly acquired skills into habits.
A key point: Workplace performance (Level 3) is influenced by many other organizational factors besides training. Is the trainee given opportunities to apply the new skills on the job? Are ongoing learning and ongoing feedback available? Does he or she get encouragement and coaching? Is the incentive system compatible with or in conflict with the new behaviors? So while it’s relatively straightforward to track improvements in workplace performance, the process doesn’t evaluate training exclusively. Realistically, Level 3 evaluation produces accountability feedback for everyone who participates in the performance improvement process: trainers, trainees, the trainees’ bosses, and the managers who set the policies and provide the resources that enable the long-term effort to transfer the training to improved on-the-job performance.
Level 4 (Results) is another matter. This kind of evaluation is almost never accomplished. And for good reason! When you consider variables such as the economy, competition, technological innovation, strategic planning, marketing, availability of resources, corporate culture, just to name a few, you see that a lot more factors contribute to business results than the several factors that contribute to changed workplace behavior. The ability of training programs to impact on business results is quickly diluted among all these powerful influences. In all fairness, it may be too difficult and costly to try to define training’s contribution among all these larger factors.
Only senior executives can influence the myriad of variables that impact on business outcomes. While it doesn’t make sense to focus accountability for business results on trainers or trainees, executives and trainers can work in partnership to define the relationship between business results, desired workplace performance and training objectives—to ensure that training programs are designed to have an impact on business results and that the workplace is structured to reinforce what is learned in training.
While much has been written about the transfer of training and how to establish accountability for it, this insight published nearly half a century ago remains as valid as ever: "It is top management, through the organizational climate or reward structure it creates, that is really doing the training, regardless of what the training staff does. The training administered by the training staff 'sticks' only if it coincides with what top management is teaching every day." (J. N. Mosel, "Why Training Programs Fail to Carry Over," in Personnel, 34 (3) 1957, pp. 56-64.
Dennis E. Coates, Ph.D., CEO, Performance Support Systems
Dave Erdman, President, Vital Learning Corporation
|
