Why is it extremely difficult to analyze the results of training programs?

Analysing the results of training programs is extremely difficult. L&D teams are challenged to demonstrate the value that programs deliver to both individual employees and the organisation as a whole. Unfortunately, few learning leaders are able to do this.

“92% of senior leaders and stakeholders want to see quantifiable results from their training functions. Yet only 9% of internal L&D organisations show them anything,” notes Paul Sampark, Ph.D., an industrial and organisational psychologist and training measurement expert.

So then, how should L&D teams go about going beyond the smile sheet to measure ROI?

Enter the Kirkpatrick Evaluation Model, perhaps the best-known model of evaluating the results of training programs. The Kirkpatrick Evaluation Model is widely used because it measures training outcomes at multiple levels. So, organisations can use the model according to their own definition of the word “evaluation.” What might be important to one team in one organiSation might not be to another.

Are you showing your senior leadership team how much ROI your training initiatives are delivering to the bottom line?

The Kirkpatrick Model’s Four Levels of Training Evaluation

The Kirkpatrick Evaluation Model was created by Donald Kirkpatrick and measures training through four levels, each of increasing depth or complexity:

Level 1 – Reaction

This basic level asks learners about how they felt about the training. Questions might be as simple as:

• Did you enjoy it?

• Did you feel it was valuable?

• Did you feel that the instructor did a good job?

These questions are usually delivered via a simple post-course questionnaire. While self-reported answers from learners might seem superficial, there is still value in providing Level 1 evaluation.

Level 1 can be helpful for instructional design teams to broadly understand whether the course resonated with learners. For example, overwhelmingly negative or even neutral reviews are a clear review that things could have been better.

Level 2 – Learning

This next level of Kirkpatrick seeks to help L&D managers determine whether learning occurred or any knowledge/skills were acquired. It gives a chance for learners to apply what they learned.

To do this, instructional design teams will usually deliver several assessments. There can be the expected one at the end of the training, but this doesn’t help managers understand how much they learned.

For a deeper look at how much knowledge or skills were gained, the L&D team can give multiple assessments. For example, one assessment at the beginning and then one similar one at the end of the course—or even months after the training has passed. This insight helps to determine how much the learner already knew and how useful or impactful the training really was.

Level 3 – Behaviour

Kirkpatrick’s Level 3 goes even deeper than Level 2 measurement. It seeks to determine just how much of the training is impacting the employee’s day-to-day work. It also looks at whether the training is relevant to their jobs.

This type of behaviour measurement requires quite a big leap from Level 2 of the model. It does not rely simply on answers to post-training assessments. To carry out Level 3, learning leaders will need to garner much more feedback, including that from co-workers, managers, and others who work regularly alongside the learner.

If the training course has had the desired effect, it will be noticeable—both qualitatively and quantitatively—to everyone involved. Additionally, a lack of feedback is also itself feedback. Subtle clues can help L&D better understand whether the training had any influence on the employee’s work.

Level 4 – Results

The highest level of the Kirkpatrick Evaluation Model has L&D determine whether the training results in a positive impact on the overall organisation and key business performance indicators.

For some departments or domains, the desired results might be clear: sales training led to more sales, which affected revenue. Or, stronger customer care training led to fewer returns or chargebacks.

Beyond scenarios like these, it can be challenging for L&D to determine the organisational impact of training. For example, measuring the impact of an empathy or leadership training course. To measure results and the impact of these types of training, L&D need to work with managers to determine metrics that can serve as the signals of positive employee performance.

You and your team put a lot of time and effort into creating training.

How do you use the Kirkpatrick Evaluation Model?

The beauty of the Kirkpatrick Model is that any level can be used anytime in any learning setting. Level 1 might be used immediately just to garner immediate learner feedback. But Levels 2 and 3 might not be implemented for 1-2 months after the training has passed.

Kirkpatrick also helps teams better understand where instructional design needs to improve. For example, say that in Levels 1 and 2 there is consistent feedback that the courses did not meet expectations or assessments showed poor retention. This demonstrates an issue not be with learners, but with the courses or course design.

Is the Kirkpatrick Evaluation Model the best way to measure learning effectiveness?

Measuring the impact of your training not only earns you more respect and partnership from business leaders. But you will also receive more recognition and rewards for your hard work—including more budget.

This budget could be used for hiring more designers and developers or outside subject matter experts. These extra resources can build courses based on training requests that you, unfortunately, had to turn down previously.

For some organisations, the Kirkpatrick Model doesn’t go high enough. Level 4 might seem the highest, as it seeks to measure the impact to the organisation as a whole. However, it might still be too vague for some organisations.

As a result, some L&D teams have added two additional levels: Level 5 and Level 6.

Level 5 – Return on Investment

In some organizations, Level 4 is pushed to Level 5, a Return on Investment (ROI). This measures any financial impact the training initiative has on the organization, such as increased revenues and lowered costs.

So Level 4 should measure, as a whole, the positive performance and behaviours studied in Level 3.

Dr. Sampark has Level 4 include a control group and attribution technique. L&D should track both trained and untrained participants, in order to measure learning impact by groups. In this way, an organisation can measure the attribution of training to large groups of employees.

Beyond this level, an organisation can go deeper to seek the financial impact, which would comprise Level 5.

Level 6 – Maximising Impact

At this level, Dr. Sampark suggests that organisations focus on the factors that are influencing the impact of a training program. Many factors beyond obvious training outcomes can easily influence the results at each level of impact. Without understanding and addressing these factors, the organisation will never improve, no matter the quality of the training.

For instance, Dr. Sampark suggests examining which “climate factors” will influence the impact and ROI of a training initiative. For example, superior manager support may lead to behavioural improvements (Level 3), greater business impact (Level 4), and a higher ROI (Level 5).