CIPD - Evaluating training and learning

Learning, training and development

Revised July 2008

This factsheet gives introductory guidance. It:

explores the origins and latest thinking on the value of learning and evaluating training
emphasises the need to decide why training and learning are to be evaluated
looks at how they may be assessed
highlights the latest CIPD research.

Traditional training evaluation

In 1959, Kirkpatrick¹ first outlined four levels for training evaluation:

reactions – ‘liking or feelings for a programme’
learning - ‘principles, facts etc absorbed’
behaviour - ‘using learning on the job’
results - ‘increased production, reduced costs, etc’.

This was helpful guidance, but 30 years later Alliger and Janak² found that the relationships between the levels were weak – in other words, each level is not definitely nor always linked positively to the next.

Various surveys³ have found that most attention is focused on evaluation of training at the reactions level because of the difficulties and time costs of measuring the other three levels.

By the mid 1980s, calls began to emerge for return on investment (ROI) analyses of training efforts. Some studies found very satisfying figures^4,5 but the key criticisms of the ROI approach remain. For example, ROI provides a snapshot at only a single point in time, whereas we might want to know more about the return on training over time. Moreover, like virtually all other approaches to training evaluation, ROI focuses primarily on the training intervention rather than any planned, concurrent activities or coincidental factors that boost ongoing learning output and outcomes.

In contrast to the training-centric approach, Guzzo & Gannet⁶ conducted a useful meta-analysis of 98 studies about ‘psychologically-based interventions to raise worker productivity’. They found that training and goal setting were best, followed by financial incentives and job design having some effect, but with management by objectives and appraisal and feedback the least effective. Unfortunately, it seems that this sort of holistic study – that is, comparing what actually helps to raise performance, rather than beginning with an assumption that training is the answer – has not been repeated seriously.

The purposes of training evaluation

By the mid 1990s, Easterby-Smith⁷ was able to draw together four main strands for the purposes of training evaluation:

Proving – that the training worked or had measurable impact in itself
Controlling – for example, the time needed for training courses, access to costly off-the-job programmes, consistency or compliance requirements
Improving – for example, the training, trainers, course content and arrangements etc
Reinforcing – using evaluation efforts as a deliberate contribution to the learning process itself.

The first two strands (proving and controlling) are summative – they seek a single, acceptable answer in order to account for training. Accordingly, they tend to rely on hard data which have to be assessed often before factors other than the training itself (such as the opportunity or encouragement to use learning from training) have an impact on people’s learning and performance. The audience for summative evaluation usually centres on funders and budget holders, and they want quick answers.

The second two strands (improving and reinforcing) are formative – they seek wider understanding about training and learning processes in order to take action on future training and performance interventions. They value rich information that includes the impact of other factors (such as stakeholders’ expectations or increased responsibilities following training) on people’s longer-term learning and performance. The audience for formative evaluation includes users and designers, and they want insightful answers.

Finding long-term proof of the value of training is much more elusive than the collection of immediate evidence that encourages managerial and organisational trust in further efforts to develop learning. For example, some scientifically-conducted summative evaluations of training have found worryingly low levels of correlation between training and results, for example, Latham and Saari’s study⁸ of a training intervention which took over a year to complete. It could be argued that stripping out all the non-training variables to prove that any training has worked is flawed and contains the seeds of evaluation futility. Formative evaluations are much more about a complex reality, including the impact of the whole range of contributory factors on people’s learning and performance, and are more likely to enable ongoing learning and organisational strategy alignment – for example, as people, organisational priorities and the socio-economic environment change.

A study by Silvester et al⁹ used the formative approach, and sets the scene for discussion of the latest CIPD thinking. This research evaluated a culture change programme in a large engineering company and found that whilst all three key stakeholder groups – managers, trainers and trainees - considered the programme would produce positive outcomes, there were important differences of view in terms of expectations:

The trainees were most optimistic – perhaps because of receiving attention and the chance to learn new skills, though not necessarily to be involved in the apparent push to change the organisational culture
The trainers were the most cynical – probably through harbouring doubts about their abilities to bring about lasting change through a single form of intervention
The managers were the most pessimistic – mainly through belief that the training was all about quality not culture, and that it would thrive only in some areas.

Yet again, it is clear that any focus on evaluating training input alone may miss the point about valuing learning, and any consideration of a wider raft of interventions, strategies and expectations that encourage learning. Training is generally an instructor-led, content-based intervention, intended to lead to specific and often immediate changes in behaviour. A firmer focus on learning should result in a self-directed, work-based process leading to increased adaptive potential in the long run.

CIPD value of learning research

In 2007, we published the results of research which explored how organisations are measuring and reporting on the contribution of learning to strategic value¹⁰. The report drew on data gathered from learning, training and development (LTD) practitioners and senior operational managers in 12 UK-based organisations.

Find out more about the research report

The research indicated that most organisations still have some way to go with the development of appropriate measures. However, four main approaches to measuring and reporting on value were identified:

learning function efficiency measures
key performance indicators and benchmark measures
return on investment measures
return on expectation measures.

It is the first of these that are of most relevance in the context of this factsheet. To quote from the research report: ‘Such information is useful to LTD practitioners and addresses important questions such as:

Is the LTD function delivering operational effectiveness?
How effectively is the functional capability of the workforce being developed?
How well are learning interventions supporting critical success factors?
How do learning operations compare with those of other relevant organisations?’

Our practical online tool for CIPD members Value of learning: assessing and reporting on the value of learning to your organisation has been produced to help employers assess and encourage the alignment of learning to strategic priorities. Such alignment is key if LTD functions and practitioners wish to ensure links between their efforts and results for the organisation, if LTD is to be focused, cost-effective and more measurable in terms of valued impact. The tool contains four assessment instruments, designed to:

identify the organisation’s strategic priorities
check the current alignment of LTD
assess the extent of constructive dialogue about the LTD function’s interaction with managers and directors
discover the skills and knowledge required for the alignment process.
- Go to the Value of learning tool

Another approach that has attractions in some appropriate circumstances involves the benchmarking of training. See our factsheet on benchmarking training for more information.

Go to our Benchmarking training activity and spend factsheet

Implications and suggestions for future evaluation efforts

The focus on learning

An immediately obvious implication for future evaluation efforts is the need to focus on learning, which may be broadly defined as some permanent or long-lasting change in knowledge, skills and attitudes, and is an output or outcome, rather than on any training itself which is an input.

At a simple level, this should mean some serious reconsideration of the role and purpose of ‘happy sheets’. There needs to be a much more attention paid to questions such as ‘What new / refreshed learning did you gain?’ and ‘How will you now apply your new knowledge / skills / insights?’. These are valuable and forward-looking questions, as opposed to the potentially sterile and input-oriented questions often posed, such as ‘How do you rate the training / trainer from 1-10?’, ‘Was the venue acceptable?’ etc. Questions about learning and its application should aim to encourage individual learners to take responsibility for, first, awareness and recognition of some sort of internalised change as a direct result of an intervention: second, for incorporating that change into their wider experience of the world so far: and, third, for readiness to apply their learning and to seek positive results.

At a deeper level, change for adults is not always easy and may take considerable time. It follows, first, that if evaluation itself is to have value it needs to be conducted over time (and not just at the end of an event when people may be tired and rushing to catch trains or planes). Second, evaluation questions need to be carefully crafted – for example, to avoid the trap of allowing learners to deny responsibility for their own learning by criticising the training (or their selection for it, or the venue and catering!), to take account of the whole range of factors affecting people’s learning and performance, and to aim to reinforce the levels of trust in the contribution of learning held by influential stakeholders. Third, evaluating a particular course in isolation is no longer sufficient. What is needed is an holistic and strategic approach to evaluating learning from all its sources – and ways of ensuring that any training is fully aligned with strategic priorities.

The focus on sophisticated evaluation efforts

As already suggested, there is a need to progress beyond both ‘happy sheets’ and the evaluation of single interventions – such as a training course – deigned to bring about learning. There is a need to see not only how training can contribute but also how a coordinated range of interventions can help to deliver strategic goals. Perhaps most importantly, there is a need to address the perceptions and expectations of the learners, line and directors about the value of learning.

Our Value of learning research report¹⁰ offers a number of suggestions. Organisations could also consider extending Guzzo & Gannet’s work⁶ to evaluate the relative contribution of a range of interventions to raise performance. Two other examples of innovative approaches to the evaluation of learning itself and learning interventions follow.

An e-mail survey of line managers in a bank¹¹revealed that asking questions about the skills and sources of learning applied to deal with critical incidents experienced by the managers encouraged the recall and review of a range of learning occasions. Some managers provided extended analyses of their learning, its sources (not always training), and their reactions (all positive) when being asked to look back over years of learning. Several reported that the survey had prompted them to ‘remember what I had forgotten’ and to ‘realise how I might expect my staff to learn’. This clearly shows the benefits of an approach to reinforce learning through evaluation.

Another survey¹² looked at participants’ experiences on some talent management programmes. The results showed that whist training courses and personal development plans (PDPs) were the most commonly experienced intervention or form of support, the interventions with the greatest perceived impact – in terms of motivation, performance and commitment - were coaching and secondments. Talent management programmes, leadership skills development and training courses were ranked equal for their impact, whilst the value of PDPs was rated as low. This approach again highlights the importance of assessing the relative value ascribed by stakeholders to various components of learning.

Some evaluation questions which bring together the three focus points above and which thoughtful LTD functions might apply are (with the underlying rationale in brackets):

What was / has been the most important learning for you? Why? How did you learn this? (encouraging reflection on immediate and more distant learning experiences, and looking to uncover those insights that help people to see that they can learn and how they learn)
How does this learning match up to your expectations, and the expectations of your manager / team / the organisation? (pointing up the importance of the value expectations of various stakeholders, and the criteria they might apply to assessing learning opportunities and LTD interventions)
How can you apply / what have been the results of applying what you learned to the needs of your job, your team / department, and the strategic priorities of the organisation? (keeping the focus on the desirable and potentially rewarding links between learning and the achievement of the organisation’s strategic intentions and related hard and soft metrics)
How could you integrate what you have learned so far into further learning opportunities available to you, and to the changing demands of your job, your team / department, the organisation as a whole? (pointing the way to future learning opportunities, and looking for the multiplicative effects of various learning opportunities and methods).

References

KIRKPATRICK, D.L. (1959). Techniques for evaluating training programs. Journal of the American Society of Training and Development. Vol 33, No 11. pp3-9.
ALLIGER, G. and JANAK, E. (1989) Kirkpatrick’s levels of training criteria: thirty years later. Personnel Psychology. Vol 42, No 2. pp331-342.
THE INDUSTRIAL SOCIETY. (2000) Training evaluation. Managing Best Practice Series. No 70. London: Industrial Society. Also the AMERICAN SOCIETY FOR TRAINING AND DEVLOPMENT Annual surveys.
MATHIEU, J. and LEONARD, R. (1987) Applying utility concepts to a training program in supervisory skills: a time-based approach. Journal of Academic Management. Vol 30, No 2. pp316-335.
MORROW, C., JARRET, Q. and RUPINSKI, M. (1997). An investigation of the effect and economic utility of corporate-wide training. Personnel Psychology, Vol 50, No 1. pp91-117.
GUZZO, R. and GANNET, B. (1989) The nature of facilitators and inhibitors of effective task performance. In: SCHOORMAN, F. and SCHNEIDER, B. Facilitating work effectiveness. Lexington: Lexington Books.
EASTERBY-SMITH, M. (1994) Evaluating management development, training and education. Aldershot: Gower.
LATHAM, G. and SAARI, L. (1979) The application of social learning theory to training supervisors through behavioural modelling. Journal of Applied Psychology. Vol 64. pp239-246.
SILVESTER, J., ANDERSON, N. and PATTERSON, F. (1999). Organisational culture change: an inter-group attributional analysis. Journal of Occupational and Organisational Psychology. Vol 72, No 1. pp1-23.
ANDERSON, V. (2007) The value of learning: from return on investment to return on expectation. Research into practice. London: Chartered Institute of Personnel and Development. Available at: http://www.cipd.co.uk/bookstore
THOMSON, I.C. (1997). The introduction of policies and procedures for training evaluation in a public sector organisation. Unpublished MSc Dissertation at Birkbeck College, University of London.
PHILLIPS, P.S. (2006). What value does coaching add to talent management programmes? Selection and Development Review. Vol 22, No 5. pp5-8.

Evaluating training and learning