institute for employment studies
publications by IES authors
Practical Methods for Evaluating Coaching
a study supported by the IES HR Network
Coaching as a development tool is increasingly prominent. However, many coaching programmes are either evaluated superficially (if at all) or only at the reaction level of the coachees. There is also very little non-partisan advice on coaching evaluation.
The IES study
The aims of the 2003-2006 research study were to:
The literature and our study suggest that for coaching to sustain credibility levels of evaluation need to increase. Suggestions arising for organisations in planning your evaluation are:
Our six featured companies come from a variety of sectors and cover a diverse range of approaches to evaluation. They include T-Mobile, Corus, a global distribution company, a building society, NHS in Wales and a UK government department.
The lessons learned from the company illustrations about using methods for collecting evaluation information will be of interest to would-be programme evaluators, and are summarised here.
If bottom-line business results are what you want, focussing on one key business indicator can be a simple approach yielding straightforward results.
It is better to plan how to evaluate the coaching before starting the programme so ‘hard’ baseline data can be collected. It is not always necessary to complicate things by calculating Return on Investment (ROI). Measuring where it is easiest provides ‘reasonable evidence’, meaning you can avoid the expense of measuring benefit elsewhere where it may be difficult to identify an appropriate measure. If you do go down the return on investment route you will need significant financial resources and statistical competence at your disposal.
Multiple viewpoints and multiple data collection methods are essential when it comes to measuring perceptions of behavioural change.
Face-to-face interviews enable behaviour change to be explored in more depth, although telephone interviews can also generate detailed information and allow probing and can be a less expensive alternative.
When designing impact questionnaires looking at behaviour change, make sure your ratings scale allows for the possibility that changes may be perceived as negative rather than positive.
Attitude surveys are a simple and non-resource-intensive method to collect reactions to coaching. Climate surveys can be useful in identifying changes in soft-skill areas such as communication, and are especially relevant to organisations implementing coaching as a style of management.
Keep survey questionnaires short to get a better response. Response rates will increase further with reminders and chasing. They can be improved by asking coaches or line managers to distribute and collect questionnaires.
Be cautious in interpreting survey findings where the size of surveyed population is small.
Comparing the results among coached with a ‘control’ group of non-coached individuals can be a very effective approach. Being able to compare before-coaching with after-coaching results can also be seen by some as more credible than examining post-coaching data alone. If you don’t have a ‘control’ you will need some form of benchmark for comparative analysis to assess whether the activity in question is relatively effective.
An evaluation framework
IES produced a provisional framework which was tested over a two-and-a half year period on an in-depth evaluation of three cohorts of a local government strategic coaching programme for HR executives. The framework was then refined.
Figure 1 summarises the two dimensions we propose as being key for would-be evaluators: three main areas of evidence sought and four main likely sources of evidence.
Source: IES, 2006
Three sets of key questions were identified relevant to the areas to seek evidence about. Evaluators need to understand the answers before the coaching begins:
The research also identified numerous perspectives that might be relevant as likely sources of evidence. However, it seems there are four main sources that it is most helpful to consider: documents, eg records of objectives, achievements, coaching contracts; coachees; coaches, whether internal or external; organisation perspective, eg line managers, sponsors, HR, staff.
The three main areas and four main sources can be presented as a simple framework, as in Figure 2.
Source: IES, 2006
This report is the product of a study supported by the IES HR Network, through which Member organisations finance, and often participate in, applied research on employment issues.
The report also contains details of how to use the framework, plus a selection of evaluation tools used by our early adopter organisation which should offer ideas and inspiration for those developing their own tools.
Practical Methods for Evaluating Coaching, Carter A. Report 430, Institute for Employment Studies, 2006.
2006 © institute for employment studies About this site