institute for employment studies

publications by IES authors

Measuring Soft Outcomes and Distance Travelled: A Review of Current Practice

Dewson S, Eccles J, Tackey N D, Jackson A
Research Report RR219, Department for Education and Employment, August 2000

research commissioned by the DfES

This summary is published as the Department for Education and Employment’s Research Brief No.219. A complete list of DfEE Research Briefs is available at DfES.

Monitoring systems for employment programmes which are targeting unemployed people have traditionally focused on ‘hard’, quantitative outcomes such as the numbers going into jobs or the numbers gaining qualifications. There is growing recognition, however, that such measures taken in isolation are inadequate in demonstrating the success of a project as a whole. In addition, these ‘hard’ measures of success do not give a complete picture of a client’s increased employability. It is often both unlikely and inappropriate for many projects to expect to achieve ‘hard’ outcomes from target groups that are socially excluded and facing multiple barriers to employment. The Institute for Employment Studies was commissioned to look at the use of ‘soft’, qualitative outcomes on projects funded by the European Social Fund (ESF). This research involved carrying out a literature review and survey of ESF-funded projects to establish what has already been done in terms of developing systems to measure soft outcomes. One of the aims of the research was to use this information to develop a model for measuring these increases in employability or ‘distance travelled’ at a national level for projects delivering support under ESF Objective 3.

Key findings

• While most ESF-funded projects are aware of the importance of measuring soft outcomes, very few have developed any systematic means of doing so.
• Where projects have set up systems to monitor soft outcomes there is a lot of variety in the types of systems set up. Systems tend to be specific to the individual project and can be subjective.
• The study recommends encouraging ESF-funded projects to set up systems to monitor soft outcomes, particularly those projects delivering support under Policy Field 2 (Equal opportunities for all and promoting social inclusion). The researchers have developed a good practice guide to help projects to do this.
• The researchers also concluded that a working group should be set up to look at ways to monitor soft outcomes at a national level.

What are ‘soft’ outcomes?

In the context of ESF Objective 3 projects and other employment programmes dealing with people disadvantaged in the labour market, soft outcomes are those which, unlike hard outcomes cannot be measured directly or tangibly. They are:

• intangible, not concrete
• subjective
• a matter of degree rather than absolute
• personal, depending on individual client needs
• intermediate (usually measuring progress towards hard outcomes such as employment and qualifications).

What are ‘soft’ indicators?

There is an interplay between indicators and outcomes in that indicators are the means by which it is possible to measure whether outcomes have been achieved. The term soft indicators is used when referring to the achievements which may ‘indicate’ acquisition or progress toward an outcome. It is clear that not all indicators, or measures of progression, are necessarily suitable for all client groups; some are target group specific.

What is ‘distance travelled’?

Put quite simply, the term distance travelled refers to the progress that a beneficiary makes towards greater employability (and subsequently harder outcomes) as a result of the project intervention. While the acquisition of certain soft outcomes may seem insignificant for certain individuals, the leap forward in achieving these outcomes can be immense for others.

What are ESF projects already doing to measure soft outcomes?

Projects reported that they measured many types of soft outcomes, most of which can be classified into two groups: core outcomes and target group specific outcomes. Core outcomes are likely to be relevant to most projects and client groups and can be classified as relating to:

• key work skills, such as the acquisition of key skills, and language and communication skills,
• attitudinal skills, for example, increased levels of motivation, confidence and self-esteem,
• personal skills, including improvements in timekeeping, attendance or personal hygiene,
• practical skills, which may be indicated by the ability to complete forms or to manage money.

Target group-specific outcomes are those which are relevant to particular groups. An example could be a project which works with people with learning disabilities or mental illness: they may identify a greater level of self-awareness, or lowered anxiety as indicators of improved personal skills.

The techniques and indicators used to measure soft outcomes vary from project to project and the scope of soft indicator systems is vast. IES found that a small number of projects do not have any systems in place to measure soft outcomes or distance travelled and their focus is primarily on measuring quantitative outcomes such as job, training and qualification successes. Conversely, some projects were much further down the line than others in terms of designing and implementing systems to measure soft outcomes and distance travelled. In particular, those projects which had received development funding seem to have been able to introduce, pilot and fine-tune sophisticated soft outcome systems for their client group.

However, from the literature and the empirical evidence reviewed as part of the study, it is clear that most ESF Objective 3 projects do actually measure soft outcomes and distance travelled albeit it, in usually informal and unsystematic ways. Most projects measure soft outcomes and (perhaps) distance travelled using paper-based techniques which examine attitudes and feelings with comparisons over time. Many projects have used ongoing reviews between trainers or assessors and clients to ‘record’ soft outcomes. These often draw on evidence produced by the client, such as portfolio evidence or examples of progress, such as working productively in a team. Reviews such as this are usually recorded on Individual Training Plans and the like although in no systematic format. Improvements over time are recorded, for example, ‘the client is feeling more confident in a group situation’ although there is no actual, formal or absolute ‘measure’ of this kind of distance travelled. Very few of the projects taking part in the review have aggregated micro-level information on individual clients’ progress to arrive at a project-wide (macro) measure of soft outcomes.

Weaknesses in ‘soft’ outcome monitoring systems

IES’s review of systems to measure soft outcomes also highlighted a number of weaknesses of such systems. These are:

• Poor integration with other sources of information. Most of the soft outcome and indicator tools reviewed did not take into account skills gained during work experience, voluntary work, or other activities as an indication of soft outcomes or distance travelled.
• Attribution. It is often difficult, if not impossible, to attribute positive, soft outcomes (and thus, distance travelled towards greater employability) solely to the training intervention.
• Subjectivity. There are some significant issues concerning the ways in which clients are assessed for soft outcomes and distance travelled. Systems which rely on self-administered questionnaires can suffer from clients overstating or understating their achievements. Similarly, tutors’ personal bias and subjectivity can affect a system of tutor-based assessments. Practitioners may also be working to their own, different definition of the same outcome, for example, what may be deemed as confidence by one tutor may be viewed as aggressive behaviour by another.
• Language. It is clear that some of the language used in soft outcome tools is inappropriate for disadvantaged client groups. Some of the language used is patronising or accusatory suggesting that there is something wrong if clients do not feel a certain way, or have certain characteristics.
• Different baselines. Although not a weakness in itself, it would seem that there are too many different client groups to devise one particular soft indicator system or model to fit all projects. The characteristics of individuals from disadvantaged ‘groups’ will vary both within and without the group and such individuality renders comparison both within and between the groups impossible.

Conclusions and Recommendations

This study has found that many disparate systems to measure soft outcomes and distance travelled exist at the project level. Whilst these systems measure similar outcomes, particularly those related to ‘core’ soft outcomes, there is no uniformity in the way projects go about measuring them.

Any nationally consistent system to measure soft outcomes and distance travelled across projects would necessarily require great prescription. All project staff would need to be similarly trained to enable them to work to the same definitions of soft outcomes and progression and measurement tools would have to be generic in order that all clients and assessors could use them. The multifarious target groups within ESF Objective 3 and other employment-related programmes renders the development of one model to measure soft outcomes and distance travelled unlikely.

However, there is a clear need for guidance to projects on soft outcomes in order to bring about more systematic measurement of these very important achievements. A Guide to Measuring Soft Outcomes and Distance Travelled has been prepared as a result of this study and is available to ESF Objective 3 projects to help them devise systems or improve the ones they already use.

Just as it is not possible to develop one model to measure soft outcomes, so too is the development of national indicators difficult. However, there is a definite need to incorporate some sort of soft outcomes monitoring at the national level if the full impact of the ESF Objective 3 programme is to be assessed. IES’s recommendations to the ESF Monitoring Committee, which have now been accepted, are to:

• undertake a longitudinal case study evaluation which explores soft outcomes and distance travelled,
• encourage ESF Objective 3 projects, particularly those delivering support under Policy Field 2 (Equal opportunities for all and promoting social inclusion) to devise soft indicator systems.
• endeavour to move towards greater routine follow-up of all ESF Objective 3 beneficiaries which will include questions relating to soft outcomes and distance travelled,
• consult more widely on developing national monitoring indicators for soft outcomes and distance travelled, and establish a working sub-group to explore the issues in more depth and to pilot various options.

Research Methodology

The study involved:

• A review of existing literature and practice on ‘soft outcomes’. The literature review drew on European and American sources of information and empirical evidence of what is happening in the realm of soft outcomes and distance travelled.
• A survey of 300 ESF-funded projects to see what systems were currently being used. The ESF Objective 3 projects taking part in the survey were funded under the 1994-99 ESF programme.
• A workshop with a small number of ESF-funded projects who had set up their own systems for measuring soft outcomes.

Measuring Soft Outcomes and Distance Travelled: A Review of Current Practice, Dewson S, Eccles J, Tackey N D, Jackson A. Research Report RR219, Department for Education and Employment, 2000.
ISBN: 978-1-84185-343-7. £4.95

• How to obtain this publication