Introduction

There are different reasons for measuring the sector’s impact, ranging from impact demonstration to impact measurement. Impact ‘measurement’ is more quantitative and externally driven whereas impact ‘demonstration’ can be seen as more internally driven and qualitative (Four Nations Voluntary Sector Group 2003).

A recent literature review, conducted by the Four Nations Voluntary Sector Research Group suggests that no single tool, approach or method can adequately capture the whole range of impacts (Four Nations Voluntary Sector Group 2002). This is both due to the diversity of the sector and the circumstances appropriate to a given organisation.

The literature review highlights ‘regeneration’ as a hotbed for existing tools, indicators and impact assessment activities. It identifies a number of approaches which pay special attention to capturing these themes.

Achieving Better Community Development (Scottish Community Development Centre) is based on the idea that the evaluation of community development should be governed by the same principles as community development itself. Evaluation is seen as an integral part of the process of community development, and community involvement as a vital part of the evaluation process.

Funders seeking to measure the impact of their project funding have also developed approaches. These include the Millennium Commission’s study of the social impact of their Millennium Award scheme and Lloyds TSB Foundation’s First Steps in Impact Assessment. First Steps uses a questionnaire as a tool for impact assessment.

Social Auditing, is a form of social accounting that has been developed, promoted and been given training throughout the UK by the Community Business Scotland (CBS) Network. Organisations that have undertaken, or are undertaking, social audits are also linked together (principally in the UK) by the Social Audit Network (SAN). A practical manual to social auditing for small organisations has also been published through the New Economics Foundation (NEF No date).

Social auditing is essentially a stakeholder centred approach to social performance measurement, which takes into account all those who are involved with, or are affected by, the organisation. It was developed in response to the fact that while most organisations regularly report on their financial performance, there is no legal obligation for them to report on their social performance. It recognises the fact that most organisations have at least some non-commercial objectives, and that there are a number of reasons why they might want to measure how well they are delivering them. It is not aimed specifically at the voluntary sector. However, it offers a particular advantage for voluntary organisations, as it provides a way of demonstrating to funders that they are achieving non-financial outcomes, which are none-the-less relevant to the organisation’s objectives and should be taken into account.

A social audit is not a one-off measurement or evaluation. It is an ongoing process that draws on information from a range of sources. Not all data need to be collected especially for the social audit. Much of the information is likely to be collected anyway as part of an organisation’s financial and personnel records. The process should be repeated every year, so that a picture of the organisation’s social performance over a number of years can be built up allowing comparisons over time to be made. Involving stakeholders in agreeing indicators and completing questionnaires means that not only can the social audit process itself contribute to increased community involvement and social capital, but stakeholders will feel some “ownership” of the audit process and are more likely to take it seriously and have an interest in the results. The interactive, “all hands on deck” nature of the social audit is also likely to help focus employees’ and other stakeholders’ ideas about what the objectives and aims of the organisation are.

Social auditing addresses a number of the issues surrounding impact measurement. The stakeholder approach means that a range of perceptions is taken into account, avoiding too narrow a view of the organisation’s impact. The use of qualitative data addresses the criticism that impact measurement is often too focused on quantitative data, and allows for better measurement of intangible and unanticipated impacts.

SCVO, in partnership with the Four Nations Voluntary Sector Research Group, has developed methodologies to assess the impact of voluntary organisations, based on the following principles:

• Defining ‘impact’ in this context as the difference made by a voluntary organisation, which may or may not have been predetermined (anticipated impact)

• Analysing impact as part of a system of interrelated activities – the ‘no organisation is an island’ approach to gauge the level of impact that is both direct and indirect, through ‘collective’ activity with other organisations or influences.

• Basing the research on the views of multiple stakeholders; partner organisations and funders, management committee and paid project co-ordinators, paid front-line staff, service users and front-line volunteers. This allows the possibility for identifying types of impact that could be construed as ‘negative’, by one or more of the stakeholders.

• Using primarily qualitative tools such as one-to-one interviews and focus groups to gauge the underlying dynamics of impact, rather than putting a number on it.

The Four Nations Voluntary Sector Research Group made the following conclusions based on their Impact Study Work. This is based on their literature review and case studies in Scotland, England, Northern Ireland and Wales in 2001/2:

• Conventional questionnaire and hard indicator-based approaches do not lend themselves to the assessment of impact that can not be predetermined or readily quantified (unanticipated impact).

• Short timescales are not conducive to the measurement of long-term impact.

• Collective impact requires the co-operation of a number of organisations or projects. Organisations may be unwilling to take part in an evaluation of collective impact over and above an interest in their own impact.

• There are also likely to be attitude or resource related barriers to measuring these particular types of impact. Organisations typically lack resources to measure the long-term impact of their activities.

• Unanticipated impacts, if not central to an organisation’s mission or objectives, are unlikely to be top priority for staff or funders. This may also apply to negative impact.

No tool can claim to assess all aspects of impact. As a result, approaches are tailored to assessing specific types of impact or to particular types of organisation. Nevertheless, there are still particular types of impact that cannot be adequately identified or assessed using current approaches. For example, no tools exist to adequately assess unanticipated impacts. Most approaches establish a framework for assessment at the beginning of the process, which might preclude the observation of unanticipated impact during the measurement process. Unanticipated impacts include the negative aspects of an organisation’s impact, rarely captured using any of the existing tools.

Similarly the short times scales of most approaches, and the need to be able to report impact upon project completion rarely allows for the assessment of long-term impact. Also, the more time that is allowed to pass between an activity and the assessment of its impacts, the more the impact of the activity will be ‘diluted’ by external factors. Finally, collective impact (impact achieved by a cluster of organisations or projects) is also difficult to measure, as it requires that the measurement process involve more than one organisation or project.

In short, these types of impact may not always be high on the list of priorities for organisations and funders for one reason or another, and this combined with the difficulties involved in measurement, means that these types of critical impact can be neglected.