Broadly speaking, evaluation is measuring the worth or value of something. This could be a programme of work, policy or service.
There is no ‘magic formula’ to evaluation. We all make daily judgements about the worth or value of something; only evaluation requires greater critical thinking and a structured approach.
It is crucial, especially in the public sector, that people have a good understanding of evaluation, not only to justify our spending to the public, but to gather insight about a programme of work. Good evaluations go beyond cost, to understanding a range of factors.
Conducting an evaluation shouldn't be something left to ‘experts’; there are a range of tools for everyone to use and ‘evaluative thinking’ is something we all do naturally without labelling it ‘evaluation’. Whether that’s asking what the costs of a programme are, its success factors or whether there will be any negative side effects of this piece of work.
Remember, planning is everything
Evaluation is often done retrospectively and can be compromised by limited time and planning. An evaluation is not just carried out at the end of a project, it should be designed alongside delivery of a service or intervention and not as an afterthought.
If evaluation is planned early on in your project, it is likely to be more robust, enabling you to be flexible in how you measure success, and with less stress at the end for everyone involved!
The main types of evaluation:
Economic, process and impact evaluations can be conducted individually or combined. In the social sciences we love to create binaries, quant vs qual, micro vs macro, agency vs structure, but largely, it is better to apply a mixed-methods approach, especially if it’s a complex piece of work.
Compared to what?
When you are measuring the changes brought about by an intervention, it's essential that you have a comparison and create a baseline. If this is an improvement of an existing service, you need to ask what was done before, how that was measured, and compare this to the new service.
One method is using a Randomised Control Trial (RCT), which compares a group who are subject to the intervention, with another group who are not (the control group), to effectively test the intervention. This is typically considered the ‘gold standard’ of testing the value of an intervention.
Where do I start?
LOGIC MODEL
A logic model or ‘theory of change’, provides a framework for how you will measure impact. This helps to set out assumptions early on in a programme about what you will do and as a result, what change you expect to see. For more information on logic models, take a look at this report from The Strategy Unit.
STAKEHOLDER ANALYSIS
It is important to understand who your key stakeholders are, who is likely to have a high impact and how to persuade them to be engaged in the project, or those who will have little impact and need minimal engagement. This can inform how you decide the roles and responsibilities of stakeholders.
It is important that stakeholders are all on the same page when you define and agree outcomes of a programme...
EVALUATION STEP-BY-STEP
A few things to tick off and help guide you in conducting an evaluation:
- Scoping your evaluation project
- Has a similar evaluation been conducted elsewhere?
- Have you agreed why you are conducting the evaluation?
- Planning your evaluation
- Identifying stakeholders (agreeing roles & responsibilities, how will you engage stakeholders, do you have a steering group?)
- Designing your evaluation (aims & objectives, evaluation methods, resources required, relevant ethics and governance, timescales)
- Who will conduct the evaluation?
- Conducting your evaluation
- Data collection (think about both qualitative and quantitative research techniques to measuring outputs and/or outcomes)
- Data analysis (making sense of all of the data gathered)
- Disseminating your findings
- Compile report
- Communicate findings
- Implementing your findings
- How will you use your findings?
There are an endless number of guides about evaluation, and a myriad of techniques, which often makes practicing evaluation seem all the more complex.
If you only take one thing from this blog, it's that the basics of evaluation are easy to understand and that if you apply them early enough in your work and consider them throughout, you will be in a good position to manage expectations, measure and monitor outcomes and inform your decision making.
Leave a comment