Evaluating appraisals training at The Church of Scotland

Eleanor Mackenzie, Learning and Development Officer at The Church of Scotland, demonstrates that you don’t have to be a data whizz to prove your programme packs a punch

Member fact file:

The Church of Scotland has been a member of the Charity Learning Consortium since 2017 and uses both the learning management system and eLearning that we provide

“Facts and stats are great for monitoring, but ultimately the real data comes from the change that you see in people, the organisation and culture.”
Eleanor Mackenzie, The Church of Scotland

Business need

At The Church of Scotland, there was a general feeling after the pandemic of needing more support to manage the transition back to work. It was therefore more important than ever to ensure that annual appraisals were really effective, and embedded into every department and team.

Completing an appraisal with their manager provides an opportunity for people to talk about what is important to them. They can share their concerns about workload, home working and wellbeing and provide feedback to the organisation on what they need to make them happier and more effective in their roles.

However, in the past, takeup, submissions and subsequent evaluation and action from appraisals had been poor, so there was little enthusiasm for them.

Data baseline

Eleanor Mackenzie, Learning and Development Officer, started looking at how many people had previously had an appraisal. The Church of Scotland has around 220 staff and an additional 150 people in its extended workforce. She was shocked to discover that most people had never had an appraisal, and some had been employed for 20 years! This was her data baseline.

Creating the programme

In 2020, Eleanor put the previously paper-based appraisal form online and also created some ‘just-in-time’ videos and handouts. She paired these with eLearning modules on effective appraisals, supplied by the Charity Learning Consortium. This created a hub of resources – available on the learning management system provided by the Consortium – that people could access whenever they wanted.

Launch and support

Eleanor then organised face-to-face workshops, to introduce the resources and get managers and employees interested in a new approach. She knew that she needed help to reinvigorate appraisals, so she teamed up with the Chief Officer at The Church of Scotland. He opened the workshops, welcomed delegates and explained how important the appraisals process was. On the few occasions that he couldn’t attend, his empowering welcome message was delivered by video instead. This gave Eleanor both the support of The Church of Scotland and the authority to show delegates the best way to perform an appraisal meeting.

Objectives

Eleanor’s two main objectives were to:

Increase the number of people participating in appraisals
Improve the quality of submissions

Did she succeed? Did her training intervention manage to change people’s attitudes, and encourage them to complete the appraisal process – and complete it well? To assess this she monitored some key numbers, like attendance and submissions.

Identifying resistance

At first people were reluctant to participate and made excuses to avoid the in-person workshops, even though the Chief Officer was attending to champion the process. Eleanor quickly realised that lack of engagement could damage the programme’s success. Monitoring attendance meant that Eleanor managed to catch this problem before it derailed the whole project.

Overcoming challenges

In response to this resistance, she offered the workshops in an alternative, online format. She also targeted senior managers to get their buy-in. This ‘top down’ approach ultimately led to appraisals becoming mandatory in 2022. Together, this increased attendance to 100%.

Eleanor also split her audience into managers and non-managers. As a result, she noticed that conversations in the sessions with managers were much more open and transparent about the difficulties they were facing.

Evaluation

Prior to the workshop Eleanor had heard ad hoc feedback from department managers along the lines of: “Why should we complete the process? It’s a waste of time.” Both during and after the workshops this became: “How do I complete the process?”, “Where do I find…?”, “Can you help me submit…?”

The real mark of success was not just that every single department and team was engaged in the process, but that the information being submitted was both meaningful and actionable. The results gave Eleanor a new insight into how employees were feeling and what support and professional development they needed.

Results and impact

With better data to work with, Eleanor could spot trends and then take action.

The data revealed that twice as much Excel training was being offered than people needed. Eleanor reduced the amount being organised and reinvested that budget into programmes people wanted. This both increased attendee numbers and decreased unnecessary costs.

Importantly, the appraisals also revealed that wellbeing was the top priority for teams, which meant it was the top priority for the organisation. Eleanor worked with colleagues to develop a new wellbeing initiative and firm up the organisation’s policies, also introducing a new menopause policy.

The results of the new-style appraisals has also informed both the structure and content of a leadership and management development programme being launched in 2023, with wellbeing of staff incorporated into that.

The whole appraisals process also identified the need for follow up sessions, so regular line manager surgeries are being launched in 2023. The organisation is actively asking managers for their feedback on organisational policies too.

It’s too soon to measure the long-term impact of these changes, and it will be interesting to see the results of the next set of appraisals. But the ripple effect from a new approach has been far reaching, with some unexpected benefits.

Markers and success

Success can clearly be seen in:

The increase in the number of people who attended appraisals training
The change in the ‘water cooler’ attitudes of employees
The increase in submissions from around 5% before to 75% after the training initiative
The quality of submissions
The response by Eleanor and the organisation, resulting in more effective use of budget and learning tailored directly to need

Learning and development practitioners often shy away from evaluation, perhaps because of a lack of confidence dealing with data. As this story proves, it’s not all about numbers. You can have excellent evaluation with barely a spreadsheet in sight. Just plain number monitoring, partnering with the business and people-watching.

Lessons learned about increasing learner engagement:

1. Partner with the business

2. Have a sponsor, and use them well

3. Meet people half way

4. Split your audience

5. Sell the value

Read the full article here

About by Eleanor Mackenzie

Eleanor Mackenzie is the Learning and Development Officer at The Church of Scotland.

This article was written by Harri Savage, Online Learning Specialist at ELK Online. Harri has spent more than a decade working with organisations to help them craft their learning stories. From designing creative learning solutions, to kick-starting engagement in existing programmes, she evolves learning from a one-time hit to a long-lasting development culture.