This year I have completed a three year evaluation of a community organising project. It was an interesting journey where I took a developmental evaluation approach with systems thinking.

Part of my considerations was to create effective conditions for the evaluation to be successful. My role was not as a first order, objective, dispassionate observer. I was an interconnected, embedded empathic member of the project team. I was the friendly challenger and critical friend. I was keen to demonstrate that I had as much care and passion for the project as the people living in the area in which the project was being carried out.

Being embedded within the evaluand

Being embedded with the evaluand was not an easy task. All of my interactions were online due to my geographical distance from the project. This meant I could not physically feel the atmosphere or walk around talking to people for myself. I was also unable to reach the wider community.

The challenges of the Covid 19 situation

Due to the constraints of the Covid 19 lockdown, I maintained contact with the evaluand via telephone, email and computer platforms such as Zoom. I attended meetings remotely and was accepted by the evaluand as a member of the team. Meetings attended on Zoom were particularly useful, as I was able to observe the dynamics in the room as well as hear the conversations.

Some of the remote ways of connecting with the evaluand were not successful. For example, I tried to run a focus group but only had one attendee. I also set up some creative and interactive exercises to draw out different perspective from the group on a Miro electronic whiteboard. I was not, however, able to get any input into this.

Working with the evaluand

During the evaluation, I was acutely aware of the pressures on the individuals involved in the project, especially around family commitments and work, so engaging without causing any additional stress was something I was particularly mindful of. What I found worked quite well were very short one-to-one phone calls or Zoom meetings. I found I could work these around the schedules of the evaluand and the interactions were not too taxing or imposing on their day. It also helped to maintain regular contact – little and often.

Providing reassurance to the evaluand

It was key to give reassurance about the positive elements of the project to the evaluand. They were doing some excellent work on forming relationships, working at multiple levels of recursion and building their own systemic sensibilities. I was able to reflect back scenes demonstrative of trust developing between members of the evaluand during meetings as stories and feelings were shared. I was also able to reflect back my observations of inner confidence growing as community leaders diligently led campaigns. There was also a role in offering a different perspective and/ or a different framing to the situation, bringing another dynamic into our collective consciousness for consideration.

Building trust with the evaluand

A condition for effective evaluation was trust. Trust between different members of the evaluand and trust between myself, as the evaluator, and the evaluand. I had one-to-one discussions with individual members which were confidential. I only disclosed that which they were happy for me to share.

Encouraging reflective conversations that enabled learning

The nature of the reflective conversations was that they were done with respect, rather than hard challenge.

Challenging my own evaluation practices – personal frames of reference and traditions of understanding

Personally, I bring several frames of reference into my evaluation that I need to be mindful of. They are that of a systems practitioner, a system changer and of my own experiences of creating the conditions for change and witnessing what I feel works and what does not. I also bring the frame of reference of a project and programme manager, a public service manager and an educator. All of these frames are involved in my perceptions of the project. I purposefully reflected on these and their potential impacts throughout the project, as I assisted the evaluand as they created value in the project.

Projection, perception and bias

With the above in mind, I was as careful as I could be of not projecting my feelings from other work I was involved in onto this project. I was consciously aware of my traditions of understanding and my frames of reference. I was trying hard not to be falsely positively biased. However, I acknowledge that I will have unconscious biases and areas of unknown that will impact on my evaluation practices.

Embodying STiP

It was imperative that as an evaluator, I worked with authenticity and integrity. I intended the evaluand to experience me as a person, not as a label as an evaluator. I put in significant effort to communicate in a way that worked for the people involved. I rarely, if ever, mentioned that I worked for a University. At times it was important to let the evaluand know that I was not employed by the commissioner of the evaluation but I was working on behalf of the project team, to help them as they enabled their project

Were the evaluand helping to shape the evaluation?

This was difficult due to the distance between us and there was the potential for more creative interaction. The evaluand were, however, shaping the evaluation because they are people and changeable and therefore, I flexed and blended with how they were moving and changing. They change, I change. I change, they change.

What was the impact of the evaluator being embedded in the project?

This is something only the evaluand can answer. Personally, I felt that the first couple of years of the project were more successful. In the final year, it felt like other pressures, such as the risk of losing funding before the project had achieved its goal, were at the forefront of people’s minds.

Did I enjoy the experience?

I enjoyed meeting and being involved with the evaluand. I liked their passion and their determination to move forwards. They achieved some great things. By the end of three years, it was definitely time for the evaluator to step back out and let them carry on with their excellent work.

The evaluation

You can find the evaluation on the community organising web page: http://www.citizensmk.org.uk/campaigns/fishermead-citizens-alliance/#:~:text=Born%20of%20the%20existing%20Citizens,Fishermead%20residents%2C%20not%20outside%20experts.