At midday on the 23rd March 2021, an historic and uniquely odd remembrance event took place in the UK as the country stopped for a minute of silence – not to commemorate a long-ago war, or an ancient patron saint, but to reflect on the year that had just happened. At Agulhas it provided the urge to sum up and reflect on the lessons that this difficult year has taught us: how we have adjusted to the various lockdowns and restrictions that COVID-19 has placed on us and how we have coped with the complexities of conducting high-quality evaluations during a global pandemic.
This is our third blog on the topic of Evaluation under COVID. In our first blog, written early on in the pandemic, we proposed a five-step process for conducting successful remote evaluations. In our second blog at the end of last year we reflected on the ethical challenges that remote evaluation and data gathering presents. For this our third blog, we conducted a survey among Agulhas staff and fellows and collected insights and observations gained from a year of conducting remote evaluations, picking out some key lessons and questions for consideration as we look ahead, to working under a “COVID-normal” future.
Lessons from the last year
Although they require planning and flexibility, and access to basic communications technology, remote evaluation can offer a range of benefits including reducing costs and carbon emissions and increasing levels of participation by different partners and stakeholders. They offer the opportunity for greater geographical cover, and reaching out to a broader audience within the same virtual ‘trip’. As one respondent to our survey stated: “remote fieldwork might give us access to somebody by phone for 30 mins that would have declined a more formal in-person meeting”. Conducting virtual visits also eliminates the time spent travelling between research sites, enabling trips to take place within a shorter period. However, this sense of flexibility might lead to over-compressed timeframes, and there might have been a tendency to assume that it would be feasible to have back-to-back “country visits” without taking into account the administrative burden of these visits. Over the past year we have discovered that although flexibility is an asset, it is important to keep to a structured and time-limited visit to avoid over-running, and constantly extending the visit period as more virtual interviews are added.
Over the past year we have relied increasingly on local partners to carry out research in partnership. The move to remote evaluation has encouraged us to collaborate more with national partners and involve them in the evaluation design – something we have always promoted as an organisation. In that sense, COVID-19 has sped up the localisation agenda[1] that we had been working towards. As one respondent put it, the experience of evaluating remotely due to COVID-19 restrictions probably means that “more use of research collaboration with local partners is also clearly in the cards and is good both from a localisation agenda, and value for money point of view”. These new partnerships have thrown up challenges, with one respondent stating that it does raise the issue “of providing adequate oversight and Quality Assurance remotely” and another found that there can be “trade-offs and contingent factors”, particularly around the challenge of finding local researchers with the required experience and skill sets. This challenge, however, varies considerably from country to country, and within countries, between capitals and harder-to-reach regions.
Questions for the future
While some interviewees may be comfortable and open with a virtual evaluation team or interviewers, the success of a remote evaluation may depend on the nature of the review and the type of data we seek to collect. The majority of the reviews that Agulhas has conducted over the past 12 months have not focused on interviewing vulnerable groups, for example, and so some of the more difficult challenges relating to consent, privacy or safeguarding were not tested in these specific cases. For projects where virtual evidence gathering included contact with vulnerable groups, detailed ethical protocols were put in place and interviews and focus groups were designed to ensure participants would not be put at risk. As a result, there is a clear risk that the robustness and accuracy of findings may be reduced when gathered remotely on sensitive topics such as gender-based violence. As one survey respondent put it, “The depth of the analysis suffers a bit, and we’re more limited in exploring sensitive issues, due to the additional challenges of creating a safe environment for respondents”. How, for example, do teams ensure that information sharing is truly secure, that engaging with interviewees does not put them at risk, or that the data collected has not been influenced by individuals present at the time of the call? The opportunities for gathering insights outside the interview room, or from the group, rather than the interviewee can also be lost; “the opportunity for direct observation and other forms of ethnography is curtailed – people may feel uncomfortable with you sitting in on a virtual meeting or training session, or it may just not be possible”.
Based on our experiences of the past 12 months, what can we say about whether virtual evaluations lead to improved equity and equal access, or, on the contrary, whether it leaves some voices out? Most of our reviews over the past year have involved talking to interviewees that had access to technology. Would offering “virtual visits only” run the risk of alienating certain groups and individuals? Video calls and interviews conducted remotely (by phone or online) can create a sense of distance and, depending on the circumstances or target audience, might not be conducive to creating the necessary climate of trust for effective, objective and transparent evaluation work. Finally, and most importantly, a great number of people we would most like to reach (e.g., in fragile and conflict-affected settings), often do not have access to the reliable technology needed to facilitate a successful remote consultation. This can lead to their inputs being negatively affected or, their being left out completely. For instance, one of our survey respondents noted that many of the community members they aimed to consult for one project “could not be reached by phone and so the consultations were limited, and even for those who could be reached by phone, some of the discussions were not as in-depth as face to face is”.
Bearing in mind the 50/50 split between those who thought that virtual data collection could be as effective as in-person, 95% of survey respondents believed that the experience of working through the COVID-19 pandemic will result in an increase in remote/virtual data collection in the future. This definitive response points to the conclusion that COVID-19 has likely raised the bar in terms of value for money, environmental considerations and how we approach collaboration. In short, it poses the question: do we really need to go? The past year has changed the way we approach research methods we are familiar with, and presented new challenges. It has removed the requirement, and possibly the urge, to travel. In particular, it has changed the expectation of in-person research and data gathering into a consideration of whether it is genuinely needed. And if it is needed, to what extent it can be done through collaboration with national partners rather than international travel. Our organisation and our partners have demonstrated that it is possible to conduct successful remote evaluations, reaching a similar number of people, to achieve comparable results. While, at the same time, reducing our carbon footprint and making efficiency savings that can be directed elsewhere – towards further research, or a broader scope.
As we move into living with a “COVID-normal” future, perhaps, as one of our survey respondents put it, a “good outcome would be if there is more consideration in the future of what can be done well virtually and what would need in-person field work, and to only do the latter when quality and effectiveness dictates”.
[1] https://had-int.org/blog/ingos-and-the-localisation-agenda/
