Research during a pandemic: Evaluating interventions in the early years with nursery and school disruption

Blog posts

19 Jan 2021

Anneka Dawson, Principal Research Fellow 

In January 2019, I wrote about the importance of pre-16 education here at IES before any of us could have predicted the turmoil that faced us all in 2020 and continues into 2021.  Since then, IES has been involved in eight different projects focusing on ways to support practitioners and teachers working in nurseries and schools, to develop their skills and to improve parental engagement in order to improve children’s outcomes at this crucial stage.

IES is evaluating five ongoing interventions in the early years where delivery has fallen during the Covid-19 pandemic and we have been able to continue all of these evaluations although some changes have been necessary which I will discuss here. The five projects are varied, including two large scale Randomised Controlled Trials  (RCTs) for the Education Endowment Foundation, Tips By Text in collaboration with NIESR and Flexible Phonics in Reception year in schools, as well as smaller projects working with early years charities directly. One is looking at parental engagement and communication and language for the OVO Foundation in collaboration with Sutton Trust and Professor Kathy Sylva (University of Oxford), another for The Mercers Foundation looking at literacy in Nursery and Reception classes and CECIL looking at Communication and language supported by Local Authority Speech and Language therapy teams (funded by the Esmée Fairbairn Foundation, The Sutton Trust and The Lindsell Foundation in collaboration with University of Oxford). The latter three all provide a critical friend approach and help support these organisations with theory of change development, internal evaluation and sustainability going forward, as well as carrying out implementation and process evaluations.

There are several ways that IES and our delivery team partners have adapted to the pandemic research context as follows:

1:  For the CECIL project (Coaching Early Conversations in Language), we knew before the Local Authority teams started delivering the intervention in September 2020 that this was likely to be another disrupted year for nurseries, particularly as the nurseries involved in this project are private and voluntary nurseries (PVIs) in areas of high deprivation (Hackney and Nottinghamshire). Nurseries like those taking part in this project were already experiencing significant challenges before the pandemic in respect to funding and are more likely to have high staff turnover. We also discussed that PVIs are more at risk of closure during the pandemic in our blog last year explaining the funding crisis in nurseries. However, we have found that nurseries in these areas were still very keen to take part in the research, with highly committed and motivated staff. 

Although some of the original pilot work was unable to take place in its original format, the LA teams were able to carry out feasibility work with local nurseries with our support, to work out what would still be possible for the project to achieve this academic year. For example, training/support was moved to a virtual platform and the timing of some of the aspects of delivery was changed, allowing more time at the start of the year and longer gaps have been given between sessions. In respect to the evaluation, we have moved theory of change workshops to a virtual setting, allowing greater numbers of the delivery and evaluation teams to attend as they take less time with no travel needed. However, long virtual sessions can be very draining and therefore allowing sufficient breaks, keeping to a tight agenda and reducing the time of the session slightly has helped make these as useful as possible.

2:  For the large scale RCT projects in schools, we have one-on-one testing of Reception aged children (4-5 years old) on a literacy measure by an external assessor (Qa Research) built into the start and end of both projects. Tips By Text had started in the 19/20 school year and therefore there was no impact on the initial testing, but the endpoint testing which was supposed to happen in June/ July 2020 had to be postponed to start in the latter part of the autumn term - and has now been paused once more due to the increased school disruption that is ongoing. For the Flexible Phonics project, we have been able to complete the initial testing period in the autumn term and managed to test more schools than initially planned as so many schools were keen to take part in the research, a great feat in the pandemic context. The testing in both projects was adapted for schools that were not allowing external visitors to come into schools and Qa Research carried out some of the assessments remotely using virtual sessions with the children with a teaching assistant present. This was very successful as a way of continuing to include those schools in the research but required a few pilots to ensure that it worked well with the young age of the children.

3:  In our project for OVO Foundation supporting three charities who deliver interventions to support parental engagement in the early years, a lot of the charities’ normal work had to be paused over the last ten months and large changes made to the delivery, which also meant we completely changed our approach for the majority of the evaluation work. Instead of looking directly at impact assessment of the charities on child outcomes, we carried out some qualitative research with parents and practitioners about how the pandemic had influenced their use of the charities’ resources and services and their perceptions of the impacts on themselves, as well as the children. This drastic change was important and allowed us to get some very valuable findings about what parents and practitioners found useful during this time.

4:  In all projects, the implementation and process evaluation aspects will now include virtual or telephone discussions rather than face-to-face interviews. These have some pros; for example, they can be more flexibly timed than face-to-face work which would have been in the school/nursery day (and therefore may require cover for the person being interviewed), and do not necessarily need to all be on the same day which may be more convenient for participants. However, as researchers we do notice that you lose the informal time and observations you make on visiting a nursery or school which can help put the findings in context. It can also be more difficult to book in lots of separate interviews rather than one visit and may require more chasing and rearranging. Finally, case studies requiring actual observations of lessons or child/ practitioner or teacher interactions have not been possible, so it has been necessary to try to plan to do these later in the school year or cancel them altogether.

As these projects are still on-going, we are unable to comment on what final attrition will look like for outcome testing (we have been predicting 15-20% attrition depending on how long school disruption continues), but numbers for interviews and surveys have been very positive so far. We will of course also need to factor the context into the analysis and report writing when that comes round. We will follow-up this blog in the summer to see how things have gone for the remainder of the academic year but continue to be positive that it is more crucial than ever to support nurseries and schools in finding out what works to improve children’s outcomes.

Subscribe to blog posts

Any views expressed are those of the author and not necessarily those of the Institute as a whole.