A team from Sheffield Hallam University (led by Liz Austen, with Rebecca Hodgson, Jill Dickinson, Caroline Heaton, Nathanial Pickering and two student researchers Rachel Mitchell and Shannon O’Connor) have undertaken an integrative literature review (IR) to explore student access, retention, attainment and progression, building on the previously Advance HE-funded work of Webb et al (2017).
The key research question asked was: What evidence-based policy or practice has had a demonstrable impact on student outcomes (access, retention, attainment, and progression) since 2016?
Researching in the time of Covid
Undertaking a team-based literature review in the time of Covid-19 brought new ways of working, with both challenges and opportunities. The team found that while utilising the shared files and ease of online meetings was reasonably effective for collaborative working, some of the advantages of physical shared space working on such an intensive piece of work were lost, and much harder to recreate via Zoom! The intermittent functionality of the referencing tool also added to the overall fun...but the team spirit and passion for the project kept us going. The team were also conscious of the opportunity to highlight work with potential relevance for the post-Covid ‘new normal’ in higher education – the rapid shift to online learning, and the likelihood of a far greater proportion being ‘blended’ in the future means that research areas once considered less ‘mainstream’ may be increasingly relevant.
The IR approach taken by the research team was to involve a key stakeholders group drawn from a wide range of sector bodies nationally and internationally, identified for their expertise in student outcomes. Key Review milestones were scrutinised by this ‘Student Outcomes Stakeholder Group (SOSG)’, in addition to the Advance HE Advisory Group (AAG). This added independent scrutiny, reduced bias and built ownership and ‘reach’ into the project. It was brilliant to work with such enthusiastic, informed, passionate colleagues to guide and shape our approach to this work – one key contribution was to ensure that the literature review was inclusive with a focus on the positioning of the student within it.
Evidence of demonstrable impact
A key ambition of the review was to try to include evidence of demonstrable impact beyond peer review publications. Many approaches which impact on student outcomes are shared open access across the sector in the spirit of collaboration. Sector agents and communities of practice produce publications which are not likely to appear in traditional journals, yet they are widely used. So, this project championed the importance of grey literature produced by sector organisations to produce an inclusive review.
The Standards of Evidence provided the quality assurance, rather than the place of publication. However, despite a prolonged deep dive into the literature looking for evidence-based studies with demonstrable impact on student outcomes, the team found that (as with the previous review) there is still a lack of studies with robust empirical evidence of change or causality. The evidence which rests in each area - access, retention, attainment, and progression – and associations between these areas was found to be complex. Definitions, methods, and measures of student outcomes, which guided the inclusion and exclusion of sources, at times appeared fluid and contested.
Evidence challenges notwithstanding, the literature review resulted in key findings that build on the Webb et al (2017) review, as might be expected. A preview of our key themes follows to whet appetites for the full report!
Findings and reflections
In Access, simple admissions processes and the availability of clear guidance is unsurprisingly key, along with targeted outreach activities into populations less likely to enter higher education. The availability of financial support once admitted to university also emerged – in fact, ‘non-academic personal support’, including financial aid, was a recurring theme across the topics, including some evidence that even small amounts of assistance have a positive impact. The research team felt that this could be related to the concept of ‘mattering’ (see for example Flett, 2018) – the idea that ‘the university’ cares - a sense of which became evident from several studies across all the areas reviewed.
Retention and Attainment had many crossover findings, many of which either explicitly or implicitly referenced the notions of ‘belonging’ and ‘self-efficacy’ as core concepts that impact on student experience and thus outcomes, but sometimes lost within outcome measurement. The literature indicated that such concepts can be embedded in the student experience via a range of interventions, many of which centred on the notion of developing communities of learning, both between academic staff and students, and between students.
Curriculum design and pedagogical interventions pose real opportunities to impact on student outcomes. The growing importance- perhaps even essential – nature of learner analysis, and the appropriate design of these packages (when and how they should be used) is another area of thematic crossover, and useful recommendations are made about this within the report.
On Progression, only one career-based intervention with a small sample provided evidence of the impact on employment, whilst other studies expressed their desire for future longitudinal exploration. Regarding the development of employability skills and professional competencies within the curriculum, impact is measured specific to the local context but does not provide robust evidence of their impact on employment or further study. Supporting the concerns of various previous studies, this review now asks for an exploration of why this is a continued gap in the evidence base.
What now for ‘what works’?
The review concludes with a call to action. We need to fix the ‘leaky pipeline’ of impact evidence - institutions and other funders should ensure that projects are accountable for the measurement of long-term impact beyond ‘alignment’ to strategy at the point of funding.
The sector and institutions have a responsibility to de-prioritise ‘quick wins’ and invest in the longevity of projects which are designed, implemented, and evaluated over a reasonable period of time.
Finally, practitioners and leaders alike need to consider how to embed ‘stickiness’ in policies and practices that are known to enhance student outcomes.
Much of what this review found was not new. How then, as a sector, do we ensure that we consistently and reliably do more of “what works”? A subject for another review…
Advance HE members can download the literature review here.
Dr Rebecca Hodgson is Associate Dean for Teaching and Learning at Sheffield Hallam University. She leads on learning, teaching, assessment, the student experience, and quality and standards across her College of approximately 14,000 students and 600+ academic staff, and is part of the University’s senior leaders team.
Dr Liz Austen leads institutional and sector wide research and evaluation in higher education as the Head of Evaluation and Research at Sheffield Hallam University and as an independent HE consultant. She is well known for her work on using evidence for enhancement and exploring methods of hearing student voices.