From January to April each year the attention of UK universities turns to the National Student Survey (NSS). Across the entire HE sector final year students are urged to complete the 28 item questionnaire about their experiences of their undergraduate courses. Their responses feed into various metrics which in turn form the basis for an array of increasingly influential league tables. Moving upwards in those tables is an important measure of apparent success. Outcomes are often the KPIs (Key Performance Indicators) of TWMs (Those Who Matter). So universities inevitably find themselves asking, ‘how do we improve our NSS scores?’ And that, of course, is the wrong question.
Asking the right question
Before we get onto what the right question might be, let’s first take a look at another wrong one. Universities are legitimately concerned about the number of first class degrees that are currently being awarded. But if they ask, ‘how do we decrease the number of first class degrees we award?’ they will, again, be posing the wrong question. Degree outcomes are symptoms of assessment systems and resultant practices. And if you treat symptoms, you won’t come up with sustainable cures. The correct question in this context is, ‘how can we make our assessment infrastructures and practices better?’ In other words, if you want to tackle grade inflation…don’t aim to tackle grade inflation. Go in deep, and the cure is more likely to last.
To illustrate how this can work let’s look at the recent values-driven approach to enhancing the assessment infrastructure at Nottingham Trent University (NTU). We took a forensic look at our structures and practices, and on that basis made various principled decisions. For example, we moved away from the arguably unfair and inflationary 0-100 scale that is widely used in educational assessment, and replaced it with a fairer, non-inflationary 0-16 scale. As a consequence of this and other decisions about core aspects of assessment practice, NTU ended the 2019 academic year with a drop (of 7 percentage points) in the number of firsts that were awarded. It turns out that enhancing fairness in the deep structure of an assessment system has many good and sustainable consequences, one of them being an appropriate profile of final awards.
Just as degree outcomes are a function of an assessment system, so NSS scores are a function of (among other things) the quality of teaching. Therefore, if you want to improve your NSS scores, aim to improve the teaching (or better still, the learning) that goes on. This might sound rather obvious. But metrics, which are a symptom of a thing, have a habit of getting used as a shorthand for talking about that thing. And before you know it they have, to all intents and purposes, become the thing itself. Ask an undergraduate psychology student about intelligence, for example. Chances are they’ll tell you about IQ.
The value of values
We have a rule, in our large NTU Psychology department; that rule says, ‘never chase the metrics’. It’s one of our core values. It’s a promise we make to colleagues that we will never do anything that is driven by the agenda of improving our NSS scores. Everything we decide to do is driven by the agenda of improving the deal we give our students and the deal we give each other. In fact the whole operation of teaching psychology at NTU is driven by a set of core values, such as these. We find that if you work at this deeper level, by developing a values-base – not a vacuous set of ‘v a l u e s’ that appear on every corporate mission statement, but a real set by which you live and work – there are benefits to all, further downstream. Including good NSS scores.
The thing about values is that, once you get buy-in, they can help carry teams of people along on even relatively complex paths. Academics can be a bolshie lot. If I stand in front of 100 lecturers and say, "we want to get better NSS scores, so we are going to do x and y", I guarantee that I'll have lost over half of them to dark chunterings about 'bloody league tables' before I've even started to say what x and y might be. If, on the other hand, I lead off with, "we want to get better at some of the things we do in our teaching, so we are going to...”, I at least have their attention. Because the other thing about academics is that they are seriously invested in their students' learning. And they want to do their teaching as well as possible. Some outside the HE sector may be sceptical about that, but in my experience it's generally the case.
The value of metrics
Incidentally, all of this is not an attack on the NSS itself. It seems entirely reasonable to ask final year students, ‘overall, are you satisfied with the quality of your course?’ And the league tables that are put together on the basis of these and other metrics are one source of information that can support students in making informed choices about where they study. I have heard these tables described as 'toxic'. I contest that notion. My career in HE started at a time when this institution was known as Trent Poly, and the only thing that students had to go on in deciding where to study was the non-empirical 'metric' of 'reputation'. That never played out particularly well for us in those days. And that was entirely unfair. Nottingham Trent always has been a great place to study, and the foundations of the University’s current success lie deep in the roots of our history as a polytechnic. So, to me, in this regard half decent metrics beat 'reputation' hands down, every time. Let’s just not let them become the thing itself.
Andrew Grayson, Associate Professor School of Social Sciences, Nottingham Trent University, @AssessmentStuff
The Advance HE Surveys and Insights Conference 2020 took place on 29 April at Sheffield Hallam University and was a great opportunity to discuss and debate strategies to increase student response rates and engagement.
The conference provided an opportunity to discuss and debate the potential of insights from surveys, metrics, qualitative research and wider methods of capturing the student voice. Through keynote speakers, papers, posters and workshops, the Surveys & Insights Conference 2020 explored real examples of how innovative approaches to measurement and feedback have been employed to deliver impactful findings which have driven real change. #SurveyConf20