I’ve been thinking a lot about student engagement since my trip to Nottingham a couple of weeks ago. The theme of that conference was ‘student engagement’ and my task was to speak about the National Student Survey. During the course of the discussion a student opined that student engagement is often seen as synonymous with doing surveys of students. And as I often hear students are getting all surveyed out.
The news that the Bill and Melinda Gates Foundation has given over $1 million to Clemson University to research “‘Galvanic’ bracelets that measure student engagement” is responded to with incredulity by Valerie Strauss in her blog for the Washington Post. Clemson University’s website describes the project thus:
Purpose: to conduct a pilot study to measure student engagement physiologically with Galvanic Skin Response (GSR) bracelets, which will determine the feasibility and utility of using such devices more broadly to help students and teachers.
According to Wikipedia (a source I only use for find out things I don’t know anything about), Galvanisitc Skin Response (GSR) is also known as skin conductance:
Skin conductance, also known as galvanic skin response (GSR), electrodermal response (EDR), psychogalvanic reflex (PGR), skin conductance response (SCR) or skin conductance level (SCL), is a method of measuring the electrical conductance of the skin, which varies with its moisture level. This is of interest because the sweat glands are controlled by the sympathetic nervous system,so skin conductance is used as an indication of psychological or physiological arousal. There has been a long history of electrodermal activity research, most of it dealing with spontaneous fluctuations or reactions to stimuli.
Whilst I share Strauss’s scepticism (despite my total ignorance of this field of study), this project brings in another dimension to debates about measuring student engagement. Is there a ‘Brave New World’ in which teacher evaluation instruments will be replaced with student sweat analysis? Will the perception of scientific objectivity appeal to policy makers?
Times Higher Education last week reported on a talk at the UCU annual congress stating that the National Student Survey is being used to evaluate the performance of individual teachers in some universities. Needless to say, but I'll say it anyway, it is not the purpose of the NSS to evaluate individual teachers.
Richardson , J., Slater J. and Wilson, J. (2007) The National Student Survey: development, findings and implications. Studies in Higher Education 32.5, pp. 557-580 :
This paper from 2007 reports on the piloting of the UK National Survey (NSS). The settled version of the questionnaire contains 22 questions, but the pilot contained 45 items. Interesting the pilot contained negative questions (where agreeing indicated a negative experience). Questions on workload were dropped due to a lack of internal consistency.
To do well on this course you mainly need a good memory
It was clear what standard was required in assessed work.
It was clear what I was required to attend, prepare, and to do throughout the course.
I found the overall workload too heavy
I feel confident in the subject knowledge I acquired
I had as much contact with staff as I needed
Overall I feel the course was a good investment
I would recommend the course to a friend
I wonder whether these would have been preferable to the "Overall, I am satisfied with my the quality of my course questions which is tends to take preference over all others.
The second pilot contained the question “It has been difficult to answer many of the questions because of the variability of my experience (Interesting only 12% strongly agreed and 18.2% agreed).
Just thought it would interesting to share the questions which did not make it.
The report from the LLAS Subject Centre National Student Survey project last academic year is now online. In the project we focused in on eight of the 22 questions. Whilst many of the questions were found to be problematic, this one was especially difficult to unpack.
Question 19: The course has helped me to present myself with confidence.
From the report
When answering this question, many students initially thought about giving oral presentations. It was also linked to employability and interviewing skills, but the question of whether this was about personal confidence or academic confidence was unclear. And where students reported an increase in confidence, was this down to the skills their course had given them, their year abroad, their work placements, or was it just part of being four years older?
One member of staff observed that the NSS is carried out at a time where students are at their most anxious, perhaps looking for work, perhaps worried about the future. In languages it was suggested that this question might be thought about in the context of L2 competence or confidence in dealing with people from other cultures. ―It’s a bit of a weird question said one student. ―It really wants you to say “yes”, because if you say “no”, you‘re saying something bad about yourself.
Some further thoughts here. Some a little pedantic maybe, but that’s what happens when you start to unpack the question with students and lecturers.
|What this question might mean
Doing oral presentations
Feeling confident in person
Able to express opinions without fear.
Able to challenge the opinions of others.
Students can stand up for themselves
Students are confident they will get a good job.
|Students were unable to present themselves with confidence at the beginning the course.
Confidence comes from going the course.
Presenting oneself with confidence is a good thing (some students might benefit from being less confident)
A course which does not help students present themselves with confidence is not a good course.
The student who answers this question in negative might have been better off doing a different course or studying at a different place.
|Confidence might come from sources other than the course e.g. student societies, increased age, work experience, time spent abroad
Does a negative answer to this question suggest that the course was in any way inadequate?
Some evidence of students thinking about L2 language confidence, but this question was for students of all disciplines.
Students who answer this in the negative are saying something bad about themselves.
Student anxiety or lack of confidence indicates poor teaching or course design.
I am currently organising a workshop for languages, linguistics and area studies lecturers on making the most of National Student Survey (NSS). The workshop will include reports by three colleagues who have been funded by LLAS to carry out research into how students and staff understand the NSS questions. What does the statement, “overall, I am satisfied with the quality of my course” really mean? The NSS has not been without its critics, but its use in league tables and websites providing information to potential students makes it impossible to ignore.
I don’t think there is much to be gained from chasing the ratings. A couple of percentage points here and there can make a big difference to a department’s position in a league table, and the potential for wasted time, money and effort in making superficial changes in the vague hope of squeezing out an extra 3% satisfaction is a real possibility. In my view the real opportunity the NSS provides is to open dialogue between students and their teachers.
The workshop will be held in London on 22 June 2011. You can register online at the LLAS website.
I have written a short article about using the National Student Survey (NSS) for the latest edition of the LLAS magazine Liaison.
I am also overseeing some projects on the NSS. I have developed an A0 dialogue sheet which can be downloaded from my humbox page.