{"id":742,"date":"2012-06-13T16:32:28","date_gmt":"2012-06-13T16:32:28","guid":{"rendered":"http:\/\/www.johncanning.net\/?p=742"},"modified":"2012-06-13T16:32:28","modified_gmt":"2012-06-13T16:32:28","slug":"why-the-national-student-surveys-shortcomings-make-it-so-useful","status":"publish","type":"post","link":"https:\/\/johncanning.net\/wp\/?p=742","title":{"rendered":"Why the National Student Survey's shortcomings make it so useful."},"content":{"rendered":"

This blog post is based on a short talk I did at the NUS-HEA Student Engagement Conference<\/a> in Nottingham on 12\u00a0June\u00a02012. I was one of four panelists talking about feedback from\u00a0students. \u00a0I was asked to give a\u00a0sceptical\u00a0\u00a0perspective\u00a0on the National Student Survey.<\/p>\n

Last year I wrote a short article on how lecturers could make use of the NSS as a tool for improving the quality of teaching and learning. I came up with 5 main tips.<\/p>\n

These were:<\/p>\n

    \n
  1. Don\u2019t chase the ratings<\/li>\n
  2. Don\u2019t blame other people<\/li>\n
  3. Look at the free-text comments (people often forget this section of the NSS exists).<\/li>\n
  4. Compare with your institution\u2019s own data<\/li>\n
  5. Talk to your students<\/li>\n<\/ol>\n

    I put a positive spin on the NSS in that article. However, I may as well have said \u201cNumber 6: Ignore the National Student survey\u201d.<\/p>\n

    I have been asked to provide a sceptical viewpoint on the NSS. Here I will give three reasons to be sceptical.<\/p>\n

    1. Questions are ambiguous<\/strong><\/p>\n

    Consider Question 19: This course has enabled me to present myself with confidence.<\/p>\n

    What does this question mean? These are some of the ideas staff and students came up with in a project I worked on last year<\/a>.<\/p>\n\n\n\n\n
    What this question might mean<\/strong><\/td>\nPossible assumptions<\/strong><\/td>\nOther issues<\/strong><\/td>\n<\/tr>\n
    EmployabilityDoing oral presentations<\/p>\n

    Feeling confident in person<\/p>\n

    Interviewing skills<\/p>\n

    Self-belief<\/p>\n

    Able to express opinions without fear.<\/p>\n

    Able to challenge the opinions of others.<\/p>\n

    Not anxious<\/p>\n

    Students can stand up for themselves<\/p>\n

    Students are confident they will get a good job.<\/td>\n

    		Students were unable to present themselves with confidence at the beginning the course.Confidence comes from going the course.<\/p>\n

    Presenting oneself with confidence is a good thing (some students might benefit from being less confident)<\/p>\n

    A course which does not help students present themselves with confidence is not a good course.<\/p>\n

    The student who answers this question in negative might have been better off doing a different course or studying at a different place.<\/td>\n

    		Confidence might come from sources other than the course e.g. student societies, increased age, work experience, time spent abroadDoes a negative answer to this question suggest that the course was in any way inadequate?<\/p>\n

    Some evidence of students thinking about L2 language confidence, but this question was for students of all disciplines.<\/p>\n

    Students who answer this in the negative are saying something bad about themselves.<\/p>\n

    Student anxiety or lack of confidence indicates poor teaching or course design.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    2. There is no significant difference between institutions at subject level<\/strong><\/p>\n

    Measures of overall satisfaction will be included in the Key Information Set which is about to be launched. The KIS is meant to help students with choosing a university. However the NSS does not really help students in differentiating between institutions. In 2010 41 institutions returned scores for French. In response to question 22: \u201cOverall, I am satisfied\u00a0with the quality of the\u00a0course\u201d<\/em>, the highest score was 100% agreement and the lowest was 70%. Quite a big difference? Yes, but when you look at the confidence limits you see that the 70% institution could have been as high as 89%. The real \u2018score\u2019 for the institution with 100% could have been as low as 83%. So statistically speaking the institution which came last might actually have a higher real score than the institution which came first!<\/p>\n

    \"French<\/a>
    French overall satisfaction with confidence intervals (NSS, 2010)<\/figcaption><\/figure>\n

     <\/p>\n

     <\/p>\n

    I think that the using the NSS to compare institutions is like to trying to identify the best cyclist in one of those cycling races where the whole peloton crosses the finishing line together. One cyclist will be first across the line, but is he or she the best? In today\u2019s stage of the Tour de France it will be one person. Next time this happens it will be someone else. Oh, and they all get given the same time anyway, so it really doesn\u2019t make any difference.<\/p>\n

    3.\u00a0No use for quality enhancement<\/strong><\/p>\n

    Those with an eye for NSS history will know that the survey was inspired by the Australian Course Experience Questionnaire (CEQ)<\/a>. The CEQ was designed a performance indicator for assessing the quality of teaching in higher education. It is not very useful in improving the student learning experience and I am not entirely sure whether it was meant to be. It can identify departments or institutions which have performance problems, but it does not contain any clues as how teaching might be improved.<\/p>\n

    Conclusion : why the NSS's weakness help\u00a0improve\u00a0the\u00a0student\u00a0learning\u00a0experience\u00a0<\/strong><\/p>\n

    So in conclusion the NSS is ambiguous, of little help to potential students, and little use in improving the student learning experience. Ironically though the NSS is actually quite useful for quality enhancement\u2014not because it is useful in itself, but because lecturers spend a lot of time thinking and talking about how to respond to the NSS. \u00a0Most of the time they are saying what a bad survey it is. That\u2019s what makes the NSS successful. It gets people talking and about teaching and learning. And because they talk and think they make improvements. Perhaps the NSS isn\u2019t so bad after all.<\/p>\n\n\n