In episode 010 of The Score Podcast, host Kathryn Baron(@TchersPet) speaks with Kylie Day and Sarah Thorneycroft, leaders in the field of design and implementation of online examinations. Kylie Day is the manager of exams and e-assessments at University of New England, in Australia, and Sarah Thorneycroft is the director of digital education at UNE.
Due to the length of our discussion, these interviews cover two episodes of “The Score” – episodes 9 and 10.
Read on for selected portions of the podcast transcript. And listen to the full episode Apple, Spotify, and The Score Website. Note: Removal of filler words and minor edits have been made for clarity.
Kathryn Baron (01:46):
Thank you for that. Last time, we explained the challenges that students face that led you to create the flexible testing program at the University of New England. Now, let’s talk about how it works. The tests are run through an exam center team. My first question is then how does that work exactly and is it just at UNE or is this the process that’s used across the country at universities and colleges?
Sarah Thorneycroft (02:14):
So this is one of my favorite things to talk about in terms of organizational structure. So the notion of having an exam center doesn’t really apply in the online space. We don’t have an exam center, but in terms of how we lead the online examinations program, historically paper based exams were treated as an administrative or logistics function. But to us, it’s very obvious that exams are an assessment function. It’s a teaching and learning practice. So early last year we went through workplace change and we brought Kylie’s team as the exams and the assessment team. We have experienced both in previously doing paper based exams and now leading online examinations into the digital education portfolio, which is the portfolio I lead that also has learning designers, all of the learning environments teams. So we have all of the virtual learning environments platforms under our umbrella, learning media, all of their teaching and learning functions are co-located.
And so that lets us work really closely with exams as an assessment practice, not as an active administration, which really lets us hook into the design aspect and the cultural aspect, working with staff to empower them, rather than just seeing it as an administrative act, where you just schedule things. And that’s really important. And it’s not something we see a lot in terms of organizational structure. And you said that that’s definitely not common in the US context, but I think it’s really important to see exams as an assessment practice and not as an administrative function.
Kylie Day (03:58):
… we do have a central team and that’s been a feature at Australian universities for a long time. But what we’ve seen at other universities in Australia lately is that’s being distributed back out to academic areas. And I think I would say that’s a loss because I think it requires professional expertise to run what is probably the largest event a university will hold, high stress, high stakes, high numbers of people, really, really quite important.
And to pull that expertise in terms of how do I wrangle 10,000 people without making them cry, to be a little bit cynical, but that’s a skill. How do I communicate with people to achieve compliance with lots of different rules? How do I get people to actually do what they need to do so that everything coincides nicely for everyone and everyone has a good experience and how do I manage academic integrity issues well? I think distributing that out to academics who already have plenty to do it might not be their area of expertise, but to outsource that to them as well. I think you lose something there.
Kylie Day (07:43):
COVID helped us because we were at about 25% online exams before COVID, in the before times. And then we had a very rapid shift to 100% of all exams had to be held online with a 24-hour window in the online proctoring. So that really helped tear the bandaid off. And I think it helped people just take that step that they might not have been keen on doing. What we, my team put a lot of effort into was to make it really safe for them and massive amounts of support for students and for staff, so that nothing was too hard and that nothing went badly. And that’s why we put effort into being on call till 1:00 AM so that there were no stories from students about how they were just left at midnight with no one to help them. And I think that really helped. And when we did have people who wanted to be a bit innovative, we went out of our way to support that.
And so those then became the stories, the good examples that we could say, Hey, your colleague tried this and here are the metrics where we can see that student success increased. Students are happier. Students have more agency over all the demands on themselves. So they’re much more settled and more engaged. And just supporting that in a really safe way with a lot of support. The whole flexibility piece did take a lot of time for people to get their heads around. And I think that exams exist as a cultural archetype, that they’re hard, they’re tricky, they’re secret, they’re tough. You have to turn up or else, all this stuff that people have embedded in their brains about exams. Helping people realize that the way exams have been managed in the past is not necessarily the way exams should be managed and really calling into question every assumption that people have consciously or unconsciously about assessment and exams and flexibility and students. So it really has been a long change piece.
Sarah Thorneycroft (10:45):
Access too is key for students that don’t have to engage in geographical travel to get to locations. That can sometimes be a real barrier for our demographic. So being able to access online in your own home makes a real difference for a lot of students.
Kylie Day (11:02):
We had a student early on who actually rang crying tears of happiness and no one rings, right, to say what a wonderful exam they’ve just had, right? It’s a occupational hazard in our line of work that you only ever hear from people who have a bad time, but this student rang early on in the project when she really realized that the flexibility that she could have. She rang, crying tears of happiness to thank us to say that she had a spinal injury, which meant she was in chronic pain. Traveling was really hard and would make her really unwell with pain. And that she asked for a comfortable chair, but our idea of comfortable chair was not the same as her idea. And we couldn’t provide what she needed in the exam venue. When she realized that she could do, she had three exams in two days and she physically was not going to be able to do that at an exam center, which meant that she wasn’t going to finish her degree, which meant that she wasn’t going to be able to get that job that she had lined up, this dream job.
Once she realized that she could actually choose the timing of her three exams and sit one on a weekend, sitting in her lounge chair, which was much better for her and lay down if she needed to, she realized that she could access those exams. She could finish her degree. She was going to get that dream job that she’d lined up and that moved her to tears and probably moved us to tears a bit too when she rang to tell us that. So exams are an institutional barrier. Traditional exams are an institutional barrier to accessibility.
Kylie Day (13:13):
Certainly easier to get those metrics in an online assessment mode rather than paper. From my perspective, we do a survey after every exam period to say, how was it? Which bits were good, which bits were bad? Why did you like it? Why didn’t you like it? What impact did it have? And we also get various other pieces of feedback. And what we know is that students really appreciate being able to choose a time that suits them. They don’t like having to sit in an exam hall with 300 other people, sniffling and tapping and wobbling their desks. They don’t like having to travel, but I think Sarah can speak on the kind of metrics that you could get that would influence design.
Sarah Thorneycroft (13:58):
So in terms of designing our approach, getting metrics around when students choose to have their exams is really useful, because you can actually see the uptake of flexibility and understand when you make this available to students, how are they making use of it? And thus, to what extent you want to make sure you’re designing your assessments to maximize that capacity. And some of the other metrics, I know that some of the ones that we use a lot are around things like the test taker experience. So this isn’t necessarily about the design of assessment. A lot of the most effective actions you can take for assessment design are the things that don’t look like assessment design. Metrics around the test taker experience in terms of satisfaction, technical issues, academic integrity issues, the incidents of actual confirmed breaches and that kind of thing.
When you’re talking about an academic or you probably use the term professor, who’s talking to a student who had a bad experience in the exam, that’s really easy to understand as oh actually online exams are bad, but understanding that out of 10,000 exams, 85 to 90% of students are having a really positive experience
Sarah Thorneycroft (18:28):
The intangible costs are an important part of the conversation. In terms of dollars for instance, it’s reasonably more expensive than our learning management system, just as an example. But the key thing is because human individualization, human proctoring is a key part of our strategy. It’s not a platform cost it’s people, it’s people cost. So I think it’s important to contextualize that way is that it’s not a really expensive piece of technology. It’s actually a part of a whole ecosystem and it’s paying for the human experience.