In episode 009 of The Score Podcast, host Kathryn Baron(@TchersPet) speaks with Kylie Day and Sarah Thorneycroft, leaders in the field of design and implementation of online examinations. Kylie Day is the manager of exams and e-assessments at University of New England, in Australia, and Sarah Thorneycroft is the director of digital education at UNE.
Due to the length of our discussion, these interviews cover two episodes of “The Score” – episodes 9 and 10.
Read on for selected portions of the podcast transcript. And listen to the full episode Apple, Spotify, and The Score Website. Note: Removal of filler words and minor edits have been made for clarity.
Kathryn Baron (06:19):
And so with the flexibility that you have in testing at UNE, what’s the process? How does it work?
Kylie Day (06:28):
I think first to understand it we need to think about why we do this, I suppose the philosophy behind this crazy thing we do. And that’s that I think students decide to cheat before they ever get into the exam. And if all of the effort is focused on what happens in the exam it becomes quite harsh and stressful, really… What’s the word I’m looking for, Sarah?
Sarah Thorneycroft (06:54):
It’s like focused on policing the transaction in situ in the exam rather than focusing on the context around the student as a whole.
Kylie Day (07:06):
Yeah, so if we put our effort towards the student’s feelings and attitudes and decisions before the exam ever starts. So in the same way as a community safety program or a community health program, you would do population-wide communications to talk about the risks involved, expected behavior, alternatives to risky behavior.
Kylie Day (07:06):
… if we put our effort towards the student’s feelings and attitudes and decisions before the exam ever starts. So, in the same way as a community safety program or a community health program, you would do population-wide communications to talk about the risks involved, expected behavior, alternatives to risky behavior. In the same way that the highway patrol police are not expected to catch every single person who might speed, they have a presence and that serves a purpose to make it risky, to dissuade people from speeding.
Kylie Day (07:49):
But that’s not the only thing that one would do if you wanted to reduce say the road toll or the incidents of people breaking the road rules, you would expect to have a community safety program and narrative happening along with that. And when we catch people who might be cheating it’s not a good outcome for them, it’s not a good outcome for us as an institution.
Kylie Day (10:05):
… we see flexibility and easy flexibility as a key factor in letting students manage their own pressures in ways that allows them to succeed and not have to cheat to do that.
Sarah Thorneycroft (10:17):
That changes the cost benefit analysis.
Kylie Day (10:20):
So, we work with online exam proctoring service where our exams live in our learning management system, but we have highly skilled and trained supervisors who can… They have a view of the students’ screen. They can use software to lock down that student’s computer in ways that we ask them to, and they can also watch the student.
Kylie Day (12:01):
And that’s the first thing that our faculty said when we started having conversations about flexibility, flexibility is an F word, if I can be cheeky. Students will cheat, and so that’s when we talk about design. The assessment needs to be designed in the mode or in the context of the mode that it’s held. It should not be that we are just doing paper exams on a web page, it’s a whole second order change.
Kylie Day (12:31):
So, the design features might include using a question bank. So you would have just enough. I get a different question one to you. It’s still the same topic, same degree of difficulty. But if I say, “Hey, what did you put on question one?” That kind of collaboration will be disrupted because we get different question ones.
Sarah Thorneycroft (15:12):
This is where it’s really useful to help people make comparisons between the paper examination paradigm in which somebody is watching them, and often in more embodied ways of walking up and down and patrolling the physical room that people are located in. But we’ve also discovered, because online the proctor and student relationship is one to one, whereas in an exam hall it’s one to many. Yes, that proctor is watching because that’s the cultural condition for examinations that we’ve agreed on regardless of where they’re held.
Sarah Thorneycroft (15:49):
But the proctor can actually also provide support in situ, which can be both technical support or general encouragement. And we’ve had a lot of comments come through student evaluation that actually talk about how helpful and supportive the proctor was. So that’s one of the key reasons that we focus on human invigilation, not AI only invigilation, because of that personalized element and the ability to also provide benefits, not just stress and monitoring.
Kathryn Baron (22:57):
Do you have online practice exams to help students as well? I thought I had read that.
Kylie Day (23:05):
We do, and that’s one of our favorite things. We call it a try it out exam. And you have to book it, it’s supervised. You have to follow the rules, but it’s got questions like, Hey, did you know this is where you can see the countdown clock on your screen?
Kylie Day (23:24):
Or a question that suggests that you change the batteries in your wireless mouse or keyboard before your exam and do all your windows updates. It’s instructional around, how do I have a good time in my online exam? It has a thing on draw us a graph, which you can do, showing the correlation between the amount of caffeine that you consume compared to the amount of assignments you have due. So it’s intentionally lighthearted, but it allows a student to work out what buttons do I have to push? How does this thing work out? What does it feel like? What does it look like? What do I need to do in my own space to conform to exam conditions? And will my computer actually sustain the technical requirements and the bandwidth that I need?
Kylie Day (30:31):
What rings in my head a lot is the phrase demonstration beats explanation. So just starting with people who wanted to come and play really and making sure that went really well. Those people then become champions. You can publicize details and say, “You know what? We can talk all we like, but we tried it and this is what happened.” And having evidence to show people.
Kathryn Baron (32:44):
What are the concrete steps that these other universities can take?
Kylie Day (32:48):
One of the pieces of advice I give to people at other universities is that they should not consider it to be an IT project, nor should it be seen as a admin logistics project. That those pieces are really important, but the structure of the team I think is one of the reasons for our success in doing it.
Sarah Thorneycroft (33:10):
Yeah, I think I tend to frustrate my sector colleagues who hope that there might be a nice recipe of concrete steps and you just follow the steps and then it works, and it’s all good. And they come and talk to us and we are like, “Oh actually it’s a cultural change piece.”