ProctorU, one of the largest and oldest remote test proctoring and security providers, announced this past week that it was doing away with its AI-only proctoring service. Going forward, the company said, they’d use only trained, human test proctors – a service they always offered.
They are eliminating the proctoring process that records a student during their test, uses AI and other technologies to analyze the session, flagging or scoring it for suspicious or abnormal events and behaviors such as routinely looking off screen or talking to someone.
That’s a pretty standard start for remote proctoring. But what made this process “AI-only” was that, after the test recordings were flagged or scored, they were turned over to faculty or the test-providers. No human on the proctoring side ever reviewed it, leaving faculty or assistants on the school side to review the flags and make decisions about what was or was not going on during the test.
In ditching the AI-only process, ProctorU said that it put too much burden on teachers to review and police test sessions – a big job for which they were often unprepared and unable to manage given their already hefty commitments. In one of the press reports about the decision, the company said that it would take a professor more than nine hours to review just the flagged portions of a single one hour exam in a class of 150 students.
Further, ProctorU executives said that the AI systems used in proctoring are sensitive by design which could flag innocent activities as suspicious. This makes sense. If the objective of a first screen is to catch things for people to review, you want to catch most anything that could be something. In other words, better to over-flag than under-flag. The problem is, this creates more flags – more work when the flags are just passed along to teachers.
As a result, ProctorU and independent audits show that very few of the flagged test sessions were being reviewed at the school. An Iowa study showed 14% were being watched by teachers. ProctorU said 11% were. Few, either way.
This pathway – sensitive AI, lack of time for review and no review – made the process nearly pointless.
Worse, we know that some professors were relying on the flags, the scores of deviation, as proof of misconduct. Given what the AI does, that’s impossible. Without a human review, a determination of any kind – cheating or insignificant – is impossible.
It’s possible that simply having an AI-only system deterred some misconduct, given the possibility for detection. But with review rates so low, it’s unlikely the deterrence was much of one. For deterrents to work, someone has to be caught once in a while.
This significant marketplace decision has a few implications for instructors and administrators. Here are five things faculty and other schools should do:
Understand the Difference
When tests or assignments are done online, the choice is not whether to proctor or not to proctor. For the record, not proctoring is a very bad idea. Research shows that not protecting a test invites academic misconduct.
If a remote assessment is to be proctored, faculty and administrators should know there are three different models – the AI-only, scan and pass along option, a record, scan and review by trained company proctors option, and an exam with live human proctors during the exam. The first type is what ProctorU is dropping and the type that has caused the most controversy.
Because they’re not the same, they don’t cost the same. Responding to the ProctorU decision to use human proctors only, an executive with Respondus, a ProctorU competitor, said, “A typical university using Respondus Monitor spends less than 25 cents per proctoring session. Live, human proctoring often costs $25 per exam. The price points and services are entirely different.”
They are very different. Understand how they’re different is important.
Be Sure You Know What Kind of System Your Proctoring Company Uses and What it Means for You
The names of proctoring companies can be confusing – ProctorU, Proctorio, Respondus, Honorlock. Their product names can be even more so.
If you’re not sure what type of proctoring your chosen company is using for your exams – ask.
If the answer is the cheaper, record, AI-scanned and passed along version, understand that the burden for reviewing the recorded and flagged sessions is probably yours. The company may be making a recording and telling you what to look at, but you’re going to have to look at it and make a decision about what’s happening.
Review Your Flags
Whatever method your proctor company uses, videos of the sessions will be passed along or made available – either flagged, flagged and reviewed or flagged and reviewed in real time. You are still going to need to review those sessions and decide what to do.
The benefit of having humans review the session before they get to you is that you’ll get far fewer things to review. One estimate was that human reviewed proctored tests cut the number of reviews required by faculty by 85% – cutting nine hours of review to just one.
Either way, review is necessary. The system does not work unless a teacher or a similar authority at the school is watching, engaging students where there are incidents and elevating misconduct incidents where necessary.
Ask for Help
If reviewing flagged test sessions isn’t possible, and it may not be, tell someone. Ask for help. Whatever it costs, remote test proctoring loses nearly all of its impact if no one on the school side is reviewing the sessions. A system in which just 11% or 14% of sessions are reviewed is a broken system.
Moreover, if it’s not possible or reasonable to review the sessions, school administrators should know the burden that’s being created.
Ask for Better
People are unquestionably better at discerning attempts to be dishonest from common behavior. Computers can flag things that are different, things that may be misconduct. Only a human can decide what actually is.
If your department or school is using a system that bypasses humans, ask for one that doesn’t. Ask for it especially if you cannot or should not be reviewing hours of computer-flagged test sessions. The better options are probably more expensive. But getting it right – avoiding false accusations, freeing up teachers to teach and mentor, having two humans involved instead of none – is probably worth it.
