Considerations when using virtual proctoring
Virtual proctoring disproportionately impacts students from an equity perspective. For example, students of colour or lower socioeconomic status may have difficulties obtaining the technology required to participate in virtually proctored tests and exams (Gonzales, Calarco, & Lynch, 2018). Students may not have access to high speed, reliable internet, their own laptop/computer, or a webcam, which can disadvantage them in their access to post-secondary education. These tools also disproportionately impact students who are parents, have eldercare or other family responsibilities, or students who cannot find a quiet space or time to take a test or exam.
References
Gonzales, A. L., McCrory Calarco, J., & Lynch, T. (2020). Technology Problems and Student Achievement Gaps: A Validation and Extension of the Technology Maintenance Construct. (external link) Communication Research, 47(5), 750–770.
Students may encounter a variety of technological issues while taking a virtually proctored test or exam. For example, their internet connectivity could lower in speed which could lead to poor video quality or students could also lose internet connection altogether, ending their exam. We recommend that instructors have plans in place to manage technological issues so as to not disadvantage students in their attempts to take a test or exam using virtual proctoring tools.
Despite the measures that virtual proctoring platforms use to ensure the integrity of a test or exam, these tools cannot prevent all types of academic misconduct. For example, these tools would not be able to identify if students have a second computer, tablet, or cell phone present, which could allow them to engage in chat rooms or search the internet for answers. In addition, there are websites that students can visit to learn how to ‘beat’ and ‘defeat’ virtual proctoring platforms. (external link) As the likelihood of obtaining tangible evidence to support a suspicion of academic misconduct is low, these platforms act as more of a deterrent than an effective tool to identify misconduct. Collectively, these limitations to virtual proctoring platforms limit the true effectiveness of monitoring students and ensuring the integrity of the exam.
The transition to remote learning is met with accessibility concerns, with virtual proctoring contributing to many of these (google doc) g concerns. (external link) Students registered with Academic Accommodation Support may have access to test accommodations that are difficult to assess through virtual proctoring such as increased washroom breaks, noise-cancelling headphones, memory aids, and scrap paper.
In addition, virtual proctoring software may pose accessibility and usability issues for students who use assistive technology like text-to-speech, dictation or screen reading software. Test settings will need to be customized uniquely depending on the students' accommodation or assistive technology. Depending on the accommodation, this may entail: allowing specific programs or cloud based software, allowing keyboard shortcuts, adjusting time limits, and more.
To ensure the virtual proctoring system works for the exam, additional preparation needs to be made in advance of the exam. Since a virtual proctoring system requires a separate application, both students and instructors will need to go through a set of steps to install and enable the virtual proctoring system, followed by an onboarding process. The onboarding process is recommended to be done at least one week before the exam date, allowing for time to troubleshoot in case there are any issues. Likewise, on the day of the exam, students will need to be ready with the system setup roughly 15 minutes before the exam start time.
Please see the (google doc) Online Examination Checklist (external link) prepared by the Centre for Excellence in Learning and Teaching for a step-by-step guide for preparing to administer an examination.
Instructors still have to decide whether or not a flag is considered academic misconduct. The number of flags in a generated AI-based proctoring system depends on many factors, including what constitutes a violation behaviour. Typically the automated system will have a list of behaviours they provide the AI to flag.
Recommendations
- Given the challenges listed above, consider alternatives to virtual proctoring.
- Provide students with the opportunity to take a sample test to ensure that they are ready when the exam time starts.
- Ensure students have their official OneCard as ID to show to the camera during onboarding. Should a student not have a OneCard, government issued ID can be displayed to the camera, showing only the picture and name (all other information can be covered by the student).
- Due to increased anxiety and potential challenges with internet stability, it is strongly recommended that the weighting of any final examination be reduced.
- Ensure you have specified the timezone when providing the date/time of the test to ensure students in other timezones are present at the correct time.
Virtual proctoring drawbacks
Virtual proctoring systems are basically surveillance systems and they all work very similarly. The software will take control of the student’s computer in terms of disabling all other applications from running except those which are needed to do the examination. The software records what the webcam captures of the student in front of it and the surroundings. With all the setup and configurations in a virtual proctoring system, there are still possibilities students can use their smartphones, tablets, or another computer, or having notes available where the webcam cannot capture activity. At the end, a virtual proctoring system is an attempt to replicate in-class invigilated tests which cannot be replicated in an online environment and act as a deterrent to academic misconduct.
Therefore, automated virtual proctoring becomes the more reasonable option. But it does not come without any drawbacks. In addition to what has been mentioned above about virtual proctoring systems in general, automated virtual proctoring systems use artificial intelligence to provide instructors with ‘flags’ of behaviours captured by the webcam which may be considered academic misconduct. Instructors will have to review these flags to confirm what the flagged behaviours represent and follow through accordingly.This requires a lot of time and attention to go through.
Live virtual proctoring, i.e. having a person watch the student as they take the exam is an additional level of proctoring available through some systems. Toronto Metropolitan University is supporting automated virtual proctoring systems. The cost for live virtual proctoring is quite high and stressful for the student as they are being watched by a stranger.
Extra prep and reviewing time for instructors.
Extra stress and anxiety for students.
Seems good in theory, but not a perfect solution.
Zoom is not recommended for virtual proctoring
Zoom should not be used for virtual invigilation for the following reasons:
- Zoom is not designed or built for virtual proctoring. Toronto Metropolitan University has a virtual proctoring tool for this purpose.
- Zoom itself is not effective in deterring or detecting issues surrounding academic integrity.
- Only recordings made within the virtual proctoring tools can be used as evidence in Policy 60: Academic Integrity matters.
- Zoom is not private: students can see and take screenshots of other students and their backgrounds during exams. If microphones are enabled, there may be disruptions during the evaluation. Writing an exam can be stressful to students, and these additional considerations can exacerbate the stress that students experience.
- The use of this technology may raise equity and access issues for students. Zoom is not conducive to the identity verification function of invigilation. Students should not be able to see each other’s ID.
D2L Quiz has various question types (), including a file upload within the Written Response question type. However, note that to grade them, instructors will have to download each uploaded file from every question individually.
Workflow:
- Create the D2L quiz: Instructor creates the exam questions in the D2L quiz and sets the availability date/time and duration. Note about the quiz availability and enforced time limit and grace period. It’s important to understand that D2L Brightspace CANNOT force a quiz to end at a particular time. What can be done is setting the Exceeded Time Limit Behaviour. The options for this setting are a) Allow the student to continue working; b) Prevent the student from making further changes; or c) Allow the student to continue working but automatically score the attempt as zero after an extended deadline.
- Students take the D2L quiz: Students can click on the “Start Quiz” button anytime within the quiz availability and work towards completing the quiz within the enforced time limit (provided duration). Students have visual cues as they move along the quiz whether their responses have been saved by D2L. Students can manually click the Save button as well. When done, students click the “Submit” button. Depending on how the instructor sets up the Exceeded Time Limit Behaviour, students may or may not be able to still submit the quiz past the enforced time limit. The instructor will see the time stamp in the student quiz attempt’s view. Students will also see a timer displayed at the top of the quiz page.
- Reviewing and grading the quiz: With D2L, it is possible to set up quizzes that grade automatically, but there are cases when you may need to manually grade a student's quiz attempt (i.e. for question types written response and short answer).
- Special Access: Instructors can configure special access to allow individual students and students requiring accommodation with different rules for the completion of the exam.
Essay-style exams can be delivered using D2L Assignment. The exam questions can be presented on screen in the Assignment tool or D2L Brightspace or be provided in a Word document or PDF that students can download from the Assignment tool. The essay can be automatically checked for its originality by enabling Turnitin.
Workflow:
- Create the D2L Assignment: Instructor creates the exam questions in the D2L Assignment. The question can be typed in directly in the D2L Assignment tool, or by uploading a Word document or PDF.
- The start, end, and due date and time to be configured according to the exam time. The start date marks the date and time students can begin submitting files to the folder. The due date marks the exact time the assignment is due. If an end date is set, students will no longer be allowed to submit a file to the assignment folder. If there is no end date, or the end date is later than the due date, students can still submit an assignment past the due date but it is marked as late. All assignments are time-stamped.
- In addition, originality detection can be enabled for D2L Assignment, which will use Turnitin.
- Students work on the D2L assignment: Within the start and due date/times of the D2L Assignment, students can access the exam questions and be able to submit the assignment.
- Reviewing and grading the exam in D2L Assignment: Instructors (TAs, and others with marking privileges) mark the exam in the Assignment tool.
- Special Access: Students who need accommodations can be given Special Access by modifying the date/time for these students which are different from the rest of the class.
Most science and engineering exams will require students to provide responses in handwritten format. Below are workflows for handwritten exams delivered through D2L Brightspace and Crowdmark.
D2L Assignment
- Create the D2L Assignment: Instructor creates the exam questions in the D2L Assignment.
- The question can be typed in directly in the D2L Assignment tool, or by uploading a Word document or PDF.
- The start, end, and due date and time to be configured according to the exam time. The start date marks the date and time students can begin submitting files to the folder. The due date marks the exact time the assignment is due. If an end date is set, students will no longer be allowed to submit a file to the assignment folder. If there is no end date, or the end date is later than the due date, students can still submit an assignment past the due date but it is marked as late. All assignments are time-stamped.
- Students work on the D2L assignment using pen and paper: Within the start and due date/times of the D2L Assignment, students can access the exam questions. Students write their exam responses on paper. Students can take photos of their responses and save them into one PDF. This document ( (google doc) Saving written exam responses on paper into one PDF (external link) ) was used in one Architecture course for the Winter 2020 final exam, where students took photos of every page of their exam responses, embedded the photos to a Google Doc, then saved the Google Doc as PDF. With the one file, students then submit it to the Brightspace assignment.
- Reviewing and grading the exam in D2L Assignment: Instructors (TAs, and others with marking privileges) mark the exam in the Assignment tool.
- Special Access: Students who need accommodations can be given Special Access by modifying the date/time for these students which are different from the rest of the class.
Crowdmark
Learn more about Crowdmark in D2L Brightspace.
- Create the Crowdmark ‘Assigned’ Assessment (external link) : Instructor creates the exam questions in Crowdmark using their remote exam feature (external link) . Instructor schedules the availability of the assessment. At the start time of the assessment, emails will be sent to students containing a link to get to the exam. This link can only be opened within the exam’s availability window.
- Students work on the exam using pen and paper: Upon receiving the exam email (external link) , students click on the exam link in the email. Students need to authenticate using their Toronto Metropolitan University credentials before they can access the exam questions. Students then write their responses using pen and paper. Once completed, students take photos of each page of their responses, embed the photos into Word or Google Doc, save the file as a PDF, and submit it to Crowdmark.
- Full instructions on completing and submitting an assignment. (external link)
- Reviewing and grading the exam in Crowdmark (external link) : Instructors (TAs, and others with marking privileges) mark the exam in Crowdmark.
- Special Access: Students who need accommodations can be given Special Access.
There may be more than one way to deliver an exam with mixed question types, e.g. multiple choice and handwritten. One way to do this is by having the exam in two parts. For example, part one of the exam is the multiple choice questions part, delivered using the D2L Brightspace Quiz. The second part of the exam is the handwritten component. This part can be delivered using the Brightspace Assignment tool or Crowdmark. In terms of timing, it’s best to have some overlap time between the first and second parts, in case some students finish early, or others are running slower on one part. Using the release condition feature in Brightspace, it can be configured that students need to submit the first part first before they can open the second part of the exam.