Putting 110,000 examinations online

600 examination centres 

Every year, the majority of the UoL assessment is via unseen, paper-based, written exams, taken at a location near to where our students live in the case of distance learning students. To support 50,000 students studying at a distance across 180 countries, UoL has established a network of over 600 verified UoL Examination Centres across the continents. Our exam system is regarded as the gold standard by many regulators, employers, students, governments and the wider public. How then to take the assessment system online, fast, in response to Covid-19 lockdowns while maintaining standards?

One of our largest undergraduate (UG) programmes, Laws, illustrates the nature and the scale of the challenge we are facing. The UG Laws programme plays a significant role in the development of the judiciary systems of several countries and it is recognised by Regulators in many jurisdictions. This means that our students are employable immediately after graduation. So the stakes are high, very high; for the UG Laws examination events we expected to hold in May this year, and of course much more so for those students in their final year, hoping to graduate and move into work. 

Exams are optional extras

For most of our students, the choice to enter for an exam is a personal one. Assessment and learning are separated, so that a student can study a unit of learning several times without electing to enter for the exam, thus maximising flexibility and accessibility. Consequently the fee systems for teaching and assessment are separate. The majority of students pay for the units of learning, and then pay a separate fee for the exam entry. 

Students booked their summer 2020 exam diet in January 2020. So, as we moved into lockdown, we knew that we had approximately 110,000 exams booked and paid for. This includes some 32,000 exams for 17,000 UG Laws students. We need to deliver those exams. But Examination Centres the world over are telling us that they are closed, and can not host the exams. 

Also, the UoL examination system is necessarily complex, for example to accommodate thousands of students sitting exams for the same modules, but scattered across 23 time-zones. We have always had to operate a time-zone examination systems for our larger programmes and different exam papers are used in different regions of the world, but they are quality assured as assessing the same learning outcomes for the same module. And globally we support our students, through a detailed and varied set of adjustments to examination conditions put into place for students with additional support needs. 

What to do?

We clearly need to transform assessment, and quickly, in order to reassure our learners, without compromising quality and security.

600 examination papers, for 35,000 students in 20+ time zones

We considered alternative forms of assessment when conventional examinations are not possible. For some programmes, the decision to move to other assessment formats has been made, for example extended essays. But, mindful of the value that external bodies such as in-country regulators and Professional Body Requirements place on examinations, we have concluded that a form of examination remains appropriate for the vast majority of those who would have sat their exams in May or June. So, at the time of writing, (late May 2020), we are developing plans to move around 110,000 paper-based examinations to a form on online assessment in June and July. These include 600 exam papers delivered to approximately 35,000 students in 20+ time zones, while ensuring that any solution is fair to students, complies with Quality Assurance Agency (QAA) and Office for Students (OfS) guidelines, and allows for the management of an end-to-end online process – authoring of assessments, sitting, marking, and publishing results. 

Three solutions

We are opting for a range of solutions that broadly fall into three categories:

• Online, time-limited, proctored (invigilated) examinations where this is essential, such as final year UG Laws assessment
• Online, time-limited, unseen, closed book examinations
• Online, longer-time (several days or one week), assessment of an open book nature

Basically, the student will, at a predetermined time, be granted access to their specific exam paper online either through the UoL VLE or through a Third Party assessment and proctoring system. They then complete their response to the exam and submit their answer. Special arrangements for students with additional needs is very much part of the development. The volume of the transformation, and the anxiety our students are facing are of central concern in our developments. The majority of the exams will be conducted from late May to July. Planning for our scheduled September and October exams is also underway, as we recognise that these may be similarly impacted by the pandemic. The overall challenge is to provide a technical solution that ensures the needs of key stakeholders are met, while all of us, students, UoL staff, Programme Directors, Examiners, technical teams and third party colleagues are working in lockdown from our homes. We are currently running technical tests to ensure deliverability.

A Project Plan, Project Board and key work streams were quickly developed. A daily Assessment call at 10 AM, joined by upwards of 25 colleagues, keeps us connected, and provides an open platform for issues and questions to be raised. Good communication is essential if the project is to stand any chance of succeeding. Key features of our solution are:

• Each examination is being held as an online timed assessment and will not be held in an examination centre.  
• Online timed assessments will be released at the same local time for all candidates.
• The time of each online assessment has been increased to allow for upload of completed answers.
• Any student who downloads the question paper from the VLE, will be deemed to have made an attempt at the online assessment.
• Students will not need to remain online within the VLE while they answer exam questions. They will be able to work off line and then upload answers to the VLE when ready (and when power supplies permit).
• Students will be able to type their answers in Word or handwrite them and then upload them into the VLE. Handwritten work will need to be photographed and scanned/ uploaded with a mobile phone or other device, through to the VLE.
• The standard question paper and format will be used, it will not look any different to past papers. But some adjustments to number of questions has been made in some exams. 
• No other materials are to be used or referred to during the examination, unless stated on the assessment question paper.
• For students writing more than one paper on the specified day, the duration of their assessment will be duly extended to cover the number of papers entered. For example, if a student takes two papers, it will be extended to two hours, for three papers to three hours and for four papers to four hours. 
• Every effort is being made to reassure the students, to keep as much as possible of the examination process familiar, and to anticipate likely difficulties, as well as to ensure appropriate security and to guard against academic malpractice.
• However, because of the unfamiliar process, provision has been made for students to undertake a practice assessment under the new conditions. 

The next post will go into more detail on issues, challenges and change processes and consider the findings from the practice runs.