The independent review of last year’s Bar Professional Training Course (BPTC) has found that the three-way relationship between the BSB, the eight law school providers of the BPTC, and the electronic testing company Pearson VUE, which delivered the online exam, “ultimately contributed to the complexities in the booking and delivery of the examinations.”
The Director General of the Bar Standards Board (BSB) promised a “lessons learned review” after last summer’s exams descended into chaos, with disabled students poorly catered to, problems with the online exam system and some students complaining they were forced to urinate in bottles because they were not allowed to leave their screen.
An independent review, conducted by Professor Rebecca Huxley-Binns of the University of Hull and Dr Sarabajaya Kumar of UCL, has now concluded.
They report that the three-way nature of the relationship between the BSB, Pearson VUE and the law schools led a system that was not joined up. “Once the examinations started, and technical and other difficulties came to light, providers were in an exceptionally difficult situation,” they concluded. “The BSB directed candidate complaints to Pearson VUE. Pearson VUE sometimes redirected candidates to the BSB. In frustration bordering on exasperation, candidates turned to their Providers, who were at a loss for solutions to help their students and had no choice but to redirect their enquiries back to the BSB or Pearson VUE. Providers had no direct means of contacting Pearson VUE themselves to make representations on behalf of their students… This is illustrative of the unfortunate piecemeal basis on which the project operated.”
Particular criticism was levelled at Pearson VUE, for providing “utterly baffling” data that was incomprehensible to anyone outside its own organisation.
As frustration mounted during the exams period, the BSB released a statement saying that students had been warned in advance that they should use the toilet before starting the test, and that “89 per cent of the exams were delivered without any reported incident and 97 per cent of exams were successfully completed.” However, this statement only provoked more anger, both from students and members of the barristers’ profession.
In the review, the authors noted that the BSB has tried to report accurately on the success rate of the exams, but that Pearson VUE’s response had been “utterly baffling”.
It noted: “In response to an apparently simple question from the BSB, “of the 828 people due to complete exams on Tuesday and Wednesday how many actually did so?”. The Pearson VUE response was not at all simple: ‘The exams scheduled to be taken on Tuesday and Wednesday = 830 – 23 candidates ( all OP ) our records are showing without a result being received so 807 exams were taken, the 23 affected exams happened during yesterday. The 98 cases raised so far relates to OP & test centre deliveries and this number will increase and decrease as the events continue – should these case figures be used to calculate the stat’s for delivered exams or success rates, as some of these cases can be for informational purposes, its the 23 candidates that couldn’t start or complete their exams that require investigation.’
The report continues: “The BSB quickly escalated this response to managers in Pearson VUE. They were at a loss because the figures and explanation were utterly baffling. The BSB took very seriously its responsibility to report to candidates on the success rate of the exams, however this was exceptionally difficult, as Pearson VUE records completed exams whether incidents occurred or not. They reported only the number of examinations that did not start at all, or stopped partway through.”
After a long exchange of emails, the BSB, with the explicit support of senior leaders of Pearson VUE, published the statement about the success rate of the exams that angered so many students.
The report goes on to say that “the statistics produced by Pearson VUE are evidently very useful internally for the company to identify trends including technical issues with candidates and technical issues with the platform. However, they do not easily translate into accessible data for the client. It is therefore impossible for us to conclude with confidence how many of the examinations were successfully completed without incident.”
It concludes: “Pearson VUE has not, throughout our meetings with them, acknowledged the volume of problems that have been reported to us… In our opinion, Pearson VUE have not understood the scale of the problems encountered by candidates.”
On the problems of students encountered with the Pearson VUE system, the report says: “When we raised these concerns with Pearson VUE, their response was one of surprise and denial. They simply had no idea this had happened, or how. Of more concern to us was their apparent lack of interest in finding out what could have caused the problems. Our impression of Pearson VUE has been that they were not prepared to accept or even acknowledge the usefulness of the critical feedback that candidates were providing.”
The BSB does not escape criticism. The report concluded that the decisions to take responsibility for the August examinations, and to contract with Pearson VUE to provide the online examinations, were correct ones in the difficult circumstances of the pandemic.
However, it said that two key decisions made at the outset – to stick with the closed book format for the exam methodology, and not to permit unsupervised breaks – were critical mistakes and that “some of the complexities in the delivery of the examinations were a direct result of these two decisions.”
“It was not foreseen that providing no breaks would have the impact that it ultimately had,” the report states. “The BSB clearly expected candidates who could not sit in front of a computer screen for up to three hours to book their examination at a test centre. One of the reasons that the decision not to permit breaks ultimately caused such issues, including candidates taking to social media reporting they had to urinate into bottles, is because the BSB could not foresee the delays many candidates would experience from logging onto the system to being able to start the examination.”
“Candidates fed back to us that some proctors permitted natural movement, but other proctors interrupted them, disrupting the examination. Word seemed to spread among the candidate community to keep eye contact with the screen at all times, or risk having the assessment terminated. Candidates became anxious to restrict their natural bodily movements; we are not designed to sit still in front of a computer for hours at a time. The lack of breaks compounded the situation.”
The report also criticises the BSB for not being flexible enough for students with disabilities, noting: “Providers worked closely with these candidates to put forward suggestions to the BSB about how to accommodate the candidates’ reasonable adjustment requirements. However, these suggestions were rejected by the BSB. The basis on which the BSB was making decisions was unclear both to the Providers and the candidates. The candidates felt it was because the proposals were coming from them.”
“An important focus for the BSB was the integrity of the examinations and ‘control of the paper’. It was reasonable, therefore, to consider how to make reasonable adjustments consistent with the rigour of the closed book examination format. However, it is not clear to us why accommodating disabled candidates necessarily poses a risk to the integrity of the assessments. It is not an ‘either/or’ situation. Concerns about ‘downgrading the examinations’ by giving disabled people reasonable adjustments is a false dichotomy. They merely want a level playing field and the opportunity to take their exams like everybody else.”
“The BSB’s general approach to the handling of reasonable adjustment needs fell short, notwithstanding the unprecedented circumstances of a global pandemic.”
Responding to the report, Matthew Hill, CEO of the Legal Service Board, said: “The report finds that there was a disproportionate negative impact on students who needed a reasonable adjustment, particularly those with disabilities, students with limited financial resources and international students. A strong and diverse legal profession is one of the statutory objectives that the BSB is responsible for promoting and is an essential component of public confidence in the sector. It is clear that the events and decisions covered in this report combined to undermine that objective.
“It is right that the BSB has apologised and published an action plan showing how it will address the report’s recommendations. The BSB and the wider sector must learn from this to ensure we dismantle barriers not create them. As the oversight regulator of legal services, we will continue to monitor the BSB’s progress through our regulatory performance framework.”