SRA could drop skills test from super-exam Part One after first pilot

The Solicitors Regulation Authority and Kaplan have finished their first pilot of Part 1 of the new Solicitors Qualification Exam (SQE) and are seeking to make changes based on their findings.

The SQE, dubbed the ‘super-exam’, is the culmination of years of work into reform of legal education which began with the Legal Education and Training Review, published in 2013.

The SRA first put forward its proposed shake-up to the legal education system in December 2015, with the concept of a two-part qualification process central to its thinking: SQE Part 1 and SQE Part 2. Part 1 would test candidates’ ability to use and apply legal knowledge though multiple-choice exams, while Part 2 would be taken at the point of qualification and test legal skills.

However, a small skills test was incorporated into Part 1, after feedback from law firms concerned that candidates could arrive in the office having passed Part 1 but without the minimum standards of legal writing and research skills that would allow them to work into a non-qualified capacity.

Kaplan was given the contract to set the exam this time last year. Its pilot of Part 1 has now been completed, with 316 active participants taking the test exam papers across 44 different test centres in the UK, as well as in France and Singapore.

Kaplan and the SRA deemed the pilot a success, but concluded that the skills test included in Part 1 was not fit for purpose. “The assessment design that we had modelled didn’t work: it didn’t meet minimum standards to be sufficiently precise at the pass/fail borderline,” said Julie Brannan, who is director of education and training at the SRA and the key figure behind the exam.

The SRA will now “take some time” to engage with stakeholders about whether to redesign the skills assessment in Part 1, but Kaplan has recommended the current version is dropped.

“For reasons of the robustness and resilience of the exam as well as reliability, accuracy, validity, fairness and equality of opportunity, a Stage 1 skills exam in its current form should not be part of the SQE,” it said in its final report.

In particular, analysis suggested that BAME participants in the trial were at a disadvantage on the skills part of the exam compared to white participants.

Kaplan also had difficulty defining the standard needed to pass the Part 1 skills test. “The passing standard was to be a ‘threshold skill level to enable candidates to work effectively in professional legal services in an unqualified capacity’,” it said. “This, however, raised considerable conceptual difficulty as to exactly what the standard was. An ‘unqualified capacity’ can have a wide variety of meanings.”

The other major change Kaplan is recommending relates to the multiple-choice exams testing functioning legal knowledge (FLK). It now is recommends there should be two 180-question exams, instead of three 120-question ones. Once again, Brannan said this would give greater precision at the pass/fail borderline.

The SRA had initially envisioned 680 questions split across six exams, but reduced the number to 360 last year, after a technical review of the SQE design conducted with Kaplan.

In the wake of the 2019 pilot the SRA has concluded that the multiple-choice questions, in which students choose the single best answer, “can appropriately differentiate between candidates”. Marks on the FLK ranged from 17 per cent to 85 per cent, with an average mark of 50 per cent. When the exam goes live for real, there will be no uniform pass mark: it will vary slightly from paper to paper depending on their exact level of difficulty.

The SRA’s chief executive, Paul Philip, said: “This pilot brings us a step closer to delivering a world class assessment. It provides confidence that the core part of SQE1 is appropriately rigorous, while helping us to improve it further.

“The pilot was also about understanding what does not work, and there is clearly more to do to establish whether an early stage skills assessment can be sufficiently robust. We will continue to work closely with stakeholders to explore the best options for assessing legal skills through the SQE.”

Peter Houillon, the CEO of professional education and assessment at Kaplan, added: “An exam leading to qualification as a solicitor must meet the very highest standards to both protect the public and be fair to candidates. A key step in this process was this pilot and the analysis of the data it has provided. The evidence suggests that an assessment of the application of the functioning legal knowledge taking the form of two papers of 180 questions each will reach those standards.”

A pilot of SQE Part 2 is scheduled to take place in December 2019, with the exam finally going live for real in autumn 2021.