Part Two Assessment: Model Answer and Rubric

Two stages hold nothing at all: Plan and design, and the data stage. Those are the two stages that decide everything downstream, which is the argument week 5 makes.

Two more hold only the supplier's evidence, and the supplier's evidence describes a configuration Meridian stopped running in January 2025.

Not in the sense that would survive a question.

Meridian can honestly say this: on 4,000 motor and home claims notified between July 2024 and June 2025, the model reached 81.0 per cent accuracy, 73.3 per cent precision and 78.1 per cent recall, measured in November 2025 at the live threshold.

Two sentences in the report's own test set description take most of the weight out of that.

The test set was not held back before the model was built. It was assembled in August 2025 from claims that had already settled. And the model has been evaluated against this set twice.

A test set loses its value the moment it stops being unseen.

Neither, until you know three things.

A system that recommended nothing at all would be right 62.2 per cent of the time. So the honest comparison is 81.0 against 62.2, not 81.0 against zero. It is a real gain and a smaller one than it looks.

Of 1,610 claims the model sent for settlement without inspection, 430 should not have gone. That is the number that describes what actually happens to people.

Different population, different test set, threshold 34. The validation report states this plainly and says it does not reconcile the two. The 94 per cent is nonetheless the figure that reached the Technology and Change Committee.

What it tells you. Claimants estimated as Black or Black British were recommended for fast-track settlement at 73% of the rate of the highest group, which is below the four-fifths convention.

The report's own analyst concluded the gap sits in the model rather than in the claims, because the base rates are close, 35.0% against 38.7%. That is a real finding, made by Meridian's own analyst, in a final report.

What it does not tell you. Why. And four things limit how far you can take it.

The group variable is estimated from surname and postcode, not declared. The report says misclassification would move the figures in a direction nobody can determine.

The image quality hypothesis was tested on a manual sample of 22 claims, and could not be tested properly because image capture metadata is kept for 90 days and the metadata for the test period no longer exists.

The effect of vehicle age could not be separated from the group estimate.

The strongest finding in this section is not the ratio. It is the sentence after the hypothesis: The follow-up work was agreed with Claims Operations and has not been scheduled.

Yes, and it is the single most consequential thing in the pack.

The threshold moved from 34 to 41
on 22 January 2025, approved in an email whose subject line was a question about volume, on the stated basis that the leakage numbers were within tolerance. No impact assessment. No committee record. Never reviewed since.

The threshold is the dial that decides how many people have their claim settled without anybody looking at it.

Moving it up sends more claims down the automated path, which changes who the system acts on and who carries the cost of its errors. The model did not change. The system did.

Three things follow.

The supplier's performance evidence is measured at 34 and therefore does not describe what Meridian runs. The coverage grid says so, in its own accuracy row, in a cell nobody has acted on.

The 2023 go live approval was an approval of a different configuration.

A blank row is not an absence of risk. It is an absence of evidence, and the two get read the same way by everybody who is not paying attention.

Reliability asks whether the system gives the same answer to the same claim twice. Nobody knows. The note against the grid says the supplier was asked on 19 November and did not answer.

That is not a blank at all, it is an open question with a date, and it should be recorded as one, with an owner and a review date, rather than as an empty cell.

The Safety row is worse and quieter. It is not blank. It was closed: "Agreed with Claims Ops that this property is not applicable to a claims triage system. No further work."

Deciding a property is out of scope is a governance decision, and this one was taken by the function that owns the system, in a conversation, with no rationale recorded.

Interpretability is blank too, with a note that the system produces no per-claim explanation, which is a finding rather than a gap in the paperwork.

A model answer at 148 words.

What makes this answer work is that it does not say the record is wrong.

It says the record is not precise enough, names the path it is talking about, and cites the sentence in Meridian's own report that proves the point.

A model at 247 words.

It opens with what was read, so she knows it is not an opinion. It calls the threshold change reasonable, because it was.

It puts the hardest finding second, where it is least likely to be read as an accusation. It separates "we cannot describe this" from "this is harming people", which is the distinction that keeps the conversation alive.

And it asks for forty-five minutes and three specific actions rather than a mandate.

It leads with the 0.73. It uses the word bias.

It implies she approved something she should not have. It asks her to pause the fast-track path.

Score yourself on each criterion. Be strict.