Credit Risk Model Validation: Catching Drift Without a Data Ticket

A credit risk model doesn’t fail loudly. It fails a few basis points at a time — default rates creeping above what the model predicted, one segment quietly outperforming its risk grade while another underperforms it. By the time it shows up in a monthly dashboard, the question every risk manager has to answer is the same one their validators and regulators will ask right back: is the model still reliable, and how do you know?

The key distinction is between model performance and model ranking. A model can still rank borrowers reasonably well — higher-risk grades defaulting more often than lower-risk grades — while its predicted probabilities are systematically too low. That’s a calibration problem, and it can point to changing portfolio conditions rather than a fundamentally broken model.

Answering that well usually means pulling loan-level performance data, comparing expected default rates against actual outcomes across cohorts, checking whether risk grades still separate good from bad accounts, and investigating whether the population or a particular segment has changed. In most shops, that means a ticket to the data team and a few days’ wait — right when the question is time-sensitive.

A Monday morning that starts with a bad number

A credit risk analyst at a consumer lender opens the monthly portfolio review and sees it: defaults on personal loans are running hot. Her model — in production for a year now — was built to estimate the probability that each account will default, but the question in front of her isn’t “what does the model say,” it’s “can I still trust the model?”

Normally that question turns into a ticket: pull loan-level outcomes, join them against model scores, compute expected-versus-actual by segment, wait for the data team to get to it. Instead, she opens Savvina and starts asking.

The five queries below are the short version. The full investigation — every question, every result, in the order they actually came up — is recorded in the Savvina credit risk demo.

“How has our credit risk model’s default rate performed over the last 12 months, compared to what it predicted?”

Actual 5.23% against an expected 4.39% — running about 0.8 points hot overall. In credit-risk terms, the model is underpredicting observed defaults. Not alarming on its own, but enough to keep going.
“Break that down by risk grade — is any particular grade running hotter than it should?”

Nothing dramatic: grades A through F are all within a couple of points of their benchmark, with the riskiest grades running only slightly hot (E at +2.14 pts, F at +2.62 pts).

This matters because the model’s discriminatory power still appears directionally intact: riskier grades are still behaving as riskier grades. The problem looks more like calibration than a complete failure of the model’s ranking ability.
“If it’s not a specific grade, is the deterioration concentrated in a particular product or origination channel?”

There it is: broker-originated personal loans are running at 10.89% actual against a 4.27% expected — a 6.6-point gap, more than double any other product/channel combination in the book.

The signal has now changed from “the model is running hot” to “something specific is happening in broker-originated personal loans.”
“For broker-channel personal loans, group those into two vintages — before November 2024, and November 2024 onward — and compare actual against expected for each.”

Loans originated before November 2024: 7.53% actual against 4.55% expected. From November 2024 on: 12.08% actual against 4.18% expected.

The deterioration is therefore concentrated in newer originations. That’s important because a vintage effect can indicate a change in underwriting, customer mix, broker behaviour, economic conditions, or another process change — not necessarily a defect in the model itself.
“The current model (v3.0) went into production in March 2025. Did switching models cause this, or does the problem predate the retrain?”

It predates it. Under the old model, v2.3, broker personal loans ran at 12.74% actual against 4.37% expected. Under the new model, v3.0, the same segment runs at 8.22% against 4.14% — better than v2.3, not worse.

The retrain isn’t what broke this; if anything, it is partly correcting for the problem. The evidence points instead to a change in the broker-channel portfolio that began before the model switch.

Five questions, no SQL, no ticket. Broker-channel personal loans carry $5.3M in live exposure and have already produced $362K in realized losses. At a 55% loss-given-default (LGD) assumption — the proportion of an exposure expected to be lost after a default — the post-November 2024 cohort — $5.0M of that exposure — carries an estimated $217K of additional expected loss beyond what the model predicted for it. That last number carries an assumption worth stating out loud: it applies a gap measured on loans with a full 12-month performance window to the cohort’s whole exposure, newer originations included. It is a projection, not a booked loss — the kind of distinction a validator will expect to see made explicitly.

That gives the risk team something actionable: not simply “the model is off,” but where the deterioration is happening, when it started, and what financial exposure it represents.

So, should they replace the model?

Not yet.

The evidence does not point to a broken model. Risk grades still broadly rank the portfolio correctly, and the newer model actually performs better than its predecessor on the affected segment.

The appropriate next step is to investigate what changed in the broker underwriting process around November 2024, validate the recent vintages against the model’s assumptions, and consider recalibration if the new risk level proves persistent. The segment should also be placed under closer monitoring until its performance stabilises.

That distinction matters: Savvina provides the evidence for the decision; it doesn’t make the decision. The risk team remains responsible for deciding whether to recalibrate, remediate the channel, or ultimately replace the model.

Why that query trail matters as much as the answer

In April 2026, US banking regulators replaced SR 11-7 with SR 26-2, issued jointly by the Federal Reserve, the OCC and the FDIC — and the new guidance is considerably more explicit about what “effective challenge” has to look like in practice. It’s not enough to show that a review happened; examiners expect a documented trail of what was compared, what the comparison revealed, and what the bank did in response — legible enough that an internal auditor or an examiner can follow it without someone there to narrate. OSFI’s Guideline E-23 asks for the same kind of demonstrable rationale north of the border.

A sequence of natural-language questions and their answers can form that trail — reproducible, timestamped, and legible to someone who wasn’t in the room when the analyst first noticed the number looked wrong. Compare that to a summary slide with a “not caused by model switch” bullet and no query behind it: correct, maybe, but not something an examiner can retrace.

Built for data that can’t leave the building

Loan-level performance data is exactly the kind of dataset compliance teams are cautious about handing to an AI tool. Savvina runs entirely inside your own infrastructure — nothing is sent to an external server. Queries are read-only against your database, so there’s no risk of a validation exercise accidentally modifying production data. Column-level exclusion keeps sensitive fields (SSNs, account numbers, PII) out of what the LLM ever sees, and audit logging records what was asked and what was accessed — metadata only, never the values.

That combination — plain-English access to your own risk data, on your own infrastructure, with a record of every question — is what turns “let me check with the data team” into “let me check right now.”

Common Questions

Does Savvina decide whether a credit risk model should be replaced? No. It produces the evidence — expected versus actual by risk grade, product, channel, vintage and model version — and the risk team decides what to do with it. Recalibration, channel remediation and model replacement are all judgement calls that stay with the people accountable for them.

Is a trail of natural-language queries enough documentation on its own? It is the evidence layer, not the whole validation file. What it gives you is the part that usually goes missing: a reproducible, timestamped record of what was compared, what the comparison showed, and in what order the analyst learned it. The memo, the sign-off and the remediation plan still get written.

Does loan-level performance data leave our infrastructure? No. Savvina is self-hosted and runs inside your own network perimeter. Query results are never sent to an external server, and if you point it at a local model, no part of the exchange leaves your infrastructure at all.

Can a validation exercise accidentally modify production data? No. Queries are read-only against your database. A validation exercise cannot write, update or delete, so running one against a production replica carries no risk of changing the book it is measuring.

How are SSNs, account numbers and other PII kept out of the model? Column-level exclusion keeps sensitive fields out of what the LLM ever sees. Excluded columns are absent from the semantic model, so they cannot be referenced in generated SQL or returned in a result.

What does the audit log actually record? Metadata only — what was asked, which tables and columns were accessed, and when. It never records the values that came back, so the log itself does not become a second copy of the data it is describing.

See it in action

Watch a full model validation investigation, start to finish, in the Savvina credit risk demo — or talk to our team about a production evaluation on your own infrastructure.

References

Board of Governors of the Federal Reserve System, Office of the Comptroller of the Currency and Federal Deposit Insurance Corporation, Supervisory Letter SR 26-2: Revised Guidance on Model Risk Management (April 17, 2026) — federalreserve.gov (full guidance, PDF)
Board of Governors of the Federal Reserve System, Supervisory Letter SR 11-7: Guidance on Model Risk Management (April 4, 2011; superseded by SR 26-2) — federalreserve.gov
Office of the Superintendent of Financial Institutions, Guideline E-23 – Model Risk Management (effective May 1, 2027) — osfi-bsif.gc.ca

The community edition is free, self-hosted, and BSL 1.1 licensed, converting to Apache 2.0 in 2030. Pull it and try it on your own data today.