I’ve reviewed a dozen rollbacks where insurers spent $1 million on RPA bots to automate underwriting exceptions. After 18 months, the bots processed fewer than 300 policies a month—about the same volume a single underwriter cleared in a week, and the remaining 99.8% of the budget vanished into it maintenance debt because the organization skipped one step.
Change management, not model accuracy, kills AI.
In my role as head of claims transformation at a top-25 P/C carrier, I’ve seen two flavors of failure. The first is the “lift-and-shift” trap: teams deploy an underwriting model in production, then discover the adjusters won’t trust the score because it contradicts their 15-year risk intuition. The second is the “pilot purgatory” trap: a data-science team builds something the business can’t operationalize, and the project dies when the principal investigator leaves for another role.
Both failures trace to the same root—the organization’s operating model was never redesigned to absorb the AI output. Change management isn’t the soft stuff; it is the hard wiring of roles, incentives, workflows, and governance that must shift before the model ever sees production data.
Start with the workflow, not the model
Insurers still anchor transformation on the model: “Train a transformer on X years of loss runs and magic will happen.” That is backwards. The model is the easy part; the hard part is rewiring the claim triage queue so that the 18% of low-complexity auto claims tagged by the ML model actually reach a desk adjuster within 30 minutes instead of languishing for 4.3 days in the exception pile.
What usually happens is this: a data scientist builds a binary classifier that flags “likely total loss.” The classifier achieves 92% precision and 89% recall on the validation set. The team rolls it out on a Friday afternoon, and on Monday the claim examiners start rejecting every flagged claim because the model’s definition of “total loss” differs from the state DMV salvage code. The business calls it a model failure; the root cause is a missing mapping layer between the ML output and the adjuster’s workflow.
I’ve seen the fix three times: Embed a post-processing rule set that translates the ML score into the adjuster’s terminology and required next actions.
Insert a new “ML triage” role between the IVR intake and the human queue. The role’s KPI is “% of flagged claims assigned to the correct adjuster within 30 minutes.” Re-allocate 15% of the claims adjuster’s bonus pool to reflect the accuracy of their acceptance/rejection of ML recommendations.
- Once the workflow is locked, the model can be tuned. Until then, you are optimizing the wrong function. The org chart must bend before the model can stand
- Insurance org charts are still built for the pre-digital era: pyramids optimized for human labor, not for machine-in-the-loop decisions. The moment you insert an AI artifact, you create a gap between the process owner and the data owner that the chart cannot describe.
- Example: a commercial property insurer deploys a catastrophe risk model that predicts probable maximum loss at the ZIP+4 level. The model sits in the cat modeling team, which reports to the CRO. The underwriting team, which actually binds the policy, sits under the CFO. The two teams share no common KPI, no shared data mart, and no single executive owns the end-to-end outcome. The result: underwriters ignore the cat model because their bonuses are tied to premium volume, not loss ratio.
I’ve forced the realignment twice:
The first time, we created a new “Risk Intelligence Office” (RIO) reporting to the CEO and staffed with underwriters seconded from underwriting, engineers from cat modeling, and a data product manager. The RIO’s mandate is to translate model output into underwriting rules within 60 days of each model release. The CRO and CFO jointly fund the office, and the underwriters’ bonus pool now includes a 5% clawback if the combined ratio in their book rises more than 3% above the corporate target.
The second time, the model was a telematics-driven auto rating engine. We did not move the actuary or the pricing team; instead, we inserted a “Pricing AI Liaison” role on the pricing team. The liaison’s KPIs are “% of manual overrides that align with model recommendations” and “time to deploy new rating factors.” The liaison reports to the pricing director but has a dotted line to the AI engineering squad. This keeps the pricing team accountable for the model’s financial outcome while preserving the actuary’s technical ownership.
Key insight: you do not need a separate AI division. You need a single role or office that owns the translation layer between model output and business action. Incentives eat data science for breakfast
Insurers reward underwriters for premium growth, not for loss ratio. Reward adjusters for speed, not accuracy. Pay actuaries for rate adequacy, not model interpretability. Introduce an AI artifact that contradicts any of these incentives and the artifact will be ignored or gamed.
A mid-size insurer deployed a claims fraud model that flagged 12% of auto bodily injury claims. The model reduced paid losses by 8% in the pilot cohort, which was statistically significant. Yet after six months the model’s lift disappeared. Why? The claims examiners started manually downgrading the flagged claims to “investigate later,” which preserved their speed metric while gaming the fraud model’s input. The true fraud rate did not change; the behavior changed.
The fix required a dual incentive shift: We added a “fraud model accuracy” metric to the claims manager’s bonus, weighted 15%.
We capped the “days open” metric at 30 days for non-fraudulent claims; flagged claims that exceeded the cap were automatically escalated to a specialized SIU team, removing discretion from the examiner. Within 90 days, the model’s lift returned to the pilot level. The behavioral change was not about training or change-control documents; it was about recalibrating the KPIs that the organization truly lives by.
How to redesign incentives without breaking the bank Stakeholder
Existing KPI AI artifact
New KPI Carrot or stick
Underwriter Premium written
- Predictive underwriting score % of bound policies matching model risk tier
- Bonus multiplier for 90%+ match Claim examiner
Claims closed per day Fraud flag
% of flagged claims investigated within 7 days Penalty if investigation > 7 days
| Cat modeler Cat model accuracy | Real-time event loss estimate % of policies rerated within 24 hours of event | Bonus tied to loss ratio improvement Actuary | Rate adequacy Dynamic pricing factors | % of manual overrides justified by documentation Bonus for <5% override rate |
|---|---|---|---|---|
| The table above is not hypothetical. I’ve seen each of these KPI shifts implemented at carriers ranging from $500 million to $12 billion in GWP. The common pattern: the new KPI is directly tied to the AI artifact’s intended business outcome, not to the artifact’s technical performance. | Governance: the step most teams skip until it’s too late | Insurers treat AI governance as an afterthought—something the compliance team will bolt on once the model is in production. That is backwards. Governance must be designed into the model’s deployment pipeline from day one, or the model will drift into irrelevance within a quarter. | I’ve reviewed three deployments where the model’s lift evaporated because no one defined the concept of “drift” for the business context: | A workers’ compensation return-to-work model started recommending earlier return dates after the state changed its disability duration guidelines. The model retrained every quarter, but no one updated the business rules that translated the ML score into a nurse case manager task list. |
| A homeowners catastrophe model used 2010 building code data for Florida; after Hurricane Ian, the model became systematically over-optimistic about roof resilience. A telematics UBI model assumed 12,000 annual miles as the baseline; after COVID commuting patterns changed, the model over-penalized low-mileage drivers. | In each case, the technical team had set up MLOps pipelines with automated retraining, but the business rules layer lacked a governance mechanism to detect when the real-world context changed. The fix required three artifacts: A “context change” checklist signed by the process owner every time a new regulation, market condition, or peril event occurs. | A drift dashboard surfaced to the process owner, not to data science. The dashboard shows business metrics (e.g., combined ratio, claim frequency, policy retention) not model metrics (e.g., AUC, F1). A single “AI owner” role with budget authority to halt or roll back the model if business metrics deteriorate beyond a threshold agreed at deployment. | Without these artifacts, the organization treats governance as a compliance checkbox instead of a risk control mechanism. The result is predictable model decay. The change-management playbook I use on day one | Here is the exact sequence I run when a new AI initiative lands on my desk: |
| Define the decision – What single business outcome will the AI affect? (e.g., reduce claims leakage by 12%, improve underwriting selection by 8%, shorten FNOL to assignment by 30 minutes.) Write it in one sentence and get the process owner to sign off. | Map the workflow – Draw the end-to-end process from trigger event to final action. Mark every human touchpoint and every system. Highlight the gap where the AI artifact will land. Redesign the role – Insert or redefine one role that owns the translation between model output and business action. Give that role KPIs and budget authority. | Redesign incentives – Adjust the bonus or clawback for every stakeholder whose behavior the AI will change. Make the incentive directly tied to the decision defined in step 1. Design governance – Define “drift” in business terms, not model terms. Create a dashboard and a single AI owner with the authority to halt the model. | Pilot with controls – Run a controlled pilot where half the population is exposed to the AI artifact and half is not. Measure the business outcome, not the model metrics. Only scale if the pilot meets the business target. | Document the operating model – After pilot success, write the new process in one page, including roles, KPIs, and governance. Circulate it to every stakeholder. This is your change-management charter; it replaces the project charter once the model is in production. |
| I’ve run this playbook four times. The first three times it felt bureaucratic; the fourth time it saved a $3 billion auto book from a costly rollback. What usually derails the playbook | Derailment Symptom | Fix Executive sponsor churn | Sponsor leaves or shifts priorities within 90 days Secure a co-sponsor from a different function (e.g., claims + underwriting) before day 30 | KPI misalignment Stakeholders cannot agree on the primary metric |
Pick one metric and freeze it; if it proves wrong, pivot later, but not until the pilot is complete Data access block
Legal or IT blocks access to the source system for months Negotiate a “shadow data mart” that replicates only the fields needed for the pilot; sign an NDA with the vendor to reduce legal friction
Vendor lock-in The vendor’s SaaS pricing model rewards usage, not outcome
Negotiate a pilot contract with a usage cap and a 30-day opt-out clause; tie the final contract to the pilot’s business outcome Change management is not a slide deck
- Insurers love PowerPoint. We’ll spend six weeks building a 40-slide deck that shows the AI’s ROC curve, precision-recall graph, and architecture diagram. Then we’ll roll out the model and discover the adjusters still use their sticky-note heuristics. The only artifact that matters is the one-page operating model signed by every stakeholder. Everything else is noise.
- I’ve seen this manifest in a Lloyd’s syndicate that spent £2 million on a blockchain reinsurance platform. After 18 months, the platform processed zero treaties because the underwriters refused to abandon the London market’s centuries-old slip process. The technical team called it a blockchain failure; the real failure was the absence of a single underwriter who agreed to own the new signing workflow.
- Contrast that with a U.S. regional carrier that deployed a telematics rating engine. The CFO insisted on a two-page charter that defined the new “Pricing AI Liaison” role, the bonus clawback for underwriters who overrode the model without documentation, and the monthly governance meeting where the liaison and the actuary reviewed model drift. Within 12 months, 78% of new auto policies used the telematics rating, and the combined ratio improved by 2.3 points. No slide deck could have achieved that result.
Final test: can you answer the “who loses their job” question?
- If you cannot name the role or the person whose work will be displaced or augmented by the AI artifact, you have not completed the change-management design. The answer is never “no one”; it is always “the person who currently performs the task that the AI will automate.”
- I’ve used this question as a litmus test at three carriers: A large auto insurer could not answer the question for its FNOL automation project. The pilot died when the claims supervisors refused to give up their exception queues.
- A mid-size specialty carrier named the role (senior underwriter, cat-exposed property), redesigned the role’s KPIs, and successfully scaled the model. A Lloyd’s syndicate named the role (the slip leader), but the role’s bonus was tied to premium volume, not efficiency. The model never scaled.
The pattern is clear: if the displaced role’s incentives are not realigned, the model will not scale. Full stop. Where to start tomorrow
Pick one AI initiative that is already funded and ask these three questions: What is the single business outcome? Write it in one sentence and get the process owner to sign it.
Who owns the translation layer between model output and business action? If the answer is “we’ll figure it out later,” stop the project. How will we measure drift in business terms, not model terms? Define the metric and the dashboard before the model ships to production.
- If you cannot answer all three, the project will fail to scale. Redirect the budget to the change-management design instead.
- I’ve done this reallocation three times in the last 24 months. Each time, the project restarted on a sounder footing and delivered measurable business value within a year. The alternative is another $1 million RPA bot gathering dust in the exception queue.
- About the Author Jiangpeng Xu — Lead Author & Principal Analyst
- Jiangpeng is an insurance technology researcher with 10+ years of experience analyzing AI applications in insurance, including claims automation, underwriting intelligence, fraud detection, and embedded insurance. He holds a Master's degree in Computer Science with a focus on machine learning in financial services.
- Was this article helpful? Comments.
