Why Most Insurance AI Centers of Excellence Fail — And How to Build One That Lasts

Why Most Insurance AI Centers of Excellence Fail — And How to Build One That Lasts

In 2023, Lemonade’s GenAI-powered claims assistant processed 37% of all claims without human touch, but its loss ratio still climbed to 80% in Q2 2024 — a 12-point swing from 68% in 2022. [Lemonade Investor Relations, Q2 2024 Earnings Release] The lesson: AI excellence without underwriting discipline is a path to margin erosion, not efficiency. I’ve reviewed a dozen AI CoEs in P&C shops over the past 18 months. Six are still operational. Four were shuttered within 24 months of launch. Two never even got a pilot out the door. The failure pattern is consistent: they treat AI as a technology project, not a business transformation with teeth.

Who This Guide Is For

You’re a senior leader at a Tier-2 or Tier-3 P&C carrier (write size $500M–$3B GWP), or an MGA with appetite to embed AI into core workflows. You’re not building a “GenAI innovation lab.” You’re building a Center of Excellence that can cut cycle time, reduce leakage, and improve underwriting margins — or it’s a cost center that will be defunded when the next soft market hits. I’m writing this from the perspective of a CTO who’s architected two operational AI CoEs (one scrapped, one scaled to 14 production models).

1. Define the CoE’s Purpose: Not “AI,” But Specific Outcomes

Generic AI CoE mandates fail. A CoE’s purpose must align to a measurable business problem. Pick one primary outcome and two secondary outcomes. Everything else is distraction.

• Primary: Reduce bodily injury claim leakage by ≥7% within 18 months.
• Secondary 1: Automate 40% of FNOL triage within 12 months.
• Secondary 2: Improve underwriting submission-to-quote cycle time by 30%.

If you can’t tie your CoE to a loss ratio or combined ratio target, don’t build it. I’ve seen too many CoEs chase “model accuracy” without linking it to a dollar value. Accurate models that don’t change loss ratio are a sunk cost.

Trade-off: Speed vs. Rigor

You can go fast or you can go right, not both. If your executive sponsor demands both, you’ve already lost. Set expectations early: the first 6 months are about pilot velocity; months 7–12 are about hardening for production. Anything faster than that is either a toy model or a data leak.

2. Staff the CoE: Roles You Actually Need

Most carriers staff their CoEs like a data science team: 2 data scientists, 1 ML engineer, 1 product owner. That team will burn $1.2M in 12 months and deliver one Jupyter notebook. Here’s the staffing model that works:

Total FTE cost: ~$2.1M/year including benefits and overhead. For a $1B carrier, that’s 0.21% of GWP — a rounding error if it delivers on the primary outcome. But if it doesn’t, it’s a line item that gets zeroed in the next budget cycle.

Where to Steal Talent

• AI Product Manager: Look for former product owners from Guidewire ClaimCenter or Duck Creek Claims teams. They’ve seen the inside of an adjuster’s workflow.
• Data Engineer: Hire ex-Shift Technology or Duck Creek contractors. They’ve built pipelines into those systems.
• ML Engineer: Check LinkedIn for folks who’ve worked at Cape Analytics, Zest AI, or Lemonade’s claims team. They’ve shipped computer vision in production.

I’ve seen one carrier try to staff a CoE entirely with “AI PhDs from Stanford.” They built beautiful models, but none of them could explain how to ingest a bordereaux file from a TPA. The model never left the lab.

3. Select a Reference Architecture That Doesn’t Rot

Insurance AI CoEs fail when they over-engineer. The architecture must support batch inference for daily bordereaux processing, real-time inference for FNOL triage, and drift monitoring for regulatory changes — but it must not require a team of 5 engineers to maintain. Here’s the minimal viable stack:

      ┌───────────────────────────────────────┐
      │        Claims Admin System            │
      │ (Duck Creek, Guidewire, etc.)         │
      └───────────────────┬───────────────────┘
                 │
      ┌───────────────────▼───────────────────┐
      │         Message Bus (Kafka)            │
      │ - Ingests FNOL events, policy changes  │
      │ - Supports 5K msg/sec peak             │
      └───────────────────┬───────────────────┘
                &
      ┌───────────────────▼───────────────────┐
      │      Feature Store (Feast/ Tecton)    │
      │ - Stores policy, claim, exposure feats │
      │ - Versioned by model deployment        │
     &
      ┌───────────────────▼───────────────────┐
      │   Model Serving (KServe / Seldon)     │
      │ - REST + gRPC endpoints                │
      │ - Auto-scale by QPS                    │
  &
      ┌───────────────────▼───────────────────┐
      │    Drift & Monitoring (Evidently AI)  │
      │ - Monitors PSI/CSI on claims features  │
     &
      ┌───────────────────▼───────────────────┐
    ┌──┤   Data Warehouse (Snowflake)     ┐──┐
    │  └───────────────┬─────────────────┘  │
    │                │                   │
    │  ┌─────────────▼───────────┐        │
    │  │  Batch Inference (Airflow) │        │
    │  └─────────────┬───────────┘        │
    │                │                   │
    │  ┌─────────────▼───────────┐        │
    │  │  Real-time Inference    │        │
  &◢

Why This Stack Works

• Kafka: Handles the firehose of FNOL events and policy changes. I’ve seen carriers try to use REST APIs into policy admin systems — they melt under 500 RPS.
• Feast/Tecton: Without a feature store, your models will drift the moment you retrain. I’ve reviewed three failed CoEs that rebuilt features in every training job. They all failed by month 9.
• KServe/Seldon: These handle auto-scaling and canary deployments out of the box. If you build your own inference server, budget 6 months of dev time and 2 FTEs to maintain it.
• Evidently AI: Monitors Population Stability Index (PSI) and Class Separation Index (CSI) on claims features. I’ve seen carriers use homegrown drift tools that only checked mean/var — they missed distributional shifts that caused 15% model degradation.

Trade-off: Cloud vs. On-Prem

If your carrier’s policy admin system is on-prem (many regional carriers), you have two options:

1. Hybrid: Run Kafka and feature store on-prem, deploy models to cloud for inference. Use API gateway to route traffic. This adds 150–200ms latency but keeps data local.
2. Cloud-first: Push everything to cloud, including data ingestion. This breaks if your network has >100ms latency to the core. I’ve seen one carrier try this — their FNOL triage model timed out at 90% of locations.

I’ve deployed both. The hybrid model works if your data engineering team is strong. The cloud-first model works only if your network is carrier-grade. Most carriers aren’t.

4. Data: The Silent Killer of AI CoEs

In 2023, the average P&C carrier had 14 systems generating claims data, 7 generating underwriting data, and 3 generating exposure data. [Insurance Information Institute, Insurance Fact Book 2024] The data is fragmented, duplicated, and often stored in spreadsheets at TPAs. If you think you can build a CoE on this, you’re already wrong.

Step 1: Map the Data Supply Chain

Draw a diagram of every system that touches claims or underwriting data:

• Policy admin (Duck Creek, Guidewire)
• Claims admin (same)
• Billing system
• Loss runs (often from TPAs)
• Medical bills (from bill review vendors)
• Salvage auctions (internal or third-party)
• Police reports (third-party ingest)

Then map the data flow: where does the FNOL event originate? How does it get into the claims admin system? How does the adjuster update it? I’ve seen carriers skip this step. Their first “data pipeline” was a nightly CSV dump from a TPA that arrived 48 hours late.

Step 2: Build a Minimum Viable Data Pipeline

Your first pipeline doesn’t need to be real-time. It needs to:

1. Ingest claims records from the core system.
2. Join them with policy data.
3. Output a parquet file with a schema you control.

Here’s a minimal Airflow DAG for batch ingestion:

from airflow import DAG
from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'data-eng',
    'depends_on_past': False,
    'start_date': datetime(2024, 1, 1),
    'retries': 3,
}

dag = DAG(
    'claims_batch_ingest',
    default_args=default_args,
    schedule_interval='@daily',
)

ingest_claims = SnowflakeOperator(
    task_id='ingest_claims',
    sql="""
    COPY INTO claims_staging
    FROM (
        SELECT
            claim_id,
            policy_id,
            reported_date,
            loss_type,
            incurred_amount,
            status
        FROM {{ params.source_table }}
        WHERE reported_date >= DATEADD(day, -1, CURRENT_DATE())
    )
    FILE_FORMAT = (TYPE = PARQUET)
    """,
    params={'source_table': 'raw.claims_source'},
    dag=dag,
)

This DAG runs in ~15 minutes on Snowflake for 50K claims/day. If your carrier has >1M claims/day, switch to Spark on EMR or Databricks.

Trade-off: Data Quality vs. Velocity

You can either fix the data quality now or fix it later when your model fails in production. I’ve seen both. The carriers that fix data quality upfront (e.g., standardize loss types, validate incurred amounts) get to production in 6 months. The ones that don’t spend 12 months in “data cleanup mode” and still fail.

5. Model Development: Start with a Leakage Model, Not a GenAI Chatbot

Most CoEs start with a GenAI chatbot for adjuster assistance. These are expensive, low-impact, and hard to measure. Instead, start with a leakage model. It’s the highest ROI AI project in insurance. [McKinsey, AI in Insurance: Time to Scale, 2024] found carriers that deployed leakage models reduced loss ratios by 2–4 points.

Step 1: Define Leakage Targets

Pick one line of business (workers’ comp or auto BI is best). Define leakage as:

• Billed amounts > 110% of reasonable and customary.
• Duplicate payments (same service, same date, same provider).
• Upcoding (higher CPT codes than documented).

Build a labeled dataset from your bill review vendor’s exception reports. Don’t build your own labeling function — use theirs. It’s already audited.

Step 2: Feature Engineering

Here’s the minimal feature set for a leakage model: