I ran into a similar situation doing due diligence for a client: This came up in a conversation with a CTO at an insurtech firm: I’m comparing six real-world stacks that have run at least 10,000 claims in production. Four are AI-first; two remain predominantly manual. Each stack is live inside at least two carriers with ≥$5B annual written premium. I’ve anonymized vendor names to avoid vendor-marketing noise and focused on four metrics that matter to an FNOL desk: median triage-to-assignment time, severity misclassification rate, combined ratio impact, and total cost of ownership per 1,000 claims.
Triage Approach Model Type Median Triage-to-Assignment (hours) Severity Misclassification Rate Combined Ratio Impact (bps) TCO per 1k Claims (USD) AI-first: End-to-end LLM + Rules Engine Multimodal LLM + fine-tuned severity classifier
| 0.7 3.1% -18 $1,420 AI-first: OCR + NLP Severity Model Document OCR + BERT severity classifier 1.4 4.2% | -12 $980 AI-first: Parametric Trigger + Simple ML Rule-based parametric trigger + logistic regression 2.1 6.7% -8 $650 | AI-first: Legacy OCR + Manual Review Legacy OCR + human adjuster override 5.3 8.9% +5 $1,850 Manual: Tiered Human Triage Human-led tiering by seniority | 8.2 11.4% +22 $2,100 Manual: Supervised Junior Adjuster Single junior adjuster triage 12.0 14.2% | +38 $1,600 Sources: Swiss Re sigma 02/2024 “AI in Claims Processing” (sample 42 carriers), Verisk 2023 “Claims Triage Benchmarks” (n=3,142 carriers), internal carrier A/B test data 2022-2024 (Tier 1 P&C). How the Metrics Map to Pain Points 1. Severity Misclassification Rate → Leakage, Reserve Accuracy, IBNR | Misclassification rate is the single best proxy for leakage. A 1% reduction in misclassification typically drops incurred-but-not-reported (IBNR) by 2-3% and improves loss ratio by 4-6 bps. The LLM stack wins because it ingests unstructured text (police reports,. medical bills, photos) and cross-checks against policy terms in real time. The parametric model loses because it assumes a fixed trigger (e.g., wind speed > 70 mph) and ignores policy sublimits, deductibles, or prior losses. |
|---|---|---|---|---|---|
| Trade-off: LLM stacks require a 4-6 week fine-tune per line of business and exhibit “prompt drift” if carrier guidelines change. A carrier I worked with saw misclassification rise from 3.1% to 3.9% after a new state regulation changed sublimit definitions; the model had to be retrained. | 2. Combined Ratio Impact → Loss Ratio + Expense Ratio | The combined ratio bps impact combines loss ratio improvement (lower severity leakage) and expense ratio change (labor cost shift). The manual stacks show positive bps because headcount dominates—adding one FTE adjuster at $85k loaded cost per year raises the expense ratio by ~3.5 bps on a $2B book. The AI-first stacks reduce expense ratio by 12-30 bps while modestly improving loss ratio; the net is negative combined ratio impact. | Trade-off: The LLM stack’s TCO hides a hidden line item—vendor lock-in. One carrier paid $240k for a 2-year license to a proprietary multimodal LLM; after renewal, pricing jumped 60%. Switching costs are high because the model is trained on the carrier’s historical claims language. | 3. Triage-to-Assignment Time → Customer Satisfaction & Litigation Risk | The median customer expects an assignment email within 4 hours. Anything slower increases the probability of a plaintiff attorney’s demand letter arriving before the adjuster does. The manual stacks fail this test; the parametric stack is borderline. The OCR+NLP stack hits the 4-hour SLA on 87% of claims but misses the 2-hour SLA on weather events that overwhelm the carrier. |
| The LLM stack achieves 94% SLA adherence at 2 hours, but only if the carrier front-loads a dedicated GPU cluster for inference. A Tier 2 carrier with a shared cloud GPU pool. saw median latency spike to 2.8 hours during a CAT event. That’s still better than the manual stack’s 12 hours, but it’s a compliance risk for SLAs tied to state insurance department bulletins. | Architecture Deep Dive: Where the Gaps Appear AI-first: End-to-end LLM + Rules Engine Stack composition: Multimodal LLM (vendor-agnostic, LLaMA 3.1 405B fine-tuned on carrier corpus) Policy graph DB (open-source JanusGraph, stores deductibles, sublimits, prior losses) Decision engine (Drools rules, 1,200+ policy clauses) Human-in-the-loop queue (10% of edge cases auto-routed to senior adjuster) Strengths: | Handles nested deductibles and endorsements without brittle regex. Reduces repetitive adjuster questions (“is this a covered peril?”) by 40%. Weaknesses: Model size (405B parameters) requires 8x A100 GPUs per region for inference; infra cost is ~$0.42 per triage. | Prompt drift: a new state regulation (e.g., California SB 1028) can change the correct triage outcome; carriers see a 0.8% misclassification jump within 30 days unless the model is retrained. When to choose this stack: Carrier writes ≥$5B GWP with complex products (homeowners, auto, cyber). IT budget allows GPU clusters per region and a dedicated ML engineering team. | AI-first: OCR + NLP Severity Model Stack composition: Document OCR (Tesseract + proprietary layout parser) BERT-based severity classifier (fine-tuned on historical loss runs) Lightweight rules engine (Python-based, 200 clauses) No GPU required; runs on CPU or small GPU cluster. Strengths: TCO is 30% lower than the LLM stack. | Latency is predictable because it’s not prompt-dependent. Weaknesses: Can’t parse handwritten FNOL forms; requires rekeying at 12% of intake. Severity model struggles with multi-peril claims (e.g., hail + water damage) unless explicitly trained. When to choose this stack: Carrier writes $1B-$5B GWP with relatively simple products (auto, small commercial). Legacy OCR already exists; minimal integration friction. Parametric Trigger + Simple ML |
| Stack composition: Rule-based trigger (NOAA wind speed > 70 mph) Simple logistic regression for claim count surge prediction No LLM, no fine-tuning required. Strengths: TCO is lowest in the table ($650 per 1k claims). Explainable; regulators love it. Weaknesses: | Misclassification spikes during blended peril events (wind + fire + looting). Can’t handle policy-specific deductibles or sublimits. When to choose this stack: Carrier writes a single peril product (e.g., named storm). Regulatory environment demands transparency over accuracy. Manual: Tiered Human Triage Stack composition: Tier 1: Junior adjuster, 2 years experience | Tier 2: Senior adjuster, 7+ years Tier 3: Supervisor, 15+ years Strengths: No model risk; outcome is explainable to a jury. Handles edge cases (policy endorsements, prior losses) without rekeying. Weaknesses: Cycle time is 12 hours median; SLA adherence is <50%. Severity misclassification is 11.4%—the highest in the table. | When to choose this stack: Carrier writes <$500M GWP or serves a niche market where model explainability is legally required. Labor arbitrage is still cheaper than GPUs (e.g., Philippines BPO cost $28k loaded per FTE) (at least that's what the slide deck says). Regulatory & Governance Risks You Can’t Ignore | Model Explainability vs. Black Box | The LLM stack is the least explainable. A California DOI examiner recently rejected a carrier’s AI triage model because the LLM’s chain-of-thought output could not be reproduced in a hearing. The OCR+NLP stack fares better: severity decisions can be traced to specific policy clauses and claim text excerpts. The parametric stack is fully explainable but brittle. |
| Trade-off: If your state enforces NAIC Model Bulletin #2023-1 on AI use in underwriting and claims, the LLM stack will require a model governance board, adversarial testing, and a formal fairness assessment—adding 6-9 months to implementation. | Data Privacy & PHI/PII Redaction | LLM stacks ingest medical bills, police reports, and repair estimates containing PHI/PII. One carrier I advised had to implement a dual-stack architecture: a PHI/PII redaction pipeline (Microsoft Presidio) before the LLM could ingest the document. That added $0.08 per page and increased latency by 1.2 seconds per claim. The OCR+NLP stack avoids PHI/PII by design—it only ingests severity indicators, not full documents. | Trade-off: If your carrier writes workers’ comp or auto injury claims, the LLM stack’s PHI/PII exposure risk may outweigh its accuracy gains unless you budget for a dedicated redaction pipeline. ROI Model: When AI Triaging Pays Back | I built a simple NPV model for a $5B GWP personal lines carrier. Assumptions: AI triage reduces severity leakage by 4 bps, expense ratio by 15 bps, and improves customer NPS by 8 points (translating to 0.3% lower lapse rate). | Metric Manual Baseline AI Triage Stack Delta Annual Severity Leakage Saved $0 $21M $21M |
| Annual Expense Saved $0 $15M $15M Annual NPS Uplift Benefit $0 $7.5M $7.5M | Annual AI Stack Cost $0 $3.6M -$3.6M Net Annual Benefit $0 $39.9M Payback Period | N/A 5.4 months Sources: PwC 2024 “AI in Insurance Claims” (ROI model template), carrier A/B test ROI deck 2023. | Key caveat: The model assumes the carrier already runs a modern OCR pipeline and policy admin system. If the carrier still relies on paper bordereaux, the TCO gap widens—CAPEX for OCR alone jumps to ~$1.2M and payback stretches to 14 months. | Which Stack to Pick: A Decision Matrix Use the table below to pick the right stack for your constraints. I’ve weighted severity misclassification and combined ratio impact at 35% each, SLA adherence at 20%, and TCO at 10%. Constraint LLM + Rules | OCR + NLP Parametric Manual Complex products (homeowners, cyber) 9.2 7.8 5.1 4.3 |
| Simple products (auto, small commercial) 8.4 9.1 7.2 6.5 Regulatory pressure for explainability 5.7 8.3 | 9.5 9.7 GPU budget >$100k/year 8.9 7.5 9.8 4.2 Labor arbitrage cheaper than GPUs | 4.1 5.2 6.7 9.3 Workers’ comp or auto injury claims 6.8 8.9 4.3 | 7.1 | Scores are scaled 1-10, with 10 being the best fit. If your constraint is “complex products + regulatory pressure + GPU budget >$100k,” the LLM+Rules stack wins (8.9). If your constraint is “simple products + labor arbitrage,” the manual stack wins (9.3). | Implementation Pitfalls That Kill Value 1. Overfitting to Historical Data One carrier trained its severity model exclusively on claims closed within 30 days. When a large hail event hit, the model under-reserved for claims that took 90 days to close. IB |