Predictive underwriting analytics: where to source insurance data and how to wire it up Predictive underwriting analytics: where to source insurance data and how to wire it up

The $14M model that never left staging: what breaks when you skip data sourcing

In 2022, a mid-size P&C; carrier in Texas greenlit a $14M predictive underwriting platform built on a feature store with 2,400 engineered variables. The model scored well in backtest. It also produced a 31% loss ratio improvement on paper that evaporated in production because the primary data feed from their claims system arrived late, incomplete, or with schema drift every six to eight weeks. The model ran on stale features for 74 days before an ops team noticed the drift. I was not on that engagement, but I have sat in post-mortems with three separate carriers that made the same mistake: they treated the ML model as the project and treated data plumbing as an afterthought.

I have spent the last eight years as CTO and VP of Engineering at two MGA-backed insurers and one large P&C; carrier. I have shipped underwriting models that score applicants for homeowners, commercial property, and worker's comp. The pattern is consistent. The model architecture is the easy part. The wiring is where projects die. This article is a practitioner's field guide on where to source insurance data for predictive underwriting analytics and how to actually connect it, with enough specificity that your platform team can start on Monday.

Where underwriting data actually lives (and why "we have the data" is a lie)

When a CFO or VP of Underwriting says "we have all the data we need," they are describing a fantasy. In my experience auditing 15+ carrier data environments, the reality is fragmented. Your core policy administration system (PAM) holds policy data. Your claims system (Guidewire ClaimCenter, CCC, or a legacy mainframe) holds loss data. Your TPA network holds 40 to 60% of claims detail for smaller carriers. Your actuarial department keeps reserving files in a spreadsheet. Your compliance team has a separate regulatory filing database. None of these talk to each other natively.

The NAIC 2023 Annual Statistical Supplement (published July 2024, covering fiscal year 2023) documents that the U.S. property/casualty industry wrote $1.08 trillion in direct premiums. The granularity of that data, from a sourcing perspective, is a mess. Carriers report to state regulators through annual and quarterly statements, but the raw transactional data that a predictive model needs, at the policy line, endorsement, and claim line level, is trapped in systems that predate REST APIs.

For health and life, the situation is slightly better. NAHP (National Association of Health Plans) publishes a data dictionary, and carriers exchange claims through Clearinghouse Standard Format (CSF) and HIPAA 5010 transactions. But even there, 60-day lag on closed claims is the norm, and the "closed" designation is unreliable until 12 months post-incident.

Here is the sourcing map I use when I onboard a new carrier or MGA:

• Core systems (PAM, billing, policy issue): Guidewire PolicyCenter, Duck Creek, AssuredParity, Salesforce LWC. Latency: real-time to 24 hours via API, or T+1 batch via flat file.
• Claims systems: Guidewire ClaimCenter, CCC Intelligent Solutions, Accuris, or legacy mainframes (Aspen, Tyler Technologies). Latency: 48 hours to 30 days depending on whether you get FNOL (first notice of loss) events or closed-claim files.
• TPA data: For workers' comp and casualty, this is where the body of claims data lives. TPA platforms like Valsafe, Sedgwick, or PPGI export via SFTP, API, or periodic CSV drops. Latency is the biggest pain: 30 to 90 days is common.
• Parametric and IoT: Weather feed (ECMWF, NOAA, or commercial providers like Windy/StormGeo), GPS telematics from telematics providers (Verizon Connect, Samsara), building sensors (Brick, Pelti). Latency: near-real-time for weather, 24 to 72 hours for IoT.
• Alternative data: Satellite imagery (ICEYE, Capella Space), credit and consumer data (LexisNexis, Equifax), public records (county assessor, FEMA flood maps), social media or job-market signals (indeed.com API, LinkedIn Sales Navigator, though the last two have compliance friction).

The critical insight, and the one that took me three failed deployments to learn: you do not need all of these on day one. You need a defensible minimum viable feature set. I have seen carriers spend 18 months trying to integrate satellite data into a homeowners model when the single biggest predictor they were missing was a 12-month lookback on neighboring claims from a TPA feed they already had access to.

The wiring problem: architecture patterns that survive contact with reality

Let me be blunt. If your engineering team proposes a single real-time streaming pipeline for all underwriting data, push back. The economics do not work. Weather data and IoT telemetry benefit from real-time ingestion via Kafka or Kinesis. TPA claims data, which arrives as monthly CSV drops via SFTP, does not. Forcing a 40-year-old claims database into a microservice event-driven architecture will stall your project for two quarters and still produce a fragile system.

The pattern I ship most often, and what I have used at two different carriers, is a tiered ingestion architecture:

Tier 1: Real-time and near-real-time (sub-hour latency)

APIs from PAM (policy issue, endorsement events), weather feeds, telematics streams. These feed into a Kafka topic or a managed stream (AWS Kinesis, GCP Pub/Sub). Your feature store (Databricks Feature Store, Tecton, or a home-built solution on PostgreSQL) consumes and materializes these into point-in-time correct features. This tier powers interactive underwriting: an adjuster or underwriter opens a quote and sees a risk score in under 200ms.

Tier 2: Batch (daily to weekly latency)

Claims data, actuarial reserving files, TPA reports, regulatory filings. These run on scheduled ETL: Airflow DAGs or dbt models that land into a data warehouse (Snowflake, BigQuery, Redshift). Your model retraining jobs pull from here. Latency of 24 hours to 7 days is acceptable because the signal you are extracting is trend, not transaction.

Tier 3: Event-driven, low-frequency (monthly to quarterly)

Large loss events, catastrophe modeling runs (RMS or AIRs), annual actuarial studies. These are not streams. They are triggers. A cat model outputs a new building damage index. You retrain. A quarterly actuarial study rewrites your loss development factors. You recalibrate. Wire these as webhooks that trigger a retraining pipeline, not as continuous feeds.

The counterargument I hear from data science leads: "But if my features are stale, my model drifts." True. But the alternative, ingesting TPA data in real-time, requires the TPA to expose an API, and most will not. You negotiate a 48-hour API window and build a reconciliation job that runs nightly. You accept the lag. You design monitoring around it. You alert when the lag exceeds SLA. This is engineering, not a science problem.

Comparison table: sourcing and integration across four data categories

Numbers are from my vendor conversations and internal budget reviews at carriers I have advised. They will vary by size and negotiation leverage. The pattern: TPA data is the most expensive relative to the value it delivers per dollar, because the licensing terms are opaque and the data quality is inconsistent across 200+ TPA providers in the U.S. market. The NAIC 2023 Annual Statistical Supplement confirms the fragmentation: over 6,000 active P&C; insurers in the U.S., many of which are MGA-managed, meaning the data chain has an extra link and an extra set of SLAs.

What Gartner and the market data say about where this is heading

Gartner's 2024 "Market Guide for AI in Insurance" (published Q2 2024) notes that fewer than 12% of insurance companies have AI models in production for underwriting, and the primary blocker cited is not model quality. It is data readiness. The report estimates that the median time from "data available" to "model in production" is 14 to 18 months for carriers without a mature data platform. McKinsey's 2023 report "Insurance underwriting: The next decade" (published in their Insurance Practice series) arrives at a similar conclusion from the demand side: carriers that have wired their data pipelines end-to-end report a 15 to 25% reduction in adverse selection losses versus carriers still running actuarial manual schedules.

Lloyd's of London's 2023 Insurance Market Report, in its section on digital transformation, flags that the market's largest underwriting innovation spend is not in model architecture. It is in data integration across the broker, managing agent, and insurer chain. For specialty lines, where a single risk might be placed across five underwriters at four different Lloyd's syndicates, the data wiring problem is exponentially harder than a single-carrier PAM integration.

Here is where I part company with the consulting narrative. The Gartner and McKinsey numbers are correct but incomplete. They assume a greenfield build. Most carriers I work with are not greenfield. They have a Guidewire PAM that was implemented in 2009, a claims system that was migrated off a mainframe in 2016, and a data lake that "works" but has no data contracts, no lineage, and no SLA. The wiring problem is not "build a new pipeline." It is "make the existing mess observable and testable before you add ML on top."

A practical wiring sequence for your next quarter

If you are a CTO or VP Eng and your board just approved a predictive underwriting program, here is the sequence I would enforce. I have used this at two carriers and one MGA. It assumes a team of 4 to 6 engineers and 2 data scientists.

Weeks 1 to 4: Data audit and contract freeze. Map every system that touches underwriting data. For each, document: what format does it export, what is the refresh cadence, who owns the schema, what is the SLA. Freeze your feature set. I have seen projects where the feature set grew from 800 to 2,400 variables over 12 months and the project never reached production. Lock 150 to 300 features for v1. You can add more. You cannot build a foundation on a moving target.

Weeks 5 to 10: Tier 1 and Tier 2 pipelines. Build the Kafka (or managed stream) ingestion for PAM events and weather feeds. Build the Airflow/dbt batch jobs for claims and TPA data. The key engineering deliverable here is not the model. It is the data contracts: a JSON Schema or Protobuf definition for every feed, validated at ingestion. If a TPA drops a CSV that is 200 rows short of the expected count, the pipeline fails loud, not silent. I have a standing rule: a data job that does not alert on schema drift is a production incident waiting to happen.

Weeks 11 to 16: Feature store and model serving. Materialize your point-in-time features. Build the serving layer. This is where you decide: batch scoring (run nightly, refresh scores, underwriter sees the latest score when they open the policy) versus real-time scoring (API call at FNOL or quote). For most P&C; lines, batch is sufficient for underwriting. Real-time matters for claims and catastrophe. Do not over-engineer.

Weeks 17 to 24: Monitoring, drift detection, and go-live. Deploy to a shadow mode first. The model scores, the underwriter makes the decision, you log the disagreement. Run this for 60 to 90 days. Track: model precision/recall vs. actual loss experience, feature drift (PSI per feature, alert at 0.2), data latency (is TPA feed on time?). Only when the shadow mode shows no unexplained divergence do you flip to "model recommends, underwriter accepts or overrides."

For more on the monitoring layer, see our related piece on model drift monitoring in underwriting and feature store architecture for P/C carriers.

Failure modes I have seen (and what they cost)

One carrier in the Southeast built a predictive model for commercial property that included a feature derived from TPA injury severity scores. The TPA changed their scoring rubric in March without notice. The model's loss ratio prediction shifted by 8 points over two weeks before anyone caught it. The carrier had to manually re-underwrite 4,200 policies. The cost: roughly $340K in re-underwriting labor and two weeks of elevated loss ratio on their quarterly report. The root cause was not the model. It was the absence of a data contract between the TPA and the carrier's data team.

Another: a health insurer wired their underwriting model to a claims feed that excluded subrogation recoveries. The model learned that certain injury types had "higher" loss because the recovery data was not in the training set. The model was not wrong given the data. The data was wrong. The fix required 11 months of backfill and a retraining cycle that pushed the model's go-live from Q3 to Q2 of the following year.

These are not edge cases. They are the default state if you do not build observability into your data pipelines before you build your model. The engineering work is less glamorous than the ML work, but it is the work that determines whether the model survives its first six months in production.

The next 18 months: what I would bet on

The data sourcing problem will not get easier in the next 18 months. The NAIC's push for standardized data reporting (their 2024 modernization agenda for statutory reporting) will help at the regulatory layer but will not touch the transactional feeds that your model actually needs. The TPA market is consolidating, which means fewer, larger data vendors, which means better SLAs but also more leverage for those vendors on pricing. I expect TPA data licensing fees to rise 15 to 25% over the next two years as AI adoption increases demand for granular claims data. You can track this trend in the TPA data licensing trends piece we published last quarter.

The architectural shift I am watching is the move from carrier-owned data pipelines to shared industry data platforms. The Insurance Data Coalition and the Lloyd's market's shared data initiatives are early, but the direction is clear: the cost of sourcing and cleaning underwriting data will be socialized across the market, and the carriers that build their internal pipelines now will get to set the standards. That is a strategic position worth protecting.

If you are starting this work next quarter, the single highest-leverage action is not buying a new ML platform. It is sitting in a room with your TPA, your PAM vendor, and your data engineering lead for four hours and writing down every feed, its format, its cadence, and its SLA. Put it in a document. Share it with the data science team. That document, more than any model architecture, is what will determine whether your predictive underwriting analytics program ships in 18 months or becomes the $14M model that never leaves staging.

Information verified as of Q2 2025. Sources: NAIC 2023 Annual Statistical Supplement (FY2023 data), Gartner 2024 Market Guide for AI in Insurance, McKinsey 2023 "Insurance underwriting: The next decade," Lloyd's of London 2023 Insurance Market Report.