State Farm filed 116 patent applications in 2023 related to AI and automated decisioning. Progressive has been running ML-based claims decisions since 2019, processing over 2 million auto claims through its ImageClaim system. The gap between these incumbents and everyone else isn't funding. It's the platform. This guide walks through building one from scratch, based on what I've learned deploying three at carriers ranging from $50M to $2B in premium.
I'm writing this from the CTO side. You don't need permission to build a better claims pipeline. You need a architecture that won't collapse when your first model goes into production on a Friday afternoon. Below is the map. Why standard ML models fail in decisioning
A credit bureau score returns a single number. A fraud model returns a probability. Neither answers the question a claims adjuster faces at 2 AM: do I pay this $8,000 claim now, escalate it, or flag it for investigation?
Decisioning platforms solve this gap by combining model outputs with business rules, regulatory constraints, and operational workflows into actionable decisions. The difference between a model that scores well on AUC and a platform that reduces loss ratio by 3 percentage points is enormous, and my first carrier client burned through $400k on a standalone ml project that produced good predictions. They had no workflow integration, no human-in-the-loop checkpoints, and no explainability for compliance officers. I still think about that failure mode every time someone asks whether they should "just build a model."
A proper decisioning platform has five layers: data ingestion, feature computation, model inference, rule engine, and action orchestration. Each layer has failure modes that will bite you. The feature store will drift. The rule engine will silently short-circuit. The model serving layer will add latency that tanks your FNOL-to-decision SLA. You need to architect around these, not after they appear.
Architecture overview The platform I'm describing is event-driven with a hybrid batch/streaming feature pipeline. It runs in production at two carriers I've consulted for, handling roughly 50,000 decisions per day across auto, property, and workers' comp lines.
Layer Function
Key Components Latency Target
Data Ingestion Event capture and validation
| Kafka, Flink, API gateway Sub-100ms | Feature Store Point-in-time correct feature retrieval | Tecton or Feast + Redis/S3 < 50ms | Model Serving Real-time inference |
|---|---|---|---|
| SageMaker, KServe, or TorchServe < 200ms | Rule Engine Business logic and regulatory constraints | Brave or custom Drools deployment < 100ms | Action Orchestration Decision routing and workflow execution |
| Temporal or AWS Step Functions Async, no hard limit | Total end-to-end latency target: under 500ms for automated decisions, under 5 seconds for escalated claims requiring human review. Anything slower and your adjusters stop using it. Step 1: Define decision categories and outcomes | Before you write a single line of code, map every decision type your platform will handle. Most carriers I work with have three buckets: Automated straight-through processing (STP): Clear-cut claims with high confidence. Pay or deny without human touch. | Assisted decisions: Model provides recommendations, human makes the final call within defined guardrails. Investigation queue: High-risk signals or policy exceptions routed to senior adjusters. |
| For each bucket, define the output schema. A decision object should contain: claim_id, decision_type (pay/escalate/investigate), confidence_score, model_version, feature_importance_summary, rule_flags, timestamp, and audit_trail. Don't skip the audit_trail. Regulators will ask for it, and you'll be scrambling if you don't have it. | Write this schema down and lock it before proceeding. Schema changes in production are the single most expensive mistake I see teams make. One carrier lost three weeks of engineering time rebuilding their decision pipeline after the product team "quickly" added a new field to the output object without versioning the schema. | Step 2: Build the feature pipeline | This is where most projects die. Not because the models are bad, but because the features are wrong, stale, or impossible to reproduce at inference time. Point-in-time correctness is non-negotiable. If your training data includes features computed from information that wasn't available at the moment of the original decision, you've introduced look-ahead bias that will destroy your production performance. |
| The feature pipeline has two modes: batch precomputation for historical model training, and real-time computation for live inference. Here's how I've implemented this with Tecton, which handles both modes: | The critical design decision here is TTL. I use a 24-hour TTL on features because insurance events (claims, FNOLs, repair estimates) are inherently asynchronous. A feature computed at 9 AM on Monday might not be stable until Wednesday when all supporting documents arrive. Setting TTL too low creates stale feature mismatches between training and serving. Setting it too high introduces unnecessary latency. | Feature store selection matters less than you'd think. Feast works fine for smaller deployments. Tecton adds operational overhead that pays off at scale. I've seen teams spend 40% of their budget on feature store configuration and maintenance in year one. Budget accordingly or build something simpler. | Step 3: Model development with deployment constraints |
| Write models for production, not for Kaggle. The models that perform best in decisioning platforms share three properties: interpretability, stability, and speed. Gradient boosting models (XGBoost, LightGBM) remain the workhorse. Neural networks offer marginal gains on structured insurance data and introduce massive explainability headaches. I've seen two teams burn months on deep learning approaches that ultimately produced less trustworthy decisions than a well-tuned LightGBM. | Here's the training configuration I use as a baseline: | Monotonic constraints are the single most underutilized tool in insurance ML. They encode domain knowledge directly into the model, reducing the chance of counterintuitive predictions that compliance will flag. Your model doesn't need to be perfect. It needs to be defensible. | Model validation requires holdout periods that mirror production distribution. I use 6-month forward validation windows, not random splits. Insurance data has seasonality (hurricane season, winter freeze claims) and trend (inflation in repair costs). Random splits will give you optimistic metrics that collapse in production. |
Step 4: Deploy the serving infrastructure
Model serving is where ML projects either become operational tools or remain beautiful Jupyter notebooks. The infrastructure must support A/B testing, shadow mode deployments, and instant rollback. Shadow mode is critical. Before any model touches a decision, run it in shadow for 30 days. Compare its output against human decisions without affecting the actual outcome. This reveals calibration issues and edge cases without risking claims payouts.
Here's the KServe deployment configuration:
- Minimum three replicas for high availability. Autoscaling based on request concurrency, not CPU, because decision latency is what adjusters measure. A model that takes 2 seconds at low load but 200ms at high load is worse than a model that takes 1 second consistently. Your SLO should be p99 latency, not average.
- Container registry strategy matters more than the framework. I use ECR with image signing. Every model artifact gets a SHA-256 hash logged in the model registry before deployment. If a model causes problems in production, you need to know exactly which artifact was deployed, not just the version number. Version numbers lie. Checksums don't.
- Step 5: Rule engine configuration
Models predict. Rules decide. The rule engine sits between model output and final action, applying business logic that models can't reliably capture: regulatory constraints, policy exclusions, contractual obligations, and escalation thresholds. Rules are easier to audit and update than models. Change a compliance requirement? Update the rule. Don't retrain the model.
Here's how I structure rules in a YAML-based engine. This approach is simpler than Drools for most insurance use cases and easier to maintain:
Rule conflict resolution is the hardest part. What happens when two rules apply to the same decision with conflicting actions? I use a priority system with explicit conflict logging. Every conflict gets recorded with timestamps, rule IDs, and the resolution taken. Regulators ask about this during audits, and having a documented conflict resolution process is worth more than any rule engine feature.
Rule performance should be measured separately from model performance. Track rule hit rate, conflict frequency, and escalation volume. A rule that triggers 99% of the time is probably too broad. A rule that never triggers should be reviewed or removed. Unused rules accumulate technical debt and slow down the engine.
Step 6: Action orchestration and workflow
# tecton_config.py - Feature materialization setup
from tecton import FeatureView, Entity, FeastSource
from datetime import timedelta
# Entity definitions
policyholder = Entity("policyholder", join_keys=["policyholder_id"])
claim = Entity("claim", join_keys=["claim_id"])
# Historical batch source for training
batch_source = FeastSource(
table_name="claims_features_batch",
timestamp_field="created_at",
online=False
)
# Real-time source for inference
stream_source = FeastSource(
table_name="claims_features_stream",
timestamp_field="event_time",
online=True
)
# Feature view combining both sources
claims_feature_view = FeatureView(
name="claims_decision_features",
entities=["claim", "policyholder"],
mode="online_and_offline",
features=[
ClaimAgeFraudScore().output,
RepairCostHistoryMean(policyholder).output,
AdjusterWorkload().output,
WeatherEventProximity().output,
],
ttl=timedelta(hours=24),
sources=[batch_source, stream_source]
)The decision arrives at the orchestration layer as a JSON object. The orchestrator translates it into business actions: payment initiation, email notification to adjuster, document request, supervisor approval workflow, or investigation queue placement. This is where the platform touches reality. Bad orchestration means good decisions never get executed.
Temporal is my preferred workflow engine. It handles retries, timeouts, and state persistence automatically: Idempotency is non-negotiable in the payment path. Network timeouts happen. Workers crash. The orchestrator retries. Without idempotency keys, you pay claims twice. I've seen it. The fix is expensive. Prevention is cheap.
Workflow observability is equally important. Every decision that flows through the platform should generate an audit log with: decision timestamp, model version, feature values at decision time, rule evaluation results, action taken, and outcome. This log becomes your ground truth for model monitoring and compliance reporting. Build it before you need it.
Step 7: Monitoring and model drift detection Models decay. Insurance data shifts with inflation, regulation, and behavior change. The monitoring layer catches this before it costs money. I track four categories of drift:
Input drift: Feature distribution changes. A new claim type enters the system with different feature characteristics. Detect with PSI (Population Stability Index) or KL divergence on feature distributions. Concept drift: The relationship between features and targets changes. Fraud patterns evolve faster than model retraining cycles. Detect with statistical tests on prediction residuals and time-series decomposition of model scores.
# training_config.py
import xgboost as xgb
from sklearn.model_selection import TimeSeriesSplit
# Insurance data has temporal dependency. Use time-based splits.
splitter = TimeSeriesSplit(n_splits=5)
# Feature importance tracking for compliance
model_params = {
'objective': 'binary:logistic',
'max_depth': 6,
'learning_rate': 0.05,
'n_estimators': 500,
'subsample': 0.8,
'colsample_bytree': 0.8,
'reg_alpha': 0.1,
'reg_lambda': 1.0,
'tree_method': 'hist',
'eval_metric': ['auc', 'logloss']
}
# Required: monotonic constraints for regulatory defensibility
# Ensures model respects known business relationships
# e.g., more prior claims should never decrease fraud probability
monotone_constraints = {
'prior_claims_3yr': 1, # monotonic increase
'policy_tenure_years': -1, # monotonic decrease
'claim_amount': 1, # monotonic increase
'distance_to_last_loss': 1 # monotonic decrease
}
# Train with explainability outputs
trainer = xgb.XGBClassifier(
**model_params,
monotone_constraints=monotone_constraints,
enable_categorical=False # One-hot encode instead for interpretability
)Performance drift: Model accuracy degrades on recent data. Detect with rolling window AUC comparison against the training baseline. Operational drift: Latency, error rates, and throughput changes in the serving infrastructure. Detect with SLO monitoring on the serving layer.
Here's the drift detection pipeline:
Drift alerts should trigger human review, not automatic retraining. Retraining on drift signals without investigation creates a feedback loop where the model chases moving targets. The correct response is usually: investigate the data change, determine if it's temporary or structural, then decide whether retraining is warranted.
Resource estimates and timeline Building a production decisioning platform is a six to nine month effort for a team of five engineers. Here's the realistic breakdown:
Phase Duration
# kserve_deployment.yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: claims-fraud-decision
namespace: decisioning-platform
spec:
predictor:
model:
modelFormat:
name: xgboost
storageUri: "s3://model-artifacts/claims-fraud-v2.3/"
env:
- name: MONOTONIC_CONSTRAINTS
value: '{"prior_claims_3yr": 1, "policy_tenure_years": -1}'
- name: SHADOW_MODE
value: "true"
resources:
requests:
memory: "4Gi"
cpu: "2000m"
limits:
memory: "8Gi"
cpu: "4000m"
minReplicas: 3
maxReplicas: 10
autoscaling:
metric: concurrency
targetValue: 50
---
# Canary routing for A/B testing
traffic:
- recipient: claims-fraud-decision
revisionName: claims-fraud-decision-v2-3
percent: 80
- recipient: claims-fraud-decision-v2-2-canary
revisionName: claims-fraud-decision-v2-2
percent: 20Team Size Key Deliverables
Requirements and schema design 2-3 weeks
1 architect, 1 product Decision schema, rule catalog, audit requirements
Feature pipeline 6-8 weeks
2 ML engineers Feature store setup, point-in-time correct training data, real-time feature computation
# decision_rules.yaml
rules:
- id: auto_pay_under_threshold
description: "Auto-approve claims under $2,500 with low fraud risk"
conditions:
- field: decision.confidence_score
operator: gte
value: 0.85
- field: claim.amount
operator: lte
value: 2500
- field: model.risk_score
operator: lte
value: 0.2
action:
type: auto_approve
timeout_seconds: 300
notification: "adjuster_team_a"
- id: mandatory_investigation
description: "Force investigation for high-value claims with any fraud signal"
conditions:
- field: claim.amount
operator: gt
value: 15000
- any_of:
- field: model.fraud_score
operator: gt
value: 0.6
- field: claim.adjuster_prior_claims
operator: gt
value: 3
action:
type: escalate
queue: "senior_investigators"
priority: high
requires_supervisor_approval: true
- id: regulatory_hold
description: "Apply state-specific regulatory holds"
conditions:
- field: claim.policy_state
operator: in
value: ["CA", "NY", "FL"]
- field: claim.claim_type
operator: eq
value: "personal_injury"
action:
type: hold
reason: "state_regulatory_review"
hold_duration_days: 14Model development 8-12 weeks
2 data scientists Trained models, validation reports, monotonic constraints, explainability outputs
Serving infrastructure 4-6 weeks
2 platform engineers KServe deployment, shadow mode, A/B testing, autoscaling
Rule engine 4-6 weeks
# decision_orchestrator.py
import temporalio
from temporalio import workflow
@workflow.defn
class DecisionWorkflow:
@workflow.run
async def run(self, decision: dict) -> dict:
claim_id = decision["claim_id"]
decision_type = decision["decision_type"]
# Log decision for audit trail
await workflow.execute_activity(
log_decision_audit,
args=[claim_id, decision],
start_to_close_timeout=timedelta(seconds=10)
)
if decision_type == "auto_approve":
# Payment initiation with idempotency key
payment_result = await workflow.execute_activity(
initiate_payment,
args=[claim_id, decision["amount"], decision["idempotency_key"]],
start_to_close_timeout=timedelta(seconds=30),
retry_policy=RetryPolicy(max_attempts=3)
)
await workflow.execute_activity(
send_adjuster_notification,
args=[claim_id, "approved", payment_result["transaction_id"]]
)
return {"status": "completed", "payment_id": payment_result["transaction_id"]}
elif decision_type == "escalate":
escalation = await workflow.execute_activity(
create_escalation_task,
args=[claim_id, decision["queue"], decision["priority"]]
)
return {"status": "escalated", "task_id": escalation["task_id"]}
elif decision_type == "hold":
hold_expiry = datetime.utcnow() + timedelta(days=decision["hold_duration_days"])
await workflow.execute_timer(hold_expiry - datetime.utcnow())
await workflow.execute_activity(
review_hold_status,
args=[claim_id, hold_expiry]
)
return {"status": "hold_reviewed"}
# Activity implementations
async def initiate_payment(claim_id: str, amount: float, idempotency_key: str) -> dict:
# Idempotency prevents duplicate payments
cache_key = f"payment:{idempotency_key}"
cached = await redis.get(cache_key)
if cached:
return json.loads(cached)
result = await payment_gateway.charge(claim_id, amount)
await redis.setex(cache_key, 3600, json.dumps(result))
return result1 backend engineer Rule configuration, conflict resolution, audit logging
Orchestration and workflow 4-6 weeks
2 backend engineers Temporal workflows, payment integration, notification system
Monitoring and governance 4 weeks
- 1 ML engineer, 1 DevOps Drift detection, model registry, compliance reporting
- Pilot deployment 8-12 weeks
- Full team Shadow mode validation, adjuster feedback, production rollout
- Annual operating cost: approximately $800K to $1.2M including cloud infrastructure, personnel, and third-party services. The biggest variable is personnel. Platform engineering is expensive. The cheapest mistake is underestimating the integration work between the decisioning platform and your existing claims systems.
Cloud costs scale with decision volume. At 50,000 decisions per day, expect $15,000 to $25,000 monthly for serving infrastructure, feature store, and workflow execution. At 500,000 decisions per day, these costs quadruple. Budget for scale from day one. The architecture doesn't change, but the instance sizes and throughput configurations do.
# drift_monitor.py
import numpy as np
from scipy import stats
def calculate_psi(expected: np.ndarray, actual: np.ndarray, bins=10) -> float:
"""Population Stability Index for feature drift detection"""
expected_bins = np.histogram(expected, bins=bins)[0] / len(expected)
actual_bins = np.histogram(actual, bins=bins)[0] / len(actual)
# Avoid division by zero
expected_bins = np.where(expected_bins == 0, 0.0001, expected_bins)
actual_bins = np.where(actual_bins == 0, 0.0001, actual_bins)
psi = np.sum((actual_bins - expected_bins) *
np.log(actual_bins / expected_bins))
return psi
def check_drift(feature_name: str, recent_data: np.ndarray,
baseline_data: np.ndarray) -> dict:
psi = calculate_psi(baseline_data, recent_data)
# Thresholds: <0.1 stable, 0.1-0.2 moderate, >0.2 significant
severity = "stable" if psi < 0.1 else ("moderate" if psi < 0.2 else "significant")
return {
"feature": feature_name,
"psi": round(psi, 4),
"severity": severity,
"timestamp": datetime.utcnow().isoformat(),
"baseline_mean": round(float(np.mean(baseline_data)), 4),
"recent_mean": round(float(np.mean(recent_data)), 4)
}
# Schedule daily drift checks
async def daily_drift_check():
features_to_check = [
"prior_claims_3yr",
"claim_amount",
"policy_tenure_years",
"distance_to_last_loss",
"adjuster_workload"
]
recent_features = await feature_store.get_features(
entity_id="all",
timestamp="now",
features=features_to_check
)
baseline_features = await feature_store.get_features(
entity_id="all",
timestamp="training_start_date",
features=features_to_check
)
drift_results = []
for feature in features_to_check:
result = check_drift(feature, recent_features[feature], baseline_features[feature])
if result["severity"] != "stable":
drift_results.append(result)
if drift_results:
await notify_team("drift_alert", drift_results)
await trigger_model_retraining_check(drift_results)Common failure modes and how to avoid them I've watched six decisioning platform projects fail or underdeliver. The reasons cluster into five patterns. Understanding these saves time and money.
Pattern 1: Building models before defining decisions. Teams fall in love with the ML. They train sophisticated models that predict well but don't produce actionable decisions. The model outputs probabilities. Adjusters need yes/no/maybe. Bridge this gap with clear decision categorization before model development starts.
Pattern 2: Ignoring the audit trail. Every decision must be reproducible. Feature values, model version, rule evaluation, and action taken. If you can't reconstruct a decision from six months ago, you don't have a compliant platform. Build the audit trail into the schema, not as an afterthought.
| Pattern 3: Underestimating feature engineering time. The feature pipeline typically takes 40% longer than estimated. Real insurance data is messy. Policy records don't align with claim records. Historical data has gaps. Build in buffer time and start feature development before model development. | Pattern 4: Shadow mode shortcutting. Skipping or shortening shadow mode is the fastest path to production problems. Run shadow mode for at least 30 days. Compare model decisions against human decisions without affecting outcomes. The discrepancies you find here are free. Discrepancies found in production are expensive. | Pattern 5: No rollback plan. When a model causes problems, you need to revert in minutes, not hours. Maintain parallel model versions in production. Keep the previous model's artifacts and configuration ready to deploy. Test rollback procedures monthly. A model that can't be rolled back is a liability, not an asset. | Compliance and regulatory considerations Insurance is regulated. Your decisioning platform operates under state department oversight, NAIC guidelines, and potentially ECOA and FCRA requirements depending on the decision types. Compliance isn't optional. It's a system requirement. |
|---|---|---|---|
| Key compliance requirements for decisioning platforms: Adverse action notices: If a model contributes to a denial, the policyholder must receive a notice explaining the decision. Your platform must generate these notices automatically with the specific factors considered. | Model interpretability: Regulations increasingly require explanations for automated decisions. SHAP values provide feature-level explanations but may not satisfy regulatory requirements for narrative explanations. Build both. Data fairness: Models cannot use protected class information directly. Indirect use through proxies is harder to detect. Audit your feature set for proxy variables correlated with protected attributes. | Documentation: Model cards, data sheets, and decision flow documentation are becoming standard requirements. Start maintaining these from day one. The National Association of Insurance Commissioners published model usage guidelines in 2023 that explicitly address automated decisioning systems. Your platform should comply with these guidelines even if your state hasn't adopted them yet. Expect adoption to accelerate. | What to build vs. what to buy Not everything should be custom-built. The decisioning platform has four components where buying beats building and four where building beats buying: |
| Buy: Feature store (Tecton, Feasteon, or internal platform team solution) | Model serving infrastructure (KServe, SageMaker, or equivalent) Workflow orchestration (Temporal Cloud, AWS Step Functions) | Audit logging and compliance reporting (built into most platforms above) Build: | Decision schema and rule configuration (domain-specific, no off-the-shelf fit) Feature engineering logic (unique to your data and use cases) |
| Model selection and tuning (your data, your problem, your answer) Integration with existing claims systems (proprietary, no vendor solution fits) | The biggest mistake I see is buying a decisioning platform vendor and then spending six months customizing it to your processes. You end up paying for both a vendor license and custom development. Either build the customization layer yourself or choose a platform that exposes its customization points clearly. | The path to production | Production deployment follows a staged rollout. Never flip a switch. Stage one runs in shadow mode for 30 days. Stage two runs parallel decisions for 60 days, comparing automated and human decisions without acting on either. Stage three is limited production for low-risk claim types only. Stage four is full production after adjuster feedback incorporates into the workflow. |
| Each stage has exit criteria. Shadow mode ends when model predictions correlate with human decisions at the same rate as the validation period. Parallel decisions end when the error rate between automated and human decisions falls below the threshold you defined in step one. Limited production ends when adjuster acceptance rate exceeds 80%. Full production requires a signed off from compliance and risk management. | The platform I described above replaced a manual claims process at a regional carrier in Q3 2024. Six months after full deployment, STP rates for auto physical damage claims rose from 23% to 67%. Average decision latency dropped from 4 hours to 47 seconds. Adjuster satisfaction scores improved because the platform handled routine decisions and flagged genuinely interesting claims. The implementation took eight months with a team of six. The ongoing maintenance requires two FTEs. | The difference between that result and a failed project was deciding on the decision schema before writing any code, running shadow mode long enough to catch calibration issues, and building the audit trail from day one. Those three choices prevented the failure modes I described above. They're also the three things I'd prioritize if building this platform again today. | What decision types are you planning to automate first? The answer determines whether this platform saves you time or creates new work. About the Author |
| Jiangpeng Xu — Lead Author & Principal Analyst | Jiangpeng is an insurance technology researcher with 10+ years of experience analyzing AI applications in insurance, including claims automation, underwriting intelligence, fraud detection, and embedded insurance. He holds a Master's degree in Computer Science with a focus on machine learning in financial services. | Was this article helpful? Comments. | |
Key Takeaways
- State Farm filed 116 AI patent applications in 2023 while Progressive processed over 2 million auto claims using its ML-based ImageClaim system since 2019.
- One carrier client lost $400,000 on a standalone machine learning project because it lacked workflow integration, human-in-the-loop checkpoints, and explainability features.
- The described event-driven platform handles roughly 50,000 decisions per day with an end-to-end latency target of under 500 milliseconds for automated decisions.
- Teams building feature stores with Tecton should budget for 40% of their initial expenses on configuration and maintenance during the first year of operation.
