I run a 40-person claims team at a specialty P&C insurer. In the last 18 months we processed 14,200 auto and property claims by hand. That’s 27 minutes per claim on average, but the real cost is the rework: 14% of initial estimates get revised upward after we find a receipt or photo we missed the first time. The adjuster who caught the error is usually the one who already knows the policy and the claimant. We can’t clone those humans, but we can give every adjuster the same second set of eyes.
Over six weeks we built a lightweight NLP pipeline that reads the first notice of loss (FNOL), the adjuster’s notes, repair estimates, and supplementary documents, then surfaces the top five discrepancies between what the claimant said and what the paperwork later shows. The model runs in a 16-core VM with 64 GB RAM and returns a JSON alert in under two seconds. After a three-month pilot on 2,100 claims it reduced rework spend by $198,000 and cut cycle time by 3.2 days. Below is the step-by-step build we wish we’d had.
Perspective: I’m the claims adjuster-turned-CTO. This guide prioritizes speed-to-value over academic purity. We’re trading a small uptick in false positives for a reduction in catastrophic rework. What you need before you start
Resources and budget People: 1 data engineer (50% FTE), 1 NLP specialist (25% FTE), 1 claims SME (ad-hoc review).
Compute: AWS m6i.4xlarge (16 vCPU, 64 GB) or equivalent on-prem. Cost: ≈ $0.68/hr on-demand. Storage: 5 TB S3 bucket in the same region as the VM. Lifecycle policy to Glacier after 90 days.
Data: 10,000 anonymized claims from the last two years (FNOL text, adjuster notes, estimates, photos). Tooling: Python 3.11, spaCy 3.7, Hugging Face transformers 4.37, scikit-learn 1.4, FastAPI 0.110.
- Time: Two weeks for the MVP, four weeks for calibration and A/B deployment. Data governance checklist
- Verify HIPAA/PHI masking: claims file numbers must be hashed before ingestion. Obtain signed data-use agreements from every adjuster who reviews sample labels.
- Store raw documents in S3 with AES-256 and SSE-KMS; restrict bucket to IAM roles with MFA. Step 1: Assemble a labeled discrepancy corpus
- 1.1 Define the discrepancy taxonomy We started with the five most costly rework patterns:
- Price drift: the repair estimate jumps by ≥15% between two submissions. Missing item: a receipt or photo exists but never made it into the estimate.
- Mismatched VIN: the VIN in the FNOL does not match the one on the repair invoice. Date mismatch: date of loss vs. date of repair differs by >30 days.
Policy exclusion: the damaged part is explicitly excluded in the policy schedule. 1.2 Build the annotation tool
- We forked Label Studio 1.7 and created a single-question interface: Field
- Type Example
- Validation Claim ID
Text (hashed) a1b2c3…
Required, regex hashed Discrepancy type
Radio Price drift
- Single selection Supporting doc IDs
- Multi-select fnol_001, adj_note_045, estimate_089
- At least one Severity
- Scale 1–5 4
- Required Adjuster comment
Text area Customer mentioned prior damage
Optional
| We gave 12 senior adjusters 100 claims each (≈ 30 minutes per batch). In two weeks we collected 3,142 labeled discrepancies out of ~10,000 claims. The inter-annotator agreement (Fleiss’ κ) was 0.79 for price drift and 0.83 for missing item, so we stopped after the second round. | 1.3 Export and split Export to JSONL: | Split 70/15/15 into train, validation, and holdout. Holdout set locked and never used until final evaluation. Step 2: Extract structured text from unstructured documents | 2.1 OCR pipeline Our FNOLs arrive as PDFs, PNGs, or DOCX. We use AWS Textract 3.1 with the “DetectDocumentText” API. Config: |
|---|---|---|---|
| Parameter Value | Note FeatureTypes | ["TABLES","FORMS"] Preserve line items in repair estimates. | S3Bucket claims-raw |
| Same region as compute. OutputS3Bucket | claims-processed Compressed JSON. | MaxItems 1000 | Parallelize with 8 workers. Latency: 1.2 s per page on m6i.4xlarge. Cost: $0.0015 per page. |
| 2.2 Normalize and merge Build a simple FastAPI micro-service that: | Pulls raw JSON from S3. Strips hyphenated line breaks. | Replaces “O” with “0” and “l” with “1” in VINs. Merges multi-page documents by claim_id. | Step 3: Train a transformer-based discrepancy classifier 3.1 Model choice |
| We benchmarked three models on the holdout set: Model | Size F1 (macro) | Inference (ms) BERT-base-uncased | 110 M 0.81 |
| 18 RoBERTa-large | 355 M 0.87 | 35 DistilRoBERTa-base | 82 M 0.85 |
12 We picked DistilRoBERTa-base and fine-tuned it on our 3,142 samples for 15 epochs. Batch size 16, learning rate 2e-5, warmup 0.1, max length 512.
3.2 Training script (single GPU) 3.3 Quantization and export
We quantized to int8 for deployment: Model size dropped from 430 MB to 110 MB. Inference latency on CPU: 12 ms per claim.
<code>
{"claim_id":"a1b2c3...","text":"2023 Honda Accord EX, 4 door, silver...","discrepancy":"price_drift","doc_ids":["fnol_001","estimate_002"],"severity":4}
{"claim_id":"d4e5f6...","text":"Roof damage noted, no receipt attached...","discrepancy":"missing_item","doc_ids":["adj_note_077"],"severity":3}
</code>
Step 4: Build the discrepancy extraction service 4.1 FastAPI endpoint
4.2 Containerize and deploy Dockerfile:
Push to Amazon ECR. Deploy as an ECS Fargate service with 2 vCPU, 4 GB memory. Auto-scaling policy: 70% CPU target. Cost: $36/month for 1000 claims/day. 4.3 Real-time alerting
We attach an EventBridge rule that triggers the service whenever a new FNOL lands in S3. The JSON alert goes to an internal Slack webhook and to the adjuster’s queue: Step 5: Calibrate precision vs. recall
| 5.1 Adjust the decision threshold On the validation set we swept the threshold from 0.1 to 0.9 and plotted precision-recall: | We picked 0.65 as the operating point: 89% precision, 76% recall. That means 11 false positives per 100 claims but only 24% of real discrepancies slip through. 5.2 Add rule-based fallbacks | We layered a lightweight spaCy pipeline for the two highest-precision patterns: VIN mismatch: spaCy matcher rule on 17-digit alphanumeric pattern; exact match against our vehicle database. |
|---|---|---|
| Date mismatch: regex on ISO dates; compare loss date vs. repair date. These rules catch 8% of discrepancies the transformer misses and run in <1 ms, so we kept them inline. | Step 6: Run the pilot and measure impact 6.1 A/B setup | We split 2,100 claims 50/50: Control: adjuster workflow unchanged. |
| Treatment: adjuster sees the NLP alert in the queue. 6.2 Metrics dashboard | Metric Control | Treatment Change |
| Initial estimate accuracy 85% | 91% +6 pp | Reopen rate (30 days) 14.1% |
| 8.3% -58% | Average cycle time 27.3 days | 24.1 days -3.2 days |
Adjuster overtime hours / week 8.2
5.7 -30%
Net rework spend $312k
- $114k -$198k
- ROI: 4.3× in the first quarter. Break-even at 4,000 claims processed. Step 7: Hardening and governance
- 7.1 Model monitoring We log every prediction to CloudWatch with:
- prediction score and top-5 labels latency histogram
<code>
import boto3, json, re
s3 = boto3.client('s3')
bucket = 'claims-processed'
def normalize(text):
text = re.sub(r'(?<=\w)-(?=\w)', '', text) # hyphens
text = re.sub(r'[Oo]', '0', text)
text = re.sub(r'[lI]', '1', text)
return text
def merge_pages(claim_id):
pages = s3.list_objects_v2(Bucket=bucket, Prefix=claim_id)['Contents']
docs = []
for p in pages:
obj = s3.get_object(Bucket=bucket, Key=p['Key'])
docs.append(json.loads(obj['Body'].read()))
merged = {'claim_id': claim_id, 'text': ' '.join(d['text'] for d in docs)}
return merged
</code>
document hash for replayability We set a CloudWatch alarm for latency >100 ms or error rate >1%.
7.2 Bias audit We ran a subgroup analysis by claim severity (low/medium/high) and by ZIP code median income. No subgroup fell below 82% F1, so we proceeded. If we see drift we’ll retrain on the next quarterly batch.
7.3 Retraining pipeline Every Sunday at 02:00 UTC an AWS Step Functions workflow:
| Pulls the last 30 days of new labels from Label Studio. Merges with historical training set. | Trains for 3 epochs with early stopping (patience=2). Runs a 5-fold cross-validation. | If macro-F1 improves >1% or drops <2%, pushes new model to ECS and updates the inference endpoint. 7.4 Documentation and runbook | Model card in Confluence: dataset size, limitations, subgroup performance. Incident runbook: how to roll back if the new model triggers >50 false positives in a day. |
|---|---|---|---|
| Adjuster FAQ: “Why did the system flag my claim?” with concrete examples. What didn’t work—and what to skip | Discarded ideas Full-text search on raw documents: 23% of discrepancies hide in a one-line VIN or a receipt number. Regex and entity matching alone were insufficient. | Fine-tuning proprietary LLMs: The 7B parameter model gave marginal F1 gain (+1 pp) but tripled latency and cost. We stuck with DistilRoBERTa. Deploying on GPU: CPU inference at 12 ms per claim is fast enough; the GPU node idled. | Adding sentiment analysis: adjuster tone rarely correlates with actual discrepancies. We cut the feature. Four-week post-mortem checklist |
| Task Owner | Due Status | Write model card v2 Data engineer | 2024-06-15 Open |
| Update adjuster FAQ with new discrepancy examples Claims SME | 2024-06-20 Open | Schedule bias audit for Q3 Compliance | 2024-07-01 Open |
Evaluate cost vs. benefit of adding photos to the pipeline CTO
2024-07-15 Open
<code>
from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments
import torch, json, pandas as pd
model_name = "distilroberta-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=5)
train_df = pd.read_json("train.jsonl", lines=True)
train_encodings = tokenizer(train_df['text'].tolist(), truncation=True, padding=True, max_length=512)
class ClaimsDataset(torch.utils.data.Dataset):
def __init__(self, encodings, labels):
self.encodings = encodings
self.labels = labels
def __getitem__(self, idx):
item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
item['labels'] = torch.tensor(self.labels[idx])
return item
def __len__(self):
return len(self.labels)
train_dataset = ClaimsDataset(train_encodings, train_df['label'].tolist())
training_args = TrainingArguments(
output_dir='./results',
num_train_epochs=15,
per_device_train_batch_size=16,
logging_steps=50,
save_steps=500,
evaluation_strategy="epoch",
fp16=True,
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=ClaimsDataset(tokenizer(train_df['text'].tolist(), truncation=True, padding=True), train_df['label'].tolist())
)
trainer.train()
model.save_pretrained("./claims_distilroberta_v1")
tokenizer.save_pretrained("./claims_distilroberta_v1")
</code>
The next gap—and how to close it
Our current model only reads text. One-third of rework starts with a photo of the damaged vehicle or property. The next sprint is to add a vision transformer that flags mismatched parts between the FNOL description and the photo. With the NLP pipeline already in place, we can reuse the same labeling interface and the same deployment topology—just swap in a ViT encoder and retrain on 10,000 labeled images. The incremental cost is another 16-core VM and a 6-hour labeling sprint.
<code>
from transformers import AutoModelForSequenceClassification
import torch
model = AutoModelForSequenceClassification.from_pretrained("./claims_distilroberta_v1")
quantized = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)
quantized.save_pretrained("./claims_distilroberta_int8")
</code>
If you try this build, start with the discrepancy taxonomy that drives rework dollars in your book. Everything else follows. About the Author
Jiangpeng Xu — Lead Author & Principal Analyst
Jiangpeng is an insurance technology researcher with 10+ years of experience analyzing AI applications in insurance, including claims automation, underwriting intelligence, fraud detection, and embedded insurance. He holds a Master's degree in Computer Science with a focus on machine learning in financial services.
<code>
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import torch, json, boto3
from transformers import AutoTokenizer, AutoModelForSequenceClassification
app = FastAPI()
device = torch.device("cpu")
tokenizer = AutoTokenizer.from_pretrained("./claims_distilroberta_int8")
model = AutoModelForSequenceClassification.from_pretrained("./claims_distilroberta_int8")
model.to(device); model.eval()
class ClaimPayload(BaseModel):
claim_id: str
text: str
@app.post("/predict")
async def predict(payload: ClaimPayload):
inputs = tokenizer(payload.text, return_tensors="pt", truncation=True, max_length=512).to(device)
with torch.no_grad():
logits = model(**inputs).logits
probs = torch.nn.functional.softmax(logits, dim=-1)
top5 = torch.topk(probs, 5)
return {
"claim_id": payload.claim_id,
"predictions": [{"label": i.item(), "score": p.item()} for i, p in zip(top5.indices[0], top5.values[0])]
}
</code>
Was this article helpful? Comments.
<code> FROM python:3.11-slim WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt COPY ./claims_distilroberta_int8 ./model COPY main.py . EXPOSE 8000 CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"] </code>
<code>
import boto3, json, os
client = boto3.client('events')
target_id = 'nlp-discrepancy-target'
response = client.put_targets(
Rule='fnol-arrival',
Targets=[{
'Id': target_id,
'Arn': 'arn:aws:lambda:us-east-1:123456789:function:discrepancy-alert',
'InputTransformer': {
'InputTemplate': '{"claim_id":,"text":""}'
}
}]
)
</code>
