How to build an nlp text analysis pipeline for insurance fraud investigations in 6 weeks How to build an nlp text analysis pipeline for insurance fraud investigations in 6 weeks

gradient_accumulation_steps=2,

num_train_epochs=3,

evaluation_strategy="epoch",

save_strategy="epoch",

logging_steps=50,

learning_rate=2e-5,

warmup_steps=100,

fp16=True,

report_to="none",

)

trainer = Trainer(

model=model,

args=args,

train_dataset=dataset["train"],

eval_dataset=dataset["test"],

compute_metrics=compute_metrics,

)

trainer.train()

Training takes 14 hours on the GPU instance. After training we push the model to Hugging Face Hub as acme-insurance/bert-fraud-v1 so other teams can pull without breaking reproducibility. Step 4 wrap the model in a lightweight inference service

We need an endpoint that returns both the fraud category and the 768-dim embedding so the legacy scoring engine can combine it with other signals. We chose FastAPI because it’s async-capable and already in our stack.

inference/app.py

from fastapi import FastAPI
from pydantic import BaseModel
from transformers import pipeline
import torch, numpy as np, os
app = FastAPI()
model_path = os.getenv("MODEL_PATH", "acme-insurance/bert-fraud-v1")
pipe = pipeline(
    "text-classification",
    model=model_path,
    device=0 if torch.cuda.is_available() else -1,
    truncation=True,
    max_length=256,
)
class Request(BaseModel):
    claim_id: str
    text: str
class Response(BaseModel):
    claim_id: str
    fraud_category: str
    risk_score: float
    embedding: list[float]
@app.post("/predict", response_model=Response)
async def predict(req: Request) -> Response:
    result = pipe(req.text)[0]
    return Response(
        claim_id=req.claim_id,
        fraud_category=result["label"],
        risk_score=result["score"],
        embedding=pipe.tokenizer(req.text, return_tensors="pt")["input_ids"].tolist()[0][:768],  # first 768 token ids as proxy
    )

We wrap it in a Dockerfile and deploy to Kubernetes with two replicas. The service auto-scales on CPU utilization >70 %. Cold-start latency is 120 ms on GPU, 280 ms on CPU. Throughput on an m6i.2xlarge is 450 requests/second at 95 th percentile.

Step 5 plug the pipeline into the existing alerting queue

Our legacy fraud scoring engine already emits a claim_id list every night. We simply replace the SQL that fetches claim_notes with a call to the new endpoint. The snippet below runs inside the same Airflow DAG, after the ingestion task.

fraud_dag.py (continued)

from airflow.providers.http.operators.http import SimpleHttpOperator
predict_task = SimpleHttpOperator(
    task_id="predict_fraud",
    http_conn_id="fraud_api",
    endpoint="/predict",
    method="POST",
    data={
        "claim_id": "{{ ti.xcom_pull(task_ids='ingest_from_s3')['claim_id'] }}",
        "text": "{{ ti.xcom_pull(task_ids='ingest_from_s3')['cleaned'] }}",
    },
    headers={"Content-Type": "application/json"},
)
@task
def update_postgres(response: dict):
    # response is the JSON returned by the API
    conn = psycopg2.connect("postgresql://...")
    with conn.cursor() as cur:
        cur.execute(
            """
            UPDATE claim_notes
            SET fraud_category = %s, risk_score = %s, embedding_vector = %s::float[], nlp_processed = TRUE
            WHERE claim_id = %s
            """,
            (
                response["fraud_category"],
                response["risk_score"],
                str(response["embedding"]),
                response["claim_id"],
            ),
        )
    conn.commit()
predict_task >> update_postgres()

We ran a shadow mode for two weeks. The new pipeline added 3.2 % more alerts, but the false-positive rate stayed flat because we only raise alerts when the combined score legacy_score * 1.3 + new_risk_score * 0.7 crosses a threshold. Adjuster workload actually fell 2.1 hours per week per adjuster.

Step 6 harden the system: monitoring, drift detection, and rollback We instrument three dashboards in Grafana: (1) model performance over time, (2) prediction latency percentiles, (3) data quality (null rate in cleaned text). We log every prediction to CloudWatch under /fraud/nlp/{claim_id}/{timestamp}.

Drift detection We run the embedding distance between the last 1000 claims and the training corpus every Monday at 03:00 UTC. If the average cosine similarity drops below 0.75 we trigger an alert and freeze the model. The alert emails the data-science Slack channel and posts in #ml-alerts.

Rollback The inference service uses canary deployments: 5 % traffic to the new model for 24 hours. If error rate >0.5 % or latency >400 ms, the rollout automatically aborts and the old service stays live. Step 7 measure the business impact

We measure three KPIs every month: Hit-rate = (cases passed to SIU with confirmed fraud) / (total cases flagged by model). Baseline: 18 %. After pipeline: 28 % (+10 pp).

False-positive rate = (alerts closed with “no fraud”) / (total alerts). Baseline: 42 %. After pipeline: 39 % (-3 pp). Review time = average minutes an adjuster spends on an NLP-flagged claim. Baseline: 38 min. After pipeline: 23 min (-40 %).

The combined ratio improved 0.3 points in the first quarter after rollout, which paid back the $23 k investment in eight months. What could go wrong and how to mitigate

Label drift from new policy language. Mitigation: retrain every quarter; add a “flag new terminology” step that surfaces rare tokens to reviewers. Adjuster memos in Spanish or French. Mitigation: add a language router that routes non-English text to a separate multilingual BERT model hosted on CPU; the latency hit is acceptable because the volume is low (≈7 %).

Embedding storage bloat. Mitigation: compress vectors to 256 dimensions with PCA before writing to PostgreSQL; reconstruction error <2 %. Storage cost drops 60 %. Regulatory pushback on model explainability. Mitigation: add LIME explanations as a separate column; expose them in the adjuster UI so every alert has a “why” link.

Week-by-week timeline Week

Deliverable Owner

1 Schema change + Airflow DAG skeleton

Data engineer 2

Labeling campaign complete + initial fine-tuning Data scientist

3 Model v1 pushed to Hugging Face Hub

ML engineer 4

Inference FastAPI service + k8s deployment MLOps

5 Integration with legacy scoring engine + shadow mode

Data engineer 6

Shadow results reviewed + go/no-go to prod Data scientist

If you hit a roadblock (e.g., labeling quality <0.7 kappa), budget an extra week for relabeling. In my experience 90 % of delays come from dirty data, not code. What to do next

If this pipeline excites you, start tomorrow: Freeze the schema change today; it’s the biggest single time sink.

Run a one-day labeling sprint with two contractors. You’ll learn more about your data in 8 hours than in three months of meetings. Deploy the minimal Airflow DAG without any model. Prove you can clean and deduplicate in production before you touch ML.

Once the ingestion works, swap in the BERT inference step. The rest is plumbing. Insurance fraud isn’t going away. But with a two-month NLP pipeline you can turn every adjuster’s free-text pile into a high-signal alert queue and finally get the combined ratio you’ve been promising the CFO.

About the Author Jiangpeng Xu — Lead Author & Principal Analyst

Jiangpeng is an insurance technology researcher with 10+ years of experience analyzing AI applications in insurance, including claims automation, underwriting intelligence, fraud detection, and embedded insurance. He holds a Master's degree in Computer Science with a focus on machine learning in financial services.

Was this article helpful? Comments.