What you type, what Claude asks, what you get

I read 31 files and the git history. Here is what I found:

• 25 model calls in src/agents/, each with a prompt file (27 prompt files, two of them shared house rules).
• A routing graph with 10 decision points, 4 places where agents run at the same time, and 7 places where work goes back to an earlier step.
• 10 places where the flow waits for a person, from the approvals queue and its config.
• 7 calls to outside systems and one rule-based check.
• A withdrawal handler that can stop a claim at any point.

How deep should we go? A quick audit is about 12 questions and 10 minutes and gives you the map and the obvious gaps. A deep audit is about 50 questions and 45 minutes and goes agent by agent through the riskiest ones, which is where weak agents get found.

Recommended: deep, because these agents decide payouts and write regulated letters.

From the code, these seven agents carry the most risk. I will grill them one at a time, in this order. The other 18 get a confirmation table at the end.

1. Compliance reviewer: its prompt asks it to review the letter and also return a corrected letter, and nothing checks what it returns.
2. Letter writer: it writes regulated letters, its prompt changed three times in six weeks, and no test references it.
3. Damage estimator: it drives the payout, runs on Opus 5.5 for an extraction job, and has no fallback in the code.
4. Settlement calculator: it drives the payout, and its reviewer uses the same model family.
5. Exclusions finder: downstream code parses its output as JSON, but there is no schema and nothing validates it.
6. Claim file summarizer: it runs on GPT-6 Astra with high thinking for a summary.
7. Coverage analyst: a wrong coverage call is the most expensive mistake in the process.

I grouped the steps into six roles: Intake, Coverage, Investigation, Settlement, Customer comms and Finance. Does that match how your team talks about the process?

Recommended: yes.

Your code sets timeouts but no target times. Should each agent's SLA be about half its timeout? The damage estimator has a 120-second timeout, so its SLA would be 60 to 70 seconds.

Recommended: yes, and tell me any you want tighter.

Your approvals config escalates denial sign-off after 24 hours, SIU after 48 hours and adjuster review after 24 hours. For denial and SIU the work stays with the same person and a notice goes out. For adjuster review it is reassigned. Is that right?

Recommended: yes.

What does a bad estimate from the damage estimator look like? Give me one you have actually seen.

Recommended: the last estimate an adjuster corrected.

How was that caught?

Adjuster review only happens on claims over $5,000. Under that, the estimate goes straight to the settlement calculator, and the settlement reviewer's prompt compares the total against the policy limits, not the line items. So on most claims a wrong line item reaches the payout unchecked. Is that right?

Which inputs make it struggle?

Your code sends the whole invoice in one call with max_tokens: 2500. A twelve-page invoice can need more than 2,500 tokens of line items, so the list gets cut off. Has it ever been tested on a long invoice?

Why Opus 5.5? This is an extraction job.

I'll record that as unanswered. What happens when it fails or times out?

So there is no fallback: no smaller model and no hand-off to a person. Should a failed estimate go to an adjuster instead of stopping the claim?

Recommended: yes.

How many claims with known-correct estimates could you give me as golden inputs?

Damage estimator: readiness 55, at risk. A wrong line item reaches the policyholder without a full check, it runs a premium model on an extraction job, and it has no fallback today. The adjuster hand-off goes on the fix list. Test to add: five long invoices with known line-item totals.

What does a bad letter look like?

When does that happen?

How do you test a new version of its prompt before it goes live?

Git shows the prompt changed three times in the last six weeks, and nothing in tests/ references it. So every version went live untested on a regulated letter. How many letters with known-correct disclosures could you give me?

Letter writer: readiness 58, at risk. Its prompts ship untested, its output reaches the policyholder, and it has no golden inputs. The disclosure check and the compliance reviewer catch some mistakes, but the next untested prompt change is the risk. Test to add: five letters with two or more adjustments in three states.

Its prompt asks it to review the letter and to return the corrected letter in full. Is it a reviewer or an editor?

When it edits, who checks its edits?

So wording legal never approved can reach a policyholder. Has that happened?

The prompt also asks it to check word count, headings and format. Code can check those exactly and for free. Who owns this prompt?

Compliance reviewer: readiness 19, at risk, the lowest in the system. It does two jobs, nothing checks its edits, its prompts ship untested, and nobody owns it. Fix list: return fixes instead of rewriting, move the format checks to code, and assign an owner.

Health needs a record of every claim, and your agents do not write one yet. Where does the claims service run in production, and where do its logs go?

Then I will add the harness recorder: one Python file with no dependencies. It writes one JSON line per event to stdout, which CloudWatch already collects. It records which agent ran, the model, token counts, timing, status, which way each decision went and each review verdict. It never records claim text, prompts or letters, and nothing goes to Juggernaut. Add it?

Recommended: yes.

I instrumented three places: src/agents/base.py for every agent call and its token usage, src/graph.py for every routing decision and review verdict, and src/approvals/queue.py for when a person picks up and finishes a task. The run id is your internal claim id, which is opaque. Next question: can you read the claims service logs in CloudWatch yourself?

Then the events have to reach you through them. Because the events hold metadata only, no claim text or customer details, this is usually an easy approval. From simplest to most convenient:

1. They export the harness events when you ask and send you the file.
2. They copy the harness events every day to a bucket you can read.
3. They give you read-only access to the filtered events in CloudWatch.

Recommended: the daily copy to a bucket, so health is one command for you and no extra work for them after setup.

Done: .harness/request-for-log-owner.md says what to copy (lines where workflow_id is claims-settlement), how often, who needs read access, and shows a sample line so they can see there is no customer data in it. One thing to be clear about: I cannot reach CloudWatch, the bucket or any of your servers. Getting the logs is always a manual step you do. Whenever you want a health check, download the files into .harness/runs/ with your read-only access, then run /juggernaut-harness:health. The command is aws s3 sync s3://claims-harness-events .harness/runs, saved in .harness/config.yaml so you do not have to remember it.

The recording works. 17 of 25 agents reported, every model call with token counts. The other 8 sit on paths this claim did not take: denial, SIU, translation, reminder, dispute, payment exception and withdrawal. Health will have data from the day this ships to production.

I cannot reach CloudWatch, the bucket or any of your servers, so I need the logs on this machine first. Download the last 30 days into .harness/runs/ with your read-only access, using the command saved in .harness/config.yaml: aws s3 sync s3://claims-harness-events .harness/runs. If you cannot run it, ask the platform team for the files, or drop them here. Tell me when they are in place.

I checked the files: 414 claims and 55,083 events over 30 days, with all 25 agents reporting. Three of 25 agents are below the threshold.

• Damage estimator, 53. It is timing out: p95 latency is 231 seconds against its 70-second SLA, and 32% of attempts are retried. It breaks its output schema on 4.7% of calls.
• Letter writer, 54. It fell from 96 after prompt v5 went live on October 5. The compliance check now sends its letters back on 79% of decisions, up from 21%.
• Settlement calculator, 59. It fell from 91 after prompt v3 went live on October 12. Its reviewer now passes only 61% of its payouts.

The risk register flagged all of these on day one. The damage estimator and the letter writer were at risk. The settlement calculator was on watch for thin testing, and its v3 shipped without the golden claims the register asked for. The compliance reviewer is rewriting letters, the exclusions finder breaks its schema on 8.4% of calls, and the claim file summarizer is 25% of all agent spend.

The top findings, with estimated savings per claim:

1. Claim file summarizer: a summary step on GPT-6 Astra. Move it to a fast model. About $0.084 per claim, plus $0.017 from lowering its thinking.
2. Letter writer: the compliance loop sends it back on 79% of decisions since prompt v5. Fix the letter writer before the reviewer, since every pass reruns both. About $0.048 per claim.
3. Compliance reviewer: it rewrites the whole letter instead of returning fixes. About $0.027 per claim.
4. Damage estimator: timing out on Opus 5.5 for an extraction job. Test Sonnet 5, which saves about $0.019 per claim.
5. Settlement calculator: worse since prompt v3. Run the golden claims against v2 and v3 and roll back if v3 loses.

Every model or prompt change needs the measured test first: rerun three to five golden claims on the new setting and compare pass rates, schema violations and a human read.

v4 passed compliance on 5 of 5 golden claims and v5 on 2 of 5. I recommend rolling back to v4. Want me to change the prompt reference in harness.yaml and the agent code?

Quick or deep? Deep grills each agent before it exists, which is where the design gets its checks.

Recommended: deep, because these agents will decide payouts.

What should come out the other end, and how will you know it is good?

Recommended: an accepted settlement paid within 2 business days, or a clean denial or fraud referral, with every letter passing compliance.

Walk me through the steps a strong claims team follows, in order, and who owns each.

Which of those should an AI agent do, and where must a person approve?

Recommended: agents read, classify, extract, estimate, score, calculate, write and review; people approve denials, SIU cases, payouts over the adjuster's authority and anything the policyholder disputes.

Settlement calculator. How will you know a payout is wrong before the policyholder does?

And claims under $5,000, which you said are two thirds of volume?

Then I recommend a settlement reviewer agent on every payout that checks line items against the estimate, from a different model family than the calculator, with a retry loop back to the calculator when it rejects.

Recommended: add it.

How will you test a change to the calculator's prompt before it ships?

Pick three to five past claims with known-correct payouts, including one replacement-cost policy and one with a prior payment. They run before every prompt change.

The import found 4 agents, 2 decisions, 1 human approval and 1 error path. The workflow cannot tell me who owns each step or what a bad output looks like, so I will grill the drafter and the reviewer. First: what does a bad reply from Draft reply look like?

Recommended: the last reply a support lead had to rewrite.

Policy review is supposed to catch that, but it runs on Claude Haiku, the same model family as the drafter, so the two tend to miss the same things. How do you test a change to the drafter's system message before it goes live?

Then every edit goes live untested on customer email. I'll record that, and the test to add is five past tickets where the right answer is no refund.

Draft reply. Its health fell from 83 to 63 after its system message was edited on October 8. Since then the policy check sends drafts back on 34% of decisions, and only 46% pass on the first try. It is also 86% of agent spend and has never been compared against a cheaper model. Run the five no-refund tickets against the old and new system message and roll back if the new one loses.