Automating a cyber insurer's support queue: four out of five AI proposals shipped untouched
Coalition's first AI product, shipped in three months. It took a 1,500-ticket-a-month queue to 65% end-to-end automation at peak, then expanded into three more ticket types, and the pattern behind it runs in three including underwriting workbench and claims.
My Role: Lead designer, sole designer, discovery through launch and two years of expansion.
Coalition, Inc. ยท Cyber insurance ยท Launched late 2024
Coalition's customer success team was absorbing roughly 14,000 Zendesk tickets a month, growing 50% year over year against a headcount plan that was not. Loss runs were 10% of that queue: 1,500 tickets a month, five minutes and seven tool switches each. Mechanical enough to automate, and risky enough that getting it wrong meant a broker receiving an incorrect document.
65% of loss run requests automated end to end at peak, roughly 1,000 tickets a month
80% of proposed runbooks applied by agents with no edit at all, at launch
~235 customer success hours returned per month
3 teams reused the interaction pattern for their own AI features
Coalition's customer success team was absorbing roughly 14,000 Zendesk tickets a month, growing 50% year over year against a headcount plan that was not. Loss runs were 10% of that queue: 1,500 tickets a month, five minutes and seven tool switches each. Mechanical enough to automate, and risky enough that getting it wrong meant a broker receiving an incorrect document.
The hardest decision was not what the AI should do. It was what it should be allowed to do without asking. I designed Customer Success Copilot around a runbook: before acting, it shows the task it intends to perform, the data it extracted, the internal actions it plans to take, and the reply it drafted. The agent approves, edits, or ignores it. Nothing happens silently. Full autonomy is unlocked per scenario, behind eligibility criteria I defined with engineering, rather than granted to the system as a whole.
That structure is why the numbers held. At launch agents applied the proposal unedited roughly 80% of the time, which says the suggestions were good enough to use as-is rather than merely fast.
The real problem was not automation. It was whether to trust one specific proposal.
The assigned problem was "automate loss runs." The actual problem was narrower and harder to state: an agent needs to decide whether to trust a specific machine-generated proposal, in under a few seconds, without leaving the ticket, and the cost of trusting a wrong one is a broker receiving an incorrect document.
Every design decision below either serves that sentence or it does not.
We chose to let agents correct, not confirm, and that created everything after it
I proposed a confirm-only interface first. My engineering manager pushed back and she was right to: at the model accuracy we had, a confirm-only design has no graceful middle, and a failed runbook would have meant a manual ticket. So agents were dropped straight into an editable state instead. It cost vertical space and asked more of the agent on every ticket, and it carried the product through a period the model alone would not have.
That made verification the agent's job. On loss runs that was fine, because the thing to check was an account. On signature bundles it was not, because the thing to check was a signature buried in a long PDF, and the panel was 452 pixels wide with no guaranteed height.
The Zendesk panel in situ: the 452px column against the full window, with the confirm-only and edit-first states side by side and the vertical cost of each.
I fixed it in the PDF exporter, not the interface
The engineer I worked with brought me extraction results that were off. On long documents the model was confidently returning the wrong signature. His read was that we needed a fallback that put the PDF in front of the agent.
I did not think the first move was a fallback. I also owned PDF design at Coalition, so I owned how those documents exported and could change the input the model was reading rather than decorate the output it produced. We labelled the signature field explicitly at export, giving the extractor a named target instead of asking it to infer one. Extraction improved and the issue was not raised again.
Labelling fixed accuracy. It did not fix trust.
An agent still had no way to see where a value came from, and I was not willing to ship a product whose core ask was "believe this." I designed two fallbacks and killed both on space: a PDF pane that consumed the panel, and before it a pop-out for long broker emails that died because the customer success team would not close overlays once opened. Two of my own proposals died on the same 452 pixels, which is when I stopped treating the constraint as an obstacle and started treating it as the brief.
What shipped was hover. Any extracted value reveals the exact snippet of the source document it was lifted from, in place, on demand. Provenance became a property of the value rather than a destination you navigate to.
The trade-off was real. Hover is not discoverable and it makes verification optional rather than required. I gave up the guarantee that anyone would check. What I bought was that checking cost nothing, and a check that costs nothing gets done far more often than a check that costs a context switch.
The killed pop-out and the PDF pane side by side, both marked rejected, one line each on why. Two of my own ideas, same cause of death.
What the AI claims sits in a box. What the human does does not.
Everything the model concluded lives in a filled, collapsible grey card headed "Step 1: Verify Request Details." Everything the agent does sits on plain white. The verb is "Verify" rather than "Review," because review is passive and verify names an action with a right answer.
Confidence scores existed as ticket tags and I deliberately kept them out of the panel. A confidence score invites people to trust the machine's self-report instead of the evidence, and a model that is confidently wrong is exactly the failure mode we already had.
The containment study: the winning layout beside two rejected alternatives, annotated with what the grey card is doing.
Where I lost one: auto-selecting intent
The PM wanted intent and work type auto-selected rather than confirmed by the agent, because he was not confident the context would survive otherwise. I wanted confirmation as an explicit act. He was closer to the failure cases than I was and I took the call. It is the decision I am least settled on.
What would not have happened without me
The upstream PDF fix, because nobody else was looking outside the interface for the cause of an interface problem and I was the only person who owned both the exporter and the panel.
The pattern crossing into claims. I was also the designer on the claims workbench, which had the identical shape: extracted values from long invoices nobody would open a document to confirm. Claims had never used Copilot at all, so I had to demonstrate the product before I could argue for reusing anything from it. Labelled fields and hover to source now run in both verticals, and the broader runbook and containment model was picked up by three teams including underwriting workbench.
I also helped define the success criteria the program was measured on, which is not a thing designers are usually invited to do. It is also the part I got wrong.
The same hover pattern running on a claims invoice. Reuse, shown not asserted.
65% at peak, and what happened to the number when coverage expanded
65% of loss run requests automated end to end at peak, over 7,000 tickets, roughly 1,000 a month
10.88 minutes to 0.25 minutes average handle time on automated tickets, across the first seven months
~235 hours of customer success time returned per month
>98% LLM coverage identifying loss run requests
Loss run automation reached 65% at its peak. Across the two years since, the blended rate sits near 50% as Copilot expanded into bind requests, broker documents and quote changes, so the percentage fell while the volume it handled rose.
Measured against control groups: bind requests down 32% on one-touch tickets and 22% on multi-touch, signature bundles down 37.3% and 61.2%, quote changes down 27%.
From launch day, captured on video while the team used it for the first time: "Oh my goodness! That was so easy!"
Before and after: the seven-tool loss run path against the single panel, with the handle time figures on each.
The measurement instrument I designed did not work
I built the measurement framework alongside the product: accepted unchanged versus edited versus unused, plus a prompt asking agents their reasoning when they overrode the proposed intent or account. The theory was that corrections would accumulate into a labelled dataset.
The theory did not hold. When a runbook was imperfect, agents did not correct it, they ignored it, which was rational because correcting cost them time and returned them nothing. So "unused" conflated "the model was wrong" with "this was not worth my time." Meanwhile the correction prompt asked users of a time-saving tool to spend time generating our training data.
Product telemetry should measure adoption and behaviour, which it can do honestly because it costs the user nothing. Model evaluation needs a properly labelled dataset, funded as its own piece of work. Using the first as a proxy for the second was convenient and wrong. I would still keep the accept, edit and unused metric. I would just stop calling it a measure of model quality and call it what it is: a measure of whether people find the output worth using.
Designing for the model being wrong
The substance of this project is what happens when the model is wrong. The runbook exists so a wrong answer is visible before it is actioned. Hover-to-source exists so checking costs nothing. Confidence scores were withheld so agents verify from evidence. Eligibility criteria exist so autonomy expands deliberately rather than by default. And the signature fix is the clearest statement of it: when a model is wrong, the interface is usually the last place to fix it and the first place everyone looks.
Since this project I prototype daily in Figma, Cursor and Claude, including a sandbox package inside Coalition's production repository that runs full screens against real-shaped fixture data through the live provider tree and design system, with unit tests, so interaction decisions get tested before anything is wired. Design-authored pull requests ship against the production codebase.
Shamraiz Gul ยท