Growth was the problem, not a lack of effort
Vault Finance's support team wasn't understaffed or badly run. They were being buried by their own growth. Thousands of new users a day meant thousands of password resets, account questions, and transaction status checks, the kind of ticket that's trivial to answer but expensive to answer thousands of times over. Response times had gone from 15 minutes to over 4 hours. Hiring linearly with user growth wasn't financially sustainable, and it wouldn't have fixed the actual problem anyway: too much of the queue was repetitive work that shouldn't have needed a human in the first place.
The easy 80% and the dangerous 20%
Building an LLM that can answer support questions accurately isn't the hard part anymore. Vectorizing a knowledge base, API docs, and historical tickets, then wiring that into a retrieval pipeline with Vertex AI and LangChain, is a well-understood pattern. We indexed all of it and integrated the agent directly into Vault Finance's existing Zendesk instance so it could intercept tickets before a human ever saw them.
The hard part was deciding what the agent should never touch. This is a financial platform. An LLM confidently giving a wrong answer about an account balance, a failed transaction, or a security concern is worse than no automation at all, it erodes trust exactly where trust matters most. So the real engineering work went into the boundary: detecting when a query fell outside what the agent could resolve with confidence, and escalating it to a human with the full conversation context attached, not a cold handoff that makes the customer repeat themselves.
Designing for graceful handoff, not just automation
We tuned the agent to be conservative rather than clever. It resolves standard account inquiries, password resets, and status checks autonomously, the genuinely repetitive tier-1 volume that was drowning the team. Anything with ambiguity, anything touching a dispute, anything the retrieval confidence score didn't clear a threshold on, went straight to a human agent, pre-loaded with the ticket history so the escalation felt seamless instead of like starting over.
That conservatism is why this worked. A support AI that tries to handle everything erodes CSAT the first time it gets something wrong with confidence. One that knows its own boundaries earns trust by being reliably right about the narrow set of things it handles.
Results, and why the CSAT number mattered more than deflection
The agent deflected 65% of incoming tickets instantly and accurately. But the number we watched closest wasn't deflection, it was CSAT, because that's the number that tells you whether automation is actually serving customers or just moving the queue around. It landed at 92%, a record for the team, and average response time on the escalated, complex tickets that actually needed a human dropped to under 10 minutes, because agents were no longer spending their day on password resets.
The takeaway we carry into every AI automation project since: deflection rate is a vanity metric if you're not also measuring whether the humans who get escalated to are having a better experience, not a worse one. Build the boundary first. The automation is the easy part.
Want the full client story behind this?
Read the Case Study →
