Debugging Latency in a Voice Agent
Our AI receptionist sometimes went silent for almost three seconds when a caller asked for someone. This is why, and how we fixed it.
How long would you sit in silence on a phone call before deciding the other side hung up?
Sometimes a caller would say “I’m looking for John” and Grace would go quiet for 2.7 seconds or longer, long enough for the caller to wonder if the agent hung up. Then Grace would answer “Sure, can I please have your name and reason for calling?”
Grace is the AI receptionist I work on at Lighthouse. We handle phone intake for personal injury firms. Grace answers around the clock, collects the caller’s details, screens them against the firm’s criteria, and connects the qualified ones to the team.
So, what happened?
When a caller asks for someone, Grace needs to know which staff handle the caller’s matter and who’s available during the firm’s business hours. It asked our backend for that list right after the caller said who they were looking for, and it didn’t answer until the list came back.
If a firm’s agent didn’t receive a call for some time, the services behind that lookup would have to cold start. What does that mean? Well, our function had to boot up, the database had to wake up, the auth provider had to pull the firm’s staff list, and the answer still had to travel back to the voice platform.
We measured the tool call end to end at ~2.7 seconds when it had to cold start. That’s why the delay happened on some calls and not others.
How we fixed it
Before Grace says hello, the voice platform already calls our backend to find out which firm owns the number and to pass along the caller ID. We added the contact list to that request and loaded it in parallel with everything else.
Now the call carries the list from the start. When a caller asks for someone, Grace checks the list it already has, with no network request. The cold starts still happen, but they just happen before the greeting now. The pause after a caller asks for someone dropped to around 300 ms, sometimes less.
Why didn’t we just add filler words when the agent was waiting on tool call responses? Because it degrades the caller’s experience.
We accepted two tradeoffs:
- If the lookup fails or runs past its time limit, we return an empty list and Grace follows its existing process for when nobody is available.
- The list stays the same for the whole call, so a call that starts just before closing time still uses whoever was available when it began. We didn’t add a second lookup mid-conversation because that lookup would introduce the delay we were removing.
Then the first version broke our test calls.
Calls started from the dashboard and from scripts skip the request that runs before the greeting, so they had no list. Every transfer on a test call went down the “nobody available” path. Loading the same list when we create a test call fixed it.
Side effects of the fix
With the pause gone, Grace started talking over people.
Callers pause while they remember a name or a date. Grace treated some of those pauses as the end of their turn and started answering.
Two settings control this. Endpointing is how long the system waits through silence before deciding the caller has finished. Responsiveness is how eagerly the agent replies once it makes that decision.
| Setting | Before | After |
|---|---|---|
| Endpointing | 200 ms | 300 ms |
| Responsiveness | 0.9 | 0.85 |
The delay went from 2.7 seconds or longer to around 300 ms. Then we added 100 ms back to endpointing, which reduces the agent’s eagerness to respond and, as a side effect, improves accuracy.
What changed for callers
Loading the contact list before the greeting cut the wait after a caller asks for someone from 2.7 seconds or longer to around 300 ms. That worked because the list was small and could stay fixed for the call.
Listening to calls then showed where Grace was cutting people off. We increased endpointing from 200 ms to 300 ms and reduced responsiveness from 0.9 to 0.85 to give callers time to finish.
If you run a personal injury firm, check out Lighthouse Legal, the best AI intake for law firms. It’s built for personal injury first, and we iterate on it daily. This fix is one example. Grace answers your intake line around the clock, screens each caller against your firm’s criteria, and connects qualified cases to your team. Firms use it to cover nights and weekends, to sit in front of an answering service, or to take overflow off an in-house team. There’s more on the Lighthouse Legal project page.