Feb 6, 2025
Feb 6, 2025
The following is a guest blog post from Adil Aijaz, CEO of Sam.
LLMs aren’t built with ngrok.
But the AI systems built on top of them are.
That’s because AI systems—whether RAG apps or AI agents—are inherently distributed. Like any distributed system, AI engineers building them need connectivity infrastructure to prototype new ideas, test code, and reliably manage traffic.
That’s where ngrok comes in.
In this post, I’ll explain how we use ngrok to build Sam, our AI Sales Engineer and Call Intelligence platform.
Before we get into the details, here’s a quick introduction to Sam.
Sales engineers help reps sell technical products by providing deep product expertise during calls. Sam does the same by providing expertise in real time, saving SEs time and ensuring customers get answers without delay.
Behind the scenes, multiple services work together to make this seamless experience possible:
This system has many moving parts. Running everything at once on a developer’s machine is impractical. That’s where ngrok makes local development feel like working in production.
For example, a developer working on the real-time transcription service can register an ngrok webhook with a staging bot server. This allows them to spin up a Zoom call, have a bot join, and send the audio signal to their transcription service running locally for development and testing.
Similarly, an LLM engineer working on question detection can register an ngrok webhook with the staging bot, which connects to the staging transcription service. This lets them receive staging transcriptions in their local environment so they can focus purely on iterating on the prompt to improve question detection.
Our engineers constantly iterate on prompts, but deploying a new prompt can be risky. If a new prompt performs worse, it creates a terrible experience for customers. To prevent that, we need a way to test a new prompt with production traffic—without impacting production itself.
Here’s how we do it:
First, LLM engineers at HeySam can register an ngrok endpoint with Sam’s production question-answering service and set a percentage of production traffic to forward to it using a feature flagging service. This pseudo-code will give you a sense of our approach:
1"""2Iterate through each shadow endpoint and if the flag service tells us to send traffic to the shadow3endpoint, send it. Catch all exceptions as we don't want to disrupt prod Q&A4 5question - the quesiton being answered by prod.6"""7try:8 for endpoint in shadow_endpoints:9 if feature_flag_service.evaluate('send_shadow_traffic', endpoint.percentage) == 'on':10 asyncio.run(send_question(endpoint.url, question))11catch Exception:12 log.error("...")Production forwards the desired percentage of real questions to shadow endpoints, allowing developers to test with live traffic on their local machine—without affecting production. The developer’s new prompt generates an answer and saves it locally alongside the original question ID.
Finally, we compare production and local answers using a comparison service that evaluates the quality of responses.
Building AI systems isn’t just about fine-tuning models; it’s about managing the complexity of distributed infrastructure. At HeySam, ngrok helps us bridge the gap between local development and production, making it seamless to prototype, test, and iterate on real-world traffic.