ngrok Introduces Network Traffic Inspection - Available directly from the dashboard | ngrok blog
By Russ Savage
March 27, 2024
By Russ Savage
March 27, 2024
A modern PCAP in the cloud
One of the big advantages of using a gateway between your application and the internet is observability. Being able to view the traffic going to your upstream API or service gives you a clear understanding of what’s going on in your network, allowing you to pinpoint and resolve issues with your application quickly and with minimal customer impact.
Today, we are excited to announce our new cloud based ngrok Traffic Inspector, now available in developer preview. It provides you and your team complete visibility into the HTTP traffic flowing through your ngrok account right from the dashboard.
The new Traffic Inspector is available to all accounts during the developer preview with a 3-day retention period. By default, only metadata is captured about your traffic, but Admins of the account can opt-in to full traffic event capture which will capture complete request and response traffic to your endpoints.
You can view the Traffic Inspector dev preview in your account right now by logging into the ngrok Dashboard and navigating to the Traffic Inspector page using the left nav. If you’re an Admin, you can enable full capture in your account settings page. Note that doing so will capture complete request and response traffic, including potentially sensitive data.
ngrok has always offered the ability to inspect the traffic routing to your upstream service using the ngrok agent local inspect interface, which is usually found at http://localhost:4040. Over the past few years, ngrok has expanded the number of agents to include SSH Reverse Tunnels (use ngrok through ssh), open source language specific SDKs (use ngrok inside your application with no external processes to manage), and a Kubernetes Operator (manage ngrok using native Ingress or GatewayClass objects).
Expanding the type of agents has unlocked new use cases for ngrok and our users, but it has come at the expense of losing that traffic inspection capability. We didn’t think adding this to each agent type made sense (or was even possible) since part of the value of ngrok is enjoying the same capabilities no matter where or how you run our agents.
With that in mind, we set out to build a new Traffic Inspector to solve this and a few other customer problems.
As mentioned previously, for anything other than the classic standalone ngrok agent, there is no local traffic inspector capability. We wanted to give our users the power of building gateway connectivity directly into their applications, but doing so came without the ability to observe traffic, limiting adoption for some of our users.
As ngrok is increasingly adopted for production usage, customers are deploying our agents to remote servers and their customer sites. Most of the time, those users disable the local inspect interface since it won’t be used and can interfere with other services in production.
During troubleshooting, it is difficult to get to those agents to enable the inspection capability. Doing so also requires a restart of the agent, which in production is usually not a great option.
Even if a customer can get to and enable the local inspection interface, if they take advantage of load balancing their traffic across multiple agents using edges, it is difficult or impossible to get complete visibility to the traffic routing through to your upstream service. In fact, the local inspect interface never fully supported displaying traffic from ngrok Edges, something we’ve wanted to fix for a long time.
Finally, we’ve heard repeatedly over the years many stories where our customers have had some sort of network issue, and the way they solved it was by temporarily spinning up an ngrok agent, routing traffic through it, and using the inspector to view that traffic. We are flattered, but as mentioned before, doing so usually involves connecting into a server remotely and setting up the agent, and finding a way to view the inspector information. It’s painful at best.
Elevating traffic inspection out of the individual agent and into the ngrok dashboard solves all of these problems.
Since we are collecting traffic directly from the ngrok endpoints themselves, we can show traffic from all endpoints and all agents, regardless of how they connect to the ngrok cloud. As we roll out new ways to connect, they will automatically show up as well. It also works the same across all versions of the ngrok agent as well as inside of your Kubernetes cluster.
The full capture of network traffic can be dynamically turned on and off without needing to restart your agents or remote into any of your infrastructure. This means that during an incident, you can toggle traffic inspection on and see the live requests hitting your service. This allows you to fix or block traffic quickly to mitigate customer impact during the incident.
Inspecting network traffic going to your upstream application is valuable during all parts of your software development life cycle.
When you are initially building your application, you can use the traffic inspector to view the requests hitting your application. This is especially useful when building a service to respond to external requests such as webhooks or callbacks. Also, as all developers know, many times, the documentation for these services is incomplete or out of date, and the only way you can know that something is going to work is to look at the specific requests routing to the endpoint. With Traffic Inspector, you can easily correlate requests that result in 4xx/5xx responses.
The Traffic Inspector is also useful for writing thorough tests for your application. Frequently, your tests need real requests to exercise specific code paths and verify the correctness of your service. The quickest way to grab these requests is usually inspecting the real traffic routing to your application. The Traffic Inspector allows you to grab the full request payloads that are hitting your application and add them to your code easily.
Production incidents are inevitable and the best way to mitigate them quickly is with the right tooling. When you are actively investigating an incident, the ngrok Traffic Inspector can be used to see exactly what traffic is going to your service so that you can respond appropriately.
You may want to take specific information from the request such as the client IP or a header value and create a new Policy for the endpoint to fix it. Other times you may need to fix and deploy a new version of your software. Using the captured request as a test case to make sure you’ve fixed the issue can be a real lifeline and accelerate resolution time.
During an active investigation, interesting requests can be shared quickly with your team or posted in incident investigation channels via slack, Microsoft Teams or your preferred communications platform. This brings everyone to the same page quickly and helps develop the best solution to an outage.
We’d love to hear from you to understand how you’re using this new capability with your team.
We are actively developing and improving the Traffic Inspector to ensure we provide the right observability to our customers. We are still missing a handful of capabilities that we know users love from the ngrok agent inspector, with the big one being the ability to replay a specific request to the upstream application. We are working on bringing Replay and Replay with Modifications to the traffic inspector in the coming weeks to make it simple to test applications that respond to webhooks or other 3rd party requests.
We know that filtering and searching are important to any observability tool, and we want to make sure our traffic inspector allows you to find the traffic you are looking for even if your endpoints are getting millions of hits a day. We will be bringing advanced filters and searching to the traffic inspector which will allow you to see what traffic is being matched in your policy expressions and use it to test new or updated expressions.
Finally, we know that 3 days of retention is not enough for everyone. We will add the ability to purchase longer retention periods as needed for your organization.
Along the way, we would love to incorporate your feedback into the process as well. During the developer preview, you can let us know what you think by emailing our product team.
We are excited to bring this new version of our Traffic Inspector to the ngrok dashboard and make it available to everyone as a developer preview. We believe this will unlock even more use cases for our users and provide developers with the tools they need to build applications and triage customer issues faster than ever before.
As we continue to add more capabilities to the traffic inspector, we look forward to hearing your feedback directly to the product team.