NOTE: You can now take course by the author with video and example traces on Wireshark. Check this post for more details.

Author’s Note: This is the last part in a six-part series about finding and solving many networking anomalies using the Wireshark network protocol analyzer. If you are new to the series, you can find part 1 here, and the whole series here.
It may seem surprising in this age of ubiquitous Gigabit Ethernet connections, but many of the calls in Enterprise networking still revolve around network delay problems. Slow access to websites, slow logons, slow email, all of these issues can potentially be related to network delays. Unfortunately, tracking down the source of network delays can be very frustrating at times. With the right filtering, however, you can eliminate much of this frustration and drill down to the source.
In some cases, the delays will be easy to spot, showing up as a per-packet delay in the traces. In others, the per-packet latency may be very small, but the upper layer latency may be huge. The simplest check, however, is typically per packet latency. This can be done easily in Wireshark by filtering down to communications between the two problematic systems. Once this is done, ensure that your time display format is set to ‘Seconds since previous packet’ in the view menu, and click on the ‘Time’ column heading to sort by time. This will show you your longest delays between packets. If you have many packets with very long delays (0.2 seconds or longer), re-sort by frame number for several of these and examine the traffic. The type of traffic will determine if the delay is a problem. For instance, a long delay on a HTTP keep-alive is not a problem, but a long delay between an HTTP Get and a HTTP 200 response is.

Note: Make sure, when looking for delay issues, that the trace was captured with a minimal amount of capture filtering. The default setting of filtering out RDP is fine (unless you are getting RDP delays, of course), but any more than that is questionable. The reasoning for this is that if you are filtering out a large chunk of the traffic, you are in effect creating gaps in the capture, which invalidates the technique discussed above.

However, once you find the delay, you may still need to dig further to locate the source. For example, if you right click on an HTTP packet in Wireshark and choose the option ‘Follow TCP stream’, Wireshark will filter based on the source and destination sockets, giving you a good summary of the TCP conversation. An example of a delayed conversation is shown below:

Looking at this conversation, you may correctly surmise that the page in question takes a long time to retrieve (around 1.7 seconds), but we still do not truly know the source. Even worse, since an HTTP page can include multiple references to additional files (such as images), which can extend over multiple TCP sessions, looking at a single stream does not necessarily give us a true appreciation of the delay. To see the full delay, we must filter just on the two IP addresses in the conversation along with the protocol (HTTP), like so:
(ip.addr==160.207.226.48 and ip.addr==160.207.12.41) and http
With this filter, we can look for each HTTP Get request and response, and determine when the last request for the page is sent, and when the response for that request is returned. We find that the last request is for the file crossbullet.gif in frame 3962, which is returned in frame 4285. We know this is the last request because the Get request for default.aspx (and all the Get requests related to this page) repeats (indicating a new page request) after this Get request. Now that we know where the conversation ends, we can filter to just this HTTP conversation by using the following filter:
(ip.addr eq 160.207.226.48 and ip.addr eq 160.207.12.41) and http and frame.number <= 4285
With this filter applied, using the Summary command under the statistics menu gives us the following information:

As the summary shows, there is a 28.5 second delay in retrieving the page. This delay is huge, much worse than we previously anticipated, so now we need to dig in further to determine why. Another Wireshark filter can help here:
(ip.addr eq 160.207.226.48 and ip.addr eq 160.207.12.41) and (http.request or http.response) and frame.number <= 4285
This filter is essentially filtering to all traffic between the two hosts that is also an HTTP request or response packet and has a frame number less than 4285. This filters the HTTP continuation packets out of the trace, dropping the trace from 689 frames to 182 frames. However, we can break it down further and look for just the high delay packets by adding frame.time_delta to the filter, like so:
(ip.addr eq 160.207.226.48 and ip.addr eq 160.207.12.41) and (http.request or http.response) and frame.number <= 4285 and frame.time_delta >= 0.2
This filter is the same as the previous filter, but now we are adding the condition that all frames displayed must have a time delta from the previously displayed frame equal to or greater than 200ms. This filters the frame down to 27 high-latency frames:

At this point, we simply need to look for a pattern, and luckily, the filter shows us the pattern very quickly. It appears that nearly all of the HTTP responses have delays of 1 second or longer. This is a huge find, and points us very close to the cause.
Initially, we may be tempted to point to the server itself as the cause of the delay, but we can’t prove that without looking at the server side trace. So we will open the server side trace and attempt to find a conversation with similar delay. It is helpful if you can find the same conversation, but not absolutely necessary. If the problem is caused by a server side issue, any conversation that is going to the same URL on the same server at approximately the same time should show a similar delay, and for the same reasons.
To find a similar delay, we will simply modify the previous filter slightly, removing the frame number restriction, like so:
(ip.addr eq 160.207.226.48 and ip.addr eq 160.207.12.41) and (http.request or http.response) and frame.time_delta >= 0.2
This filter will show us some frames with a very striking similarity to the ones seen in the previous trace:

This proves that the problem is not due to delay in the link, but is something related to the web server’s delivery of the requested page. To dig deeper, we now need to select an example of the problem, say the Get request for owsbrows.js and the response seen in frames 3273 and 3751, respectively, and filter to all traffic between this request and response. This is to see any additional traffic that our previous filters may have hidden. To do this, we will use the following filter:
frame.number >= 3273 and frame.number <= 3751
This filter filters down to all frames between these two numbers, including the first and last frames (request and response). However, this leaves us with 500 or so frames to look through, which is a bit more than we might wish. Looking at the first 36 packets, however, we can begin to determine what is occurring:

Looking at the bolded frames, we can see two things of importance. First, the client is requested for authentication from the server for the Get, to which it responds in a normal fashion (Get, 401, Get Negotiate, 401 Challenge, Get with Credentials). The interesting thing to note is the speed with which the server responds to each step of this process. You will notice that the server’s 401 responses occur consistently in less than 0.001 seconds. This is not consistent with our previous view of a web server performance problem. The second thing to notice is what the web server does in the last frame (3308): It makes an RPC call to a domain controller, presumably to verify the authentication provided by the client. This is perhaps the most important find in the trace, as it shows interaction with another server occurs before the client is validated and given the page he has requested.
Now that we have found this new piece of information, let’s filter to determine how long the response to this RPC request takes. This is most easily done by selecting the RPC request and right-clicking to choose ‘Follow TCP Stream’. Once this is done, you can very easily see the RPC request and response, identified by their Call ID:

The result of this final filter is interesting. Our previous analysis showed that there was a total delay between the HTTP Get and the Web Server’s response of 1.095 seconds (the delay between frames 3273 and 3751). The analysis of the RPC traffic shows that the delayed response from the domain controller accounts for 1.08 seconds of that delay! Using this information, we would be able to examine all of the other RPC calls to the DC and verify this finding, proving that the domain controller is the primary cause of the web server delay.
This information is the key to resolving the issue, since no amount of resource allocation towards the network infrastructure, web server, or other services (like SQL) would actually improve the situation at all. But an additional DC to handle the authentication traffic may completely resolve the issue.