Recently at work I spent a few weeks tuning a network service across three platforms (Solaris, Linux, and AIX) to get within 10% of the theoretical maximum throughput. In this short article, I’ll walk through the various tools I used to improve the performance of the application.
This application is very specialized in that the two machines are connected directly through an ethernet switch. This means that the MTU could easily be determined from each end of the link and the extra work to determine the maximum segment size for the transit network (see RFC 1191) was unnecessary. This also made it very easy to watch the traffic between the two hosts as well as the system calls they were using to transfer and receive the data.
Before I get into the steps I took to tune the service, I’d like to introduce the tools used:
- Truss: a tracing utility which displays system calls, dynamically loaded user level function calls, received signals, and incurred machine faults. This is available for many platforms, but I use it most on AIX.
- DTrace/DTruss: a dynamic tracing compiler and tracing utility. This is an amazingly powerful tool from Sun, originally for Solaris but slowly spreading to other platforms. See Sun’s How To Guide.
- strace: a dynamic tracing utility which displays systems calls and received signals under Linux.
- mpstat: collects and displays performance statistics for all logical CPUs in the system.
- prstat: iteratively examines all active processes on the system and reports statistics based on the selected output mode and sort order.
- tcpdump: a utility for capturing network traffic.
- Wireshark: a network protocol analyzer. It replaces the venerable Ethereal tool and allows you to either capture network traffic on demand or load a captured session for analysis. Find out more here.
- gprof: a tool for profiling your code to determine where the performance bottle-necks are. See the manual for more information.
- c++filt: a tool for demangling C++ method names. It is part of the GNU binutils package.
Since I already had the service up and running, I simply ran the two
components and captured the traffic between them using tcpdump. While
the processes were running, I also used dtruss, truss, or strace
(depending on the platform) to capture the system calls being made.
Since this is a network service, I focused on calls to select, send,
and recv.
13455/15: 2143177 2994 4 pollsys(0xFFFFFD7EBADDB910, 0x1, 0xFFFFFD7EBADDBA30) = 1 0
13455/15: 2143180 5 0 pollsys(0xFFFFFD7EBADDB8D0, 0x1, 0xFFFFFD7EBADDB9F0) = 1 0
13455/15: 2143185 8 4 recvfrom(0x11, 0xB384A0, 0x10000) = 1416 0
13455/15: 2143253 5 0 pollsys(0xFFFFFD7EBADDB8D0, 0x1, 0xFFFFFD7EBADDB9F0) = 0 0
13455/15: 2143262 12 8 send(0x11, 0xB084D0, 0x14B8) = 5304 0
13455/15: 2143268 365 4 pollsys(0xFFFFFD7EBADDB910, 0x1, 0xFFFFFD7EBADDBA30) = 1 0
13455/15: 2143270 4 0 pollsys(0xFFFFFD7EBADDB8D0, 0x1, 0xFFFFFD7EBADDB9F0) = 1 0
13455/15: 2143275 8 4 recvfrom(0x11, 0xB384A0, 0x10000) = 1416 0
13455/15: 2143343 5 0 pollsys(0xFFFFFD7EBADDB8D0, 0x1, 0xFFFFFD7EBADDB9F0) = 0 0
13455/15: 2143348 9 4 send(0x11, 0xB084D0, 0x14B8) = 5304 0
13455/15: 2143353 1000 4 pollsys(0xFFFFFD7EBADDB910, 0x1, 0xFFFFFD7EBADDBA30) = 1 0
Looking at the results above you can see that select (pollsys) is
being called each time we need to send or receive data over the network.
Since the socket is non-blocking we can rely on the immediate return
when the outgoing socket buffer is full as well as when there is no data
available to read. By selecting at the very top of the receive loop we
can bundle multiple receive calls together, increasing the application’s
throughput. Now the output looks like this:
16712/9: 16202 1560 6 pollsys(0xFFFFFD7EBB9DB940, 0x1, 0xFFFFFD7EBB9DBA30) = 1 0
16712/9: 16217 10 6 recv(0xB, 0x8A6450, 0x10000) = 1416 0
16712/9: 16246 9 5 send(0xB, 0x876480, 0x540) = 1344 0
16712/9: 16267 7 3 send(0xB, 0x876480, 0x540) = 1344 0
16712/9: 16285 5 1 send(0xB, 0x876480, 0x540) = 1344 0
16712/9: 16680 10 5 recv(0xB, 0x8A6450, 0x10000) = 1416 0
16712/9: 16712 11 7 send(0xB, 0x876480, 0x540) = 1344 0
16712/9: 16733 7 3 send(0xB, 0x876480, 0x540) = 1344 0
16712/9: 16753 6 2 send(0xB, 0x876480, 0x540) = 1344 0
16712/9: 16768 4 0 recv(0xB, 0x8A6450, 0x10000) = -1 Err#11
You’ll notice that now we are able to process two requests and send out
six responses in the time that it previously took to call select and
receive a single request. When there is nothing left to read, the call
to recv returns errno 11 (EAGAIN). This change made the single
biggest performance impact on the code. I also changed the calls
recvfrom to recv since the application did not make use of the
foreign address.
At this point the performance was much better but I noticed that under heavy load the sending socket would block as the ratio of requests to responses was 1:3. As this was a UDP application, having the sending buffers fill up seemed strange as we assumed that additional packets would simply be dropped on the floor.
On the server, I checked the UDP socket buffer size using ndd (this
was under Solaris. For AIX the command is no and for Linux the command
is sysctl).
The following code was added to the socket initialize (minus the error handling) to ensure that the socket buffers were large enough.
unsigned size = 1024 * 1024; // 1MB
int ret = setsockopt(desc, SOL_SOCKET, SO_SNDBUF, &size, sizeof(size));
ret = setsockopt(desc, SOL_SOCKET, SO_RCVBUF, &size, sizeof(size));
Now that the application was performing acceptably I decided to run it under the profiler. This turned up the function which was adding responses to the in-memory packet. It turned out that as responses were being added to the packet, the headers were being recalculated each time. I removed this unnecessary work and only made the calculations right before the packet was sent. This improved performance a few percentage points more.
By binding the network interrupts to a particular core and keeping the
sending thread off of that core we were able to eek out additional
performance from the application. To accomplish this, the application
allows the operator to specify which core(s) it should bind to using
sched_setaffinity (Linux) and processor_bind (Solaris). You can also
accomplish this using taskset (Linux) and pbind (Solaris) if you don’t
wish to modify your application.
Looking at the network traffic with tcpdump, I saw that I could fit an additional response in the response bundle packet if I reduced or removed some of the items in the packet header. At this point the analysis and tuning had gone on for a few weeks and we had a schedule to meet. Since the performance was where we needed it, the application was wrapped up and sent to quality assurance.
The single most important lesson I learned from this exercise was to use
non-blocking sockets to their fullest by continually calling
recv/send until the call would block and then using select to idle
the process until there is work to do.