ErinGary has NEXRAD images and commentary of yesterday’s storm here in Northern, VA. For about half-an-hour yestersay, the folks at work milled around in interior offices, hallways, and the stairwell while my family at home hid out in the basement. Thankfully there were not much damage in my neighborhood, though I saw several fallen trees on homes during my commute home.

As Chad Fowler says, “The best reason to learn a new programming language is to learn to think differently.” Here in Northern VA, we have a new opportunity to think differently with the introduction of the NoVA Languages group. Chris Williams from Iterative Designs posted to the NoVA RUG mailing list this morning and received quite a bit of interest. Here’s what he has to say about the new group:

Thoughts on the makeup of the group include obtaining (however you want) a book, working through the book 1 chapter per week on one night of that week with a group of like minded individuals.

To start, the group will work through Joe Armstrong’s Programming Erlang book starting on Monday, June 16th (hopefully I’ll be able to work out a schedule with my wife so I can attend). This is an excellent choice on several levels: every developer should know a functional programming language; single core processors are increasingly rare and the number of cores in commodity hardware should only increase in the coming years; and I already own the book.

If Erlang doesn’t pique your interest, the NSCoderNight DC group is going to work through the 3rd Edition of Aaron Hillegass’s Cocoa Programming for Mac OS X book on Tuesday nights. For several months now, I’ve been playing around with Cocoa and while I’m getting used to the syntax of Objective-C, the XCode IDE and Interface Builder still seem foreign to me (I never liked IDEs having been weened on Emacs).

It’s a great time to be a programmer in Northern Virginia!

Update: It turns out that there’s another Cocoa group right around the corner from me in Reston: CocoaHeads. They meet the second Thursday of each month.

Tonight I had the opportunity to attend a special Refresh DC meeting on the challenges of starting your own business. The format was a panel of folks from the DC startup scene moderated by Jackson Wilkinson of Viget Labs. The panel consisted of Brian Williams of Viget Labs, Andrew Lee of Publi.us, Eric Rupert from Odeo, Eddie Frederickof Hungry Machine, and Sean Greene of LaunchBox Digital.

These were the main points I took away:

1. Be passionate. The most important quality for startup founders is to be passionate about their product or service. Starting a company can be exhilarating but the setbacks can be really difficult. If you’re not passionate about what you’re doing each of these roadblocks will be an excuse for you to quit. Figure out what you’re passionate about and work on that.
2. Focus on your product. Don’t focus on the pie-in-the-sky potential valuations of your company or worry about leasing office space, hiring an attorney, finding an accountant, etc. Instead you should focus on the product or service you’re going to sell. Get something out there quickly – pick the most important feature and get it out there in front of your customers quickly.
3. Have your customers influence your product. If you follow the previous advice and get your product out there quickly you can use customer feedback to iterate your product. You may not know all of the difference ways customers will use your product, so this feedback is critical in charting your product roadmap.
4. Equity is control, don’t surrender it easily. One of the attendees asked about using equity to pay for services if money was tight. Brian said to avoid giving out equity as you make that person a partner in your business. While they may not have a controlling interest, they are still an owner and have some influence.
5. Don’t overfund. All of the panelists warned of the dangers of taking too much money as it does more than dilute the founders’ ownership. Venture capitalists are looking to make a large return o their capital for themselves and their limited partners. If you need to pay a vendor or contractor find another way – defer payment, use credit, etc.
6. Hire slow, fire fast. Early employees can make or break your company. You will be working long hours aside these folks so you must ensure they’re a good fit. If you make a bad hire, you need to resolve the situation quickly, don’t let emotion get in the way.

Some of the books recommended were: Art of the Start, Getting to Yes, and Founders at Work. Andrew also recommended Startup School run by Y Combinator (YC). I’ve been reading Paul Graham’s essays for around seven years now and following YC’s investments. I’m glad to see an early stage investor like YC in the DC area (LaunchBox Digital) and hope that the startup scene in DC and suburbs becomes more vibrant.

Thanks to Strategic Analysis for hosting this. It was a great venue and hope they’ll offer to host Refresh DC again.

Recently at work I spent a few weeks tuning a network service across three platforms (Solaris, Linux, and AIX) to get within 10% of the theoretical maximum throughput. In this short article, I’ll walk through the various tools I used to improve the performance of the application.

This application is very specialized in that the two machines are connected directly through an ethernet switch. This means that the MTU could easily be determined from each end of the link and the extra work to determine the maximum segment size for the transit network (see RFC 1191) was unnecessary. This also made it very easy to watch the traffic between the two hosts as well as the system calls they were using to transfer and receive the data.

Before I get into the steps I took to tune the service, I’d like to introduce the tools used:

• Truss: a tracing utility which displays system calls, dynamically loaded user level function calls, received signals, and incurred machine faults. This is available for many platforms, but I use it most on AIX.
• DTrace/DTruss: a dynamic tracing compiler and tracing utility. This is an amazingly powerful tool from Sun, originally for Solaris but slowly spreading to other platforms. See Sun’s How To Guide.
• strace: a dynamic tracing utility which displays systems calls and received signals under Linux.
• mpstat: collects and displays performance statistics for all logical CPUs in the system.
• prstat: iteratively examines all active processes on the system and reports statistics based on the selected output mode and sort order.
• tcpdump: a utility for capturing network traffic.
• Wireshark: a network protocol analyzer. It replaces the venerable Ethereal tool and allows you to either capture network traffic on demand or load a captured session for analysis. Find out more here.
• gprof: a tool for profiling your code to determine where the performance bottle-necks are. See the manual for more information.
• c++filt: a tool for demangling C++ method names. It is part of the GNU binutils package.

Since I already had the service up and running, I simply ran the two components and captured the traffic between them using tcpdump. While the processes were running, I also used dtruss, truss, or strace (depending on the platform) to capture the system calls being made. Since this is a network service, I focused on calls to select, send, and recv.

13455/15:   2143177    2994      4 pollsys(0xFFFFFD7EBADDB910, 0x1, 0xFFFFFD7EBADDBA30) = 1 0
13455/15:   2143180       5      0 pollsys(0xFFFFFD7EBADDB8D0, 0x1, 0xFFFFFD7EBADDB9F0) = 1 0
13455/15:   2143185       8      4 recvfrom(0x11, 0xB384A0, 0x10000)                    = 1416 0
13455/15:   2143253       5      0 pollsys(0xFFFFFD7EBADDB8D0, 0x1, 0xFFFFFD7EBADDB9F0) = 0 0
13455/15:   2143262      12      8 send(0x11, 0xB084D0, 0x14B8)                         = 5304 0
13455/15:   2143268     365      4 pollsys(0xFFFFFD7EBADDB910, 0x1, 0xFFFFFD7EBADDBA30) = 1 0
13455/15:   2143270       4      0 pollsys(0xFFFFFD7EBADDB8D0, 0x1, 0xFFFFFD7EBADDB9F0) = 1 0
13455/15:   2143275       8      4 recvfrom(0x11, 0xB384A0, 0x10000)                    = 1416 0
13455/15:   2143343       5      0 pollsys(0xFFFFFD7EBADDB8D0, 0x1, 0xFFFFFD7EBADDB9F0) = 0 0
13455/15:   2143348       9      4 send(0x11, 0xB084D0, 0x14B8)                         = 5304 0
13455/15:   2143353    1000      4 pollsys(0xFFFFFD7EBADDB910, 0x1, 0xFFFFFD7EBADDBA30) = 1 0

Looking at the results above you can see that select (pollsys) is being called each time we need to send or receive data over the network. Since the socket is non-blocking we can rely on the immediate return when the outgoing socket buffer is full as well as when there is no data available to read. By selecting at the very top of the receive loop we can bundle multiple receive calls together, increasing the application’s throughput. Now the output looks like this:

16712/9:     16202    1560      6 pollsys(0xFFFFFD7EBB9DB940, 0x1, 0xFFFFFD7EBB9DBA30) = 1 0
16712/9:     16217      10      6 recv(0xB, 0x8A6450, 0x10000)                         = 1416 0
16712/9:     16246       9      5 send(0xB, 0x876480, 0x540)                           = 1344 0
16712/9:     16267       7      3 send(0xB, 0x876480, 0x540)                           = 1344 0
16712/9:     16285       5      1 send(0xB, 0x876480, 0x540)                           = 1344 0
16712/9:     16680      10      5 recv(0xB, 0x8A6450, 0x10000)                         = 1416 0
16712/9:     16712      11      7 send(0xB, 0x876480, 0x540)                           = 1344 0
16712/9:     16733       7      3 send(0xB, 0x876480, 0x540)                           = 1344 0
16712/9:     16753       6      2 send(0xB, 0x876480, 0x540)                           = 1344 0
16712/9:     16768       4      0 recv(0xB, 0x8A6450, 0x10000)                         = -1 Err#11

You’ll notice that now we are able to process two requests and send out six responses in the time that it previously took to call select and receive a single request. When there is nothing left to read, the call to recv returns errno 11 (EAGAIN). This change made the single biggest performance impact on the code. I also changed the calls recvfrom to recv since the application did not make use of the foreign address.

At this point the performance was much better but I noticed that under heavy load the sending socket would block as the ratio of requests to responses was 1:3. As this was a UDP application, having the sending buffers fill up seemed strange as we assumed that additional packets would simply be dropped on the floor.

On the server, I checked the UDP socket buffer size using ndd (this was under Solaris. For AIX the command is no and for Linux the command is sysctl).

The following code was added to the socket initialize (minus the error handling) to ensure that the socket buffers were large enough.

unsigned size = 1024 * 1024; // 1MB
int ret = setsockopt(desc, SOL_SOCKET, SO_SNDBUF, &size, sizeof(size));
    ret = setsockopt(desc, SOL_SOCKET, SO_RCVBUF, &size, sizeof(size));

Now that the application was performing acceptably I decided to run it under the profiler. This turned up the function which was adding responses to the in-memory packet. It turned out that as responses were being added to the packet, the headers were being recalculated each time. I removed this unnecessary work and only made the calculations right before the packet was sent. This improved performance a few percentage points more.

By binding the network interrupts to a particular core and keeping the sending thread off of that core we were able to eek out additional performance from the application. To accomplish this, the application allows the operator to specify which core(s) it should bind to using sched_setaffinity (Linux) and processor_bind (Solaris). You can also accomplish this using taskset (Linux) and pbind (Solaris) if you don’t wish to modify your application.

Looking at the network traffic with tcpdump, I saw that I could fit an additional response in the response bundle packet if I reduced or removed some of the items in the packet header. At this point the analysis and tuning had gone on for a few weeks and we had a schedule to meet. Since the performance was where we needed it, the application was wrapped up and sent to quality assurance.

The single most important lesson I learned from this exercise was to use non-blocking sockets to their fullest by continually calling recv/send until the call would block and then using select to idle the process until there is work to do.

Almost a year ago, I submitted three recipes to Advanced Rails Recipes. Unfortunately, one which was originally selected back in June has been dropped from the book as it’s not really advanced. So I thought I’d post it here for everyone to enjoy.

Using AJAX with REST

Problem:

You have finally mastered REST (Representational State Transfer) and wish to enhance your application with JavaScript but don’t know which of the resource URLs to call.

Ingredients:

This recipe requires Rails 1.2 or later as it relies on RJS for the Ajax portion and REST for the controller design.

Solution:

In this example we need users to be able to register (and unregister) for meetings so that we can properly report attendance. So we have three models: User, Meeting and Registration.

Since Registrations are associated with a specific meeting, we’ve created them as a nested resource, like so:

~~~~ {lang=”ruby”} map.resources :meetings do |meetings|

meetings.resources :registrations

end ~~~~

To register for a meeting we want to create a new Registration instance tied to the Meeting and User. Unregistering from a meeting should destroy the Registration object. So our RegistrationsController looks like this:

~~~~ {lang=”ruby”} # POST /registrations def create

@meeting = Meeting.find(params[:meeting_id])
current_user.register_for(@meeting)
respond_to do |format| 
  format.html { redirect_to meetings_url }
  format.js   # create.rjs
  format.xml  { head :ok }
end

end

# DELETE /registrations/1 def destroy

@registration = Registration.find(params[:id])
@registration.destroy
respond_to do |format|
  format.html { redirect_to meetings_url }
  format.js   # destroy.rjs
  format.xml  { head :ok }
end

end ~~~~

In the view we want to use Ajax to allow the user to modify their registration, so we need to use link_to_remote:

~~~~ {lang=”ruby”} <% if !current_user.registered_for(meeting) -%>

<%= link_to_remote "Register!", :url => registrations_path(meeting), 
                              :meeting_id => meeting.id, :method => :post %>

<% else -%>

<%= link_to_remote "Unregister", 
                              :url => registration_path(meeting, 
                                         current_user.registration_for(meeting)),  
                              :confirm => 'Are you sure?', :method => :delete %>

<% end -%> ~~~~

The RJS to update the page looks like this:

~~~~ {lang=”ruby”} page[” meeting_#{@meeting.id}”.to_sym].replace_html

   :partial => 'shared/meeting', :object => @meeting

page[” meeting_#{@meeting.id}”.to_sym].visual_effect :highlight, :duration => 5 ~~~~

Discussion:

The resource URLs provided by Rails are extremely useful, but they can be confusing. Tools like FireBug make it easy to see what is going on behind the scenes. When developing Rails applications, it is usually easier to develop without Ajax and then go back and add Ajax to those portions of the application where it would make sense. With this approach, you can gracefully degrade your service back to the original mode for those who don’t have or don’t enable JavaScript. It also allows you to click through your application with FireBug enabled and watch which URLs are hit and with which methods (GET, POST, PUT, or DELETE).

The table below outlines which resource URLs to use with your remote helpers:

Intent Action Resource URL Method

---

Creating a new instance create plural form of model name (i.e. registrations_path) POST Retrieving an instance show singular form of model name with id (i.e. registration_path(registration) GET Retrieving all instances index plural form of model name (i.e. registrations_path) GET Modifying an existing instance update singular form of model name with id (i.e. registration_path(registration)) PUT Deleting an existing instance destroy singular form of model name with id (i.e. registration_path(registration)) DELETE

Once you are comfortable with the resource URLs, adding Ajax to your RESTful is much easier.

Further Reading:

If you are unfamiliar with REST, read Chapter 20 of Agile Web Development with Rails, 2nd Edition by Dave Thomas and David Heinemeier Hansson (Pragmatic Programmers, 2006). There is also a nice screencast on the subject from PeepCode.

For more on Ajax in Rails, you can check out Ajax on Rails by Scott Raymond (O’Reilly & Associates, 2007).