You may think of us as a vendor. We think of ourselves as a bunch of engineers who sit around and ask questions like: “is NginX faster and if so why?”. If we deliberately gamed a benchmark that would take all of the fun out of it. We like the competition. Resin has always been really fast, but competition from NginX has made us faster.

(more…)

These are performance numbers as part of the stress testing to verify Resin’s performance hasn’t degraded for the release. The other web servers are included as a comparison.

As part of the Resin 3.2.0 stress test, I loaded Resin with 25,000 comet clients for several hours.

The point of the test was to stress the socket/thread attachment and detachment code as well as well as any issues with large number of connections. And double-check any leaks of course.

The huge numbers of clients is possible because comet requests are normally idle, e.g. a chat application is just waiting for the next request. Since the request isn’t doing anything, Resin can release the thread and just hold the socket.

Because 25,000 requests takes up 25,000 connections and ServletRequest and ServletResponse objects, we needed to add a -Dcaucho.smallmem attribute to shrink the default buffer sizes. Normally, Resin’s buffers are 16k and each request uses several of these buffers. So 25k requests would need a few gigabits of memory just for the buffers. With -Dcaucho.smallmem, the buffer sizes are reduced.

I did a simple benchmark comparing Resin 3.1.4 to Tomcat 6.0.14 and the results are a bit astounding. Beyond the fact that Tomcat crashed and burned, Resin Open Source is about 45% faster than Tomcat, Resin Professional is about 80% faster than Tomcat. I made an 8k jsp that just contains static text. Each server was run with it’s default configuration, in console mode. I gave each server a warmup of 20 requests, and then did three runs of 20000 requests:

Resin OS 3.1.4:

Resin Professional 3.1.4:

Tomcat 6.0.14:

Tomcat died after 16000 requests of the second run with a java.lang.OutOfMemoryError: Java heap space.

Reproducing

This is fairly easy to try on your own machine, install is simple:

1. download/unpack Resin from http://www.caucho.com/download
2. copy an 8k txt file to webapps/bench/8k.jsp
3. $ bin/httpd.sh
4. in another terminal window, run ab
   $ ab -c 4 -n 20 http://localhost:8080/bench/8k.jsp
   $ ab -c 4 -n 20000 http://localhost:8080/bench/8k.jsp
   $ ab -c 4 -n 20000 http://localhost:8080/bench/8k.jsp
5. download/unpack Tomcat
6. copy an 8k txt file to webapps/bench/8k.jsp
7. $ bin/catalina.sh run
8. in another terminal window, run ab
   $ ab -c 4 -n 20 http://localhost:8080/bench/8k.jsp
   $ ab -c 4 -n 20000 http://localhost:8080/bench/8k.jsp
   $ ab -c 4 -n 20000 http://localhost:8080/bench/8k.jsp