Resin REST adminstration interface
Starting with Resin 4.0.2 (due out within a week), the Resin administration application supports a REST interface for grabbing server information. In this blog post, I’ll give an overview of the REST interface, how to write plugins, and even how to integrate with Nagios.
Resin REST administration architecture
A number of monitoring applications are able to integrate with servers like Resin, but they need an easy way to access the server state programmatically. Our new REST interface does just that. We provide a PHP framework in which you can easily access server MBeans and output useful information.
Why PHP? Well, the rest of our Resin administration application is actually written in PHP because Quercus allows us to integrate PHP and Java. If there’s one thing that PHP makes easy, it’s plugins. To create a REST resource, all you have to do is create a PHP script with a .rest extension in a specified directory. The Resin administration application handles everything else — namely calling your script and providing you with easy access to MBeans.
Let’s make this more concrete. Say you want to print the status of the Resin thread pool. You could create a file named “threads.rest” like the following:
<?php
$jvm_thread = $g_mbean_server->lookup("java.lang:type=Threading");
$thread_pool = $g_server->ThreadPool;
echo "pool active: {$thread_pool->ThreadActiveCount}, " .
"pool idle: {$thread_pool->ThreadIdleCount}, " .
"pool count: {$thread_pool->ThreadCount}, " .
"jvm count: {$jvm_thread->ThreadCount}, " .
"jvm peak: {$jvm_thread->PeakThreadCount}";
?>
By requesting http://<yourserver>/resin-admin/rest.php?q=threads, this would print out a line like:
pool active: 11, pool idle: 14, pool count: 25, jvm count: 35, jvm peak: 43
Looking at the code, a couple of facilities are provided for you automatically. Specifically, we use Resin’s Server MBean (the global variable $g_server) and the generic JVM Threading MBean ($jvm_thread) to get the current thread status.
Now that we see how to print the status information from the server, we can next look at how to get the data to an external monitor like Nagios.
Integrating with Nagios
Architectural overview of the Resin REST engine
and how it integrates with Nagios via the shell script.
One common external monitoring application is Nagios. Nagios has an easy and straightforward plugin mechanism that can monitor resources like those exposed via the REST interface. This mechanism executes a process (on the OS level) that you specify, saves the standard output, and uses a simple process exit code standard to identify a service as ok, in a warning state, in a critical state, or in some unknown state. For our purposes we can use a shell script for this command that uses curl to access the resources, then parses the output and exits with an appropriate code.
There is a small protocol between the shell script and the REST resource which allows
- The REST resource to know what signifies a normal, warning, or critical state and
- the shell script to know which state the resources is in and thus what exit code to use.
Don’t worry about the protocol though — it’s extremely simple and we’ve provided a PHP library which does all the work for you on the Nagios wiki page.
Let’s see how this works by converting the script above into one suitable for Nagios.
<?php
include_once "nagios.php";
$jvm_thread = $g_mbean_server->lookup("java.lang:type=Threading");
$thread_pool = $g_server->ThreadPool;
$msg = "pool active: {$thread_pool->ThreadActiveCount}, " .
"pool idle: {$thread_pool->ThreadIdleCount}, " .
"pool count: {$thread_pool->ThreadCount}, " .
"jvm count: {$jvm_thread->ThreadCount}, " .
"jvm peak: {$jvm_thread->PeakThreadCount}";
nagios_output($thread_pool->ThreadActiveCount, $msg);
?>
You can save the “nagios.php” script referenced above from our Nagios wiki page. It contains the nagios_output() function. It takes two arguments, a number and a message, then outputs the status of the resource in a way that’s easily parsed by our shell script (also available on the wiki page). When checking services in Nagios, it’s common to use ranges to specify for what values the status should be considered ok, warning, or critical. The shell script is able to pass these ranges to the REST resource and the nagios_output() function can parse and check what status its number argument should indicate. Note that neither Nagios nor the shell script interpret this number; your REST script uses the ranges provided with whatever metric is appropriate for the given resource.
For example, say we have configured the Resin thread pool for a maximum of 512 threads and we think that having up to 400 threads is normal, 400-450 threads should be a warning, and above 450 threads is critical. Then using the Nagios range format, we could set our warning range to “400″ and our critical range to “450″. To call our shell script with these values, we would execute:
check_resin.sh --host=localhost --port=8080 --resource=check_thread_pool \
--username=admin --password=admin \
--warning-range=400 --critical-range=450
This assumes that we named the Nagios thread REST script “check_thread_pool.rest”. If there were, for example, 412 threads running when we called the shell script, we might get output like this:
WARNING - pool active: 412, pool idle: 20, pool count: 432, jvm count: 450, jvm peak: 462
The shell script would then use exit code “1″ to indicate that the thread resource is in a warning state to Nagios. The nagios_output() function that we used above determines that 412 is a warning state, prepends the “WARNING” header, then the shell script simply parses that header to determine its exit code. (I said it was a simple protocol :-)).
Now all we have to do is configure Nagios to call our shell script. We do this by creating two Nagios objects, a command and service. The command might look like the following:
define command{
command_name resin_threads
command_line /usr/local/nagios/resin/check_resin.sh --host=$HOSTADDRESS$ --port=$ARG1$ --resource=check_thread_pool \
--username=admin --password=admin \
--warning-range=$ARG2$ --critical-range=$ARG3$
}
Writing the command this way allows you to use it for multiple hosts with various different configurations and warning values. If you like, you could also templatize the resource argument as well.
Next add a service to poll the plugins:
define service {
use local-service ; Name of service template to use
host_name localhost
service_description Resin threads
check_command resin_threads!8080!256!384
notifications_enabled 0
}
After a restart, Nagios will start recording the status of this service and be able to create graphs and event logs. Here’s a screenshot of the overall status page:
The other resources shown there are available on the wiki page as well. That page also fills in a number of other details such as where to put your .rest scripts, etc.
What’s nice about this approach is that once you get it set up initially, it only takes a few lines of code to monitor another resource. You can use jconsole or the Resin administration application’s JMX view to find other MBeans to monitor. If you’ve written your own MBeans, you can also monitor those as well. If you come up with some good scripts that you’d like to share, please reply here or send them to the mailing list and forum!