Join Lookout at JRubyConf EU 2014!

Regular readers might have the impression that Lookout uses JRuby quite a bit. Turns out we do! JRuby Cardinal As big fans of JRuby we’ve also submitted a talk to this year’s JRubyConf EU to discuss some of our most recent work with JRuby, Storm and Kafka.

Our talk is titled: Building a scalable messaging fabric with JRuby and Storm and will be given by R. Tyler Croy and Ian Smith.

The abstract is as follows:

As Lookout has grown the number of backend Ruby services the need for reliable, asynchronous service-to-service messaging has gone from “nice to have” to “absolute requirement.”

Our first attempts included some names you may be familiar with: ActiveMQ, Resque, Redis, and Sidekiq. As our infrastructure grew, we found we were reinventing many of the concepts “Storm,” an open source real-time computation/stream-processing system, was specifically designed to handle such as message routing, durability, streaming from multiple inputs, delivering to multiple outputs, etc.

In this talk we’ll cover how we built a scalable service-to-service messaging fabric on top of Storm and Kafka, with JRuby playing a starring role.

JRubyConf EU will be held as part of Eurucamp on August 1st in Potsdam, southwest of Berlin. Please join us to talk about Ruby, JRuby, Lookout, mobile security and scalable systems!

posted in: · · · ·

Profiling remote JVMs with VisualVM and JConsole

Recently I found myself hosting a bit of a “bake-off competition” between servlet containers for JRuby applications here at Lookout.

The goal of the bake-off was to determine whether we should host some warbled JRuby applications in Tomcat or Jetty. Not having a huge amount of experience, or bias, towards one or the other I elected to run a simple Hello Warld application in both and see how well the containers performed.

The Bake-Off Environment

For the bake-off of servlet containers I used to identical EC2 instances. EC2 instances were chosen instead of running both containers on my local machine to keep the machines consistent, isolated and more representative of the environment we would be running webapps in. The test bed specs were:

  • Ubuntu 12.04 LTS
  • m1.small instance size (hey, I’m not made of money!)
  • us-west-2 region
  • A security group with everything open to the other machines in that security group. This is important for later.
  • OpenJDK 7 (u55)

The machines were then provisioned with the Tomcat 7 and Jetty 6 respectively, only because those were available directly from the native packages on Ubuntu 12.04.

Both containers were also set up to perform hot-deploys; a feature which relies on live-reloading of an application without restarting the JVM.

Problems, ahoy!

After performing a number of successive hot-deploys in Tomcat, I found my logs clobbered with the following errors:

Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space

The actual problem here will need to be covered in another blog post, but something fishy was clearly going on with hot-deployments in Tomcat.

Profiling in the cloud

My favorite tool for understanding a running JVM is definitely VisualVM, with JConsole running in a close second place. Fortunately, both tools are quite easy to set up for connecting to remote JVMs.

Setting up jstatd

Previously I mentioned the importance of the EC2 security group. It’s important that port 1099 is open within the security group. This is the port jstatd runs on by default. jstatd is what will provide VisualVM with live instrumentation data from the JVMs running on the machines.

It’s also important to provide a liberal security policy file to jstatd. If this were anything more than a simple test implementation, I would recommend a more restrictive policy, but we’re optimizing for easiness here, so a wide open policy is fine:

grant codebase "file:${java.home}/../lib/tools.jar" {

Save the policy above into a file named jstatd.policy, which we can run on each machine:

ec2-tomcat% sudo jstatd &
ec2-jetty% sudo jstatd &

JMX for interactivity

jstatd only gives us half the picture we want. We also want JMX to be configured for both containers to allow us to extract more information and interact with the running JVMs.

Again, we’ll set up very liberal security policies since this is for testing only! The native packages for both containers put a file in /etc/defaults which contains a JAVA_OPTS variable, to which the following should be added: \ \ \

After restarting the container processes, they will now have the right JMX settings and we should be able to finally be able to connect VisualVM or JConsole to the JVMs.

Proxying for connectivity

In order to give tools running locally on my machine access to these processes running inside of a security group within EC2, I’ll rely on ssh’s ability to provide a SOCKS5 proxy;

kiwi% ssh -D 9696

This will provide a path for both VisualVM and JConsole to use when accessing the JMX information (port 1098) and the jstatd information (port 1099) on the machines running inside of a security group within EC2. While not wholly necessary, exposing these ports to the wide-open internet seems like a Bad Idea™.

Running JConsole with a proxy

JConsole doesn’t have any GUI configuration for a proxy, so it’s necessary to set some command line parameters:

kiwi% jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=9696

Once JConsole is up and running, you only need to enter the EC2 hostname and appropriate JMX port (1098) to connect to the running JVM:

JConsole with EC2

After clicking “Connect”, JConsole will use the SSH-based proxy to connect to the host, and you should be able to poke around with a real live JVM:

JConsole connected to

Running VisualVM with a proxy

Unlike JConsole, VisualVM allows for a GUI-based configuration of a SOCKS proxy:

VisualVM proxy

With the proxy configuration saved, we can then add a remote host by right-clicking on “Remote” and selecting “Add remote host”.

Adding remote VisualVM

Provided jstatd is running on the remote host, your SSH-based proxy is running and the remote JVM is running, you should be able to connect to the remote JVM and start profiling it like you would a local JVM!

Profiling Tomcat

Both JConsole and VisualVM give you access to a lot of the instrumentation data available from a running Java Virtual Machine, but neither will magically identify or solve performance problems. There’s still more work to be done to triage and ultimately resolve those kinds of issues, but at least these tools give you the information you need to know what’s going on, and knowing is half the battle.

- R. Tyler Croy

posted in: · · · · ·

Open Source Rollup for Week #26

Yet another great week of open source hacking here at Lookout, and fortunately another week where we’ve open sourced more projects!

A while back we introduced FactoryJS, in the same vein of work we’ve now open sourced Oraculum, a set of Factory-driven Backbone.js behaviors based on Chaplin.

From the README:

Though a large portion of Oraculum’s behavior is inherited from Chaplin, Oraculum employs a signficiantly different strategy surrounding the issues of inheritance, structure, and coupling. Applications built with Oraculum take full advantage of the aspect-oriented programming and composition paradigms offered by FactoryJS. The purpose of this project is to collect abstract, reusable behaviors into a framework that can be used by anyone building complex applications with Backbone.

We also open sourced Spongebob, a tool to diff nmap runs.

Last week we mentioned that ngx_borderpatrol was in the process of learning how to talk to statsd. After a solid round of discussion and review on this pull request, support for statsd tracking of login successes and failures has been merged.

This week’s raw numbers:

  • 3 git pushes
  • 6 pull requests created
  • 16 comments on pull requests
  • 8 issues created
  • 2 repositories forked
  • 2 repositories open sourced








posted in: · ·