JRuby, sponsored in part by Lookout

23 December 2014

In my time at Lookout we’ve had to grow in just about every way imaginable. New products, new applications, new people, new teams; growth in every dimension. In order to help engineering continue to grow and be successful within our service-oriented infrastructure, we created a team named “Core Systems” late this year. Among the team’s responsibilities are building tools, support engineering and maintaining systems foundational to a number of our products such as Kafka, Storm, Cassandra and so on. The systems we’ll cover in later blog posts, but for this post we’ll focus on our support of the JRuby toolchain.

JRuby Jay!

Since the beginning Lookout has primarily been a Ruby-based engineering team. As our deployment and tooling requirements have evolved, we’ve become more and more invested in the JVM, which means JRuby and some of the associated tooling (e.g. jbundler) are critical to the efficiency of our day-to-day work.

To help us support the JRuby toolchain internally we’ve welcomed Christian Meier, a very talented JRuby hacker, to the Lookout engineering team.

The benefits have almost been instantaneous, with a great number of changes contributed by Christian on Lookout’s behalf making their way into: JRuby 1.7.17, JRuby 1.7.18, jbundler 0.7.0 and jruby-openssl 0.9.6.

This post doesn’t aim to cover each commit and bug fix specifically, but will provide a general overview of the areas of focus for our bug fixes and improvements.

JRuby

In JRuby 1.7.13 and 1.7.14 the LoadService was refactored quite a bit to support operations such as: File.exists?, Dir['*'], and require 'somefile. Despite best intentions, a number of regressions did sprout up after the releases went public.

File operations without “native” support

One series of regressions occurred when the native support from FFI would not be properly loaded, and the fallback to Java failed. The cases where the FFI libraries cannot load aren’t necessarily bugs since JRuby supports explicitly disabling loading of “native code” at runtime.

The failure of the Java-based fallback code would impact operations such as: jruby -S gem install my.gem if ~/.gem/jruby/1.9 was not a directory. Basically operations that would require:

And so on. The JRuby test suite does contain many tests for these calls, but it was only being run with native code enabled.

Fixing this class of bugs required not only fixing the tests to run with and without native code enabled, but also required fixing a number of underlying bugs in Java code.

Testing on various JRuby “installations”

Previously much of JRuby’s testing focus was on the JRuby which you might install directly on your system. Because of the versatility and embeddability of JRuby however, there are a number of different forms of “installations” which look different than what you might get from RVM:

The biggest difference between those methods and the regular JRuby installation (via rbenv, rvm, etc) is that the JRuby stdlib is inside a jar. Some features like the “default gems” bundled with JRuby (e.g. jruby-openssl) do not work when you set jruby.home to be classpath:/META-INF/jruby.home. This means users could not pick the correct version of the jruby-openssl or even get a nasty MethodNotException if they attempted to.

JRuby gives some (unnecessary) preference for the Thread.currentThread().getContextClassLoader() and assumes that JRuby is loaded via this classloader. When running inside of a Frankenwar, or a Storm topology this is not always going to be the case.

By adding some tests to exercise these alternative forms of JRuby installations, a number of bugs were identified and corrected here as well.

URI-like paths

As mentioned in the previous section, JRuby supports a number of different forms of installations and can be easily embedded. A side-effect of this functionality is that a “file path” in JRuby can be a number of different things:

These URI-like paths can pop up very quckly when using constructs such as File.dirname(__FILE__) from a file within the stdlib, JRuby kernel or even code packed inside of a .jar file.

Sometimes these URI-like paths would need to be translated into a java.io.InputStream like with X509Store#add_cert. The LoadService refactoring already had some support for handling these use-cases but it was error prone. The older mechanisms for doing this had already been dropped in the next major development branch of JRuby (also known as JRuby 9000) but the 1.7.x branch needed some fixes too.

jruby-openssl

Concurrency problems

The X509Store had some synchronization issues discovered by some of our applications running in production. While not an egregious bug, fixing it was a good exercise for Lookout of getting production-level issues reported, triaged and merging fixes into an open source project.

In this case the mutex for an X509Store instance was used by several classes and needed to be fixed and synchronized properly.

Certificate loading from URI-like paths

Related to the URI-like paths changes above, jruby-openssl also needed some updates to ensure that it could load certificates from within .jar files like jruby-stdlib.jar. Previously attempting to load these bundled certificates would always fail due to the aforementioned URI-like paths, but starting with JRuby 1.7.17 this behavior has been finally corrected.

Misc. fixes

jruby-openssl did see a lot of refactoring around Digest and Certificates prior to the 0.9.6 release. Ruby code was moved into Java, in addition to some cases where code was using Bouncycastle directly instead of the Java Crypto Extensions. After that regression there were some hairy test regressions which needed to be fixed against both JRuby 9000 and the JRuby 1.7.x branch of development in order to get the jruby-openssl 0.9.6 gem released.


Overall we’re thrilled to be sponsoring JRuby development and helping “raise all the boats” by getting bugs we’re finding into the open source ecosystem.

I’m certainly looking forward to what we can do in 2015 as an engineering organization, building great products with great technology, with regular open source contributions of course!

If you’re interested in helping to build a great security company and enjoy challenges, join Lookout. We’ve got a number of positions open, not only on Core Systems but across the board in engineering from Android, to iOS, Security, Operations, and ‘Platform and Infrastructure.’