Introducing Oraculum

Oraculum is a javascript MVC framework and a collection of mixins for Backbone Models, Collections and Views written for FactoryJS. It inherits all of its application structure, many behaviors, and is generally inspired by Chaplin.

Why another JavaScript MV* framework?

Playing with Prototypes: A Cautionary Tale.

Lookout App Intel Console
Lookout App Intel Console

Oraculum emerged to support the development of our App Intel Console, a dynamic single page application that allows complex querying of our malware/security knowledge. We knew that the App Intel Console would one day serve as the codebase for our internal research/analysis toolchain, and would need to be embedded into other products/services. With this in mind we decided that we needed to adopt a library that supported workflows and processes for cross-product, cross-team code portability.

Initially evaluating several frameworks, we were all pretty smitten with Backbone and Chaplin. We really liked Chaplin’s introduction of a proper Controller, as well as the many improvements to Backbone’s Views, Models, Collection and Router it provided. What’s more, Chaplin introduced the CollectionView, which beautifully solved the problem of rendering Collections in an elegant and performant way. The tide only started to turn once we got into the nitty gritty of building an application of great complexity. Some of the implicit behaviors Chaplin provided suddenly weren’t what we wanted or needed. Modifying Chaplin’s underlying object prototypes to change its behaviors was a tedious and error-prone task, often leading to very tightly coupled behaviors and strange side effects. Foregoing this option and using CoffeeScript’s faux classes/extension provided some relief, but eventually led to deep inheritance, and, once again, tight coupling/strange side effects. Opting for a mixin-based strategy helped to some extent, but the fact that we were still bound to overriding methods on a prototype continued to lead to the same outcome.

Frankly, the entire paradigm had to change.

Enter FactoryJS

Around the time we thought we had most of this stuff reasonably figured out, Gabriel Hernandez joined Lookout supporting our Research & Response team as a Senior Software Engineer. He brought with him a passion for the concepts of factories, flyweights and mixins, which quickly caught my ear as a brilliant solution to the structural challenges we were facing. After a few asks, the initial version of FactoryJS was born. We started wrapping Backbone and Chaplin in our fancy new factory container, and abstracted our behaviors out into mixins. Immediately our codebase became more modular, more stable, and easier to test. Code sharing increased dramatically between the App Intel Console product and our internal research tools.

Simply using Backbone and Chaplin wrapped in the factory container took us a long way towards a functional product, but as the App Intel Console continued to increase in complexity, we once again ran into problems related to tight coupling. Chaplin’s underlying implicit behaviors were still forcing us to stub out methods we didn’t want, modify method implementations, and prohibited the use of things we had taken for granted in vanilla Backbone (like the initialize() method). Eventually we decided that while we loved the additional behaviors Chaplin provided, we couldn’t continue to use them as they were written. We also needed to abstract these behaviors out into mixins. Thus Oraculum was born.

So how does this thing work?

Oraculum is actually a library with three separate types of components and two utility functions. The official documentation goes into great detail about all of these components, but from a high-level perspective, these components are:

  1. A set of application/MVC components.
  2. A set of mixins/definitions for rendering tabular data.
  3. A library of Abstract, Model, Collection, and View behaviors.
  4. A set of utility functions that allow instance methods to emit events.

Application/MVC components

Oraculum Application Structure
Oraculum Application Structure

If you’re already familiar with Chaplin, Oraculum’s application components will look familiar to you. Aside from the implementation details in the underlying classes, and the named definition resolution, structuring an application in Oraculum almost exactly the same as Chaplin 1.1.x. However, just like in Chaplin, these components are completely optional and aren’t required to use Oraculum.

The Oraculum application components are essentially nothing more than Chaplin 1.1.x’s Application, Composer, Composition, Controller, Router, Route, and Dispatcher. These objects have been ported directly from Chaplin and wrapped in the FactoryJS container, having several of their implicit behaviors replaced by mixins that perform the same (or similar) functions. As a result, these mixins are available to any definition that wants to use them. They include mixins for publishing/subscribing to a global event bus, making objects disposable in a memory-safe way, freezing objects after construction, and several others.

Rendering tabular data

The tabular interface provided by Oraculum was introduced to solve the incredibly common problem of rendering Collections in a tabular format with configurable columns, sortable attributes, etc. These components are actually an excellent candidate to be broken out into a separate Oraculum plugin in a separate repository, but we decided to keep them in core since it is such a ubiquitous use case. To this end, Oraculum provides several mixins and definitions that allow a Collection to be rendered in a tabular fashion with a simple interface comparable to Backgrid, but more configuration-oriented.

Behavior library

Oraculum comes with a large library of mixins that aim to solve some of the most common use case problems when building a Backbone application. For example, if you need to bind an attribute of a Model to a particular element in a View, you may choose to use DOMPropertyBinding.ViewMixin, which has both a configuration and data-attribute based interface for one-way model -> element binding. If you want a Model to automatically invokes fetch() after it’s been constructed, you could use AutoFetch.ModelMixin, etc. At the time this blog was written, here is the exhaustive list of available mixins:

$ find src -type f | grep mixins

  • src/mixins/callback-provider.coffee
  • src/mixins/disposable.coffee
  • src/mixins/evented-method.coffee
  • src/mixins/evented.coffee
  • src/mixins/freezable.coffee
  • src/mixins/listener.coffee
  • src/mixins/middleware-method.coffee
  • src/mixins/pub-sub.coffee
  • src/models/mixins/auto-fetch.coffee
  • src/models/mixins/disposable.coffee
  • src/models/mixins/dispose-destroyed.coffee
  • src/models/mixins/dispose-removed.coffee
  • src/models/mixins/last-fetch.coffee
  • src/models/mixins/pageable-interface.coffee
  • src/models/mixins/remove-disposed.coffee
  • src/models/mixins/sort-by-attribute-direction-interface.coffee
  • src/models/mixins/sort-by-attribute-direction.coffee
  • src/models/mixins/sort-by-multi-attribute-direction-interface.coffee
  • src/models/mixins/sort-by-multi-attribute-direction.coffee
  • src/models/mixins/sortable-column.coffee
  • src/models/mixins/sync-machine.coffee
  • src/models/mixins/xhr-cache.coffee
  • src/models/mixins/xhr-debounce.coffee
  • src/views/mixins/attach.coffee
  • src/views/mixins/auto-render.coffee
  • src/views/mixins/cell.coffee
  • src/views/mixins/column-list.coffee
  • src/views/mixins/dom-cache.coffee
  • src/views/mixins/dom-property-binding.coffee
  • src/views/mixins/html-templating.coffee
  • src/views/mixins/layout.coffee
  • src/views/mixins/list.coffee
  • src/views/mixins/region-attach.coffee
  • src/views/mixins/region-publisher.coffee
  • src/views/mixins/region-subscriber.coffee
  • src/views/mixins/remove-disposed.coffee
  • src/views/mixins/static-classes.coffee
  • src/views/mixins/subview.coffee
  • src/views/mixins/templating-interface.coffee
  • src/views/mixins/underscore-templating.coffee

Aspect-oriented programming

Instead of relying on faux classical inheritance for modifying an object’s prototype, Oraculum provides two utilities: makeEventedMethod, and makeMiddlewareMethod. These utilities create a wrapped version of an instance’s target method that emit :before and :after events on a target event emitter. Oraculum also provides this functionality as a mixin via EventedMethod.Mixin and MiddlewareMethod.Mixin, which allow methods to be wrapped immediately after an object is constructed.

makeEventedMethod and makeMiddlewareMethod are how Oraculum provides shallow composition over deep inheritance, and they form the heart of Oraculum’s aspect-oriented programming strategy for logic decoupling.

Conclusions

By favoring composition over inheritance, and combining the concepts of factories, flyweights, mixins, and aspect-oriented programming in a dependency injection container, our code has become more stable, testable, dynamic, and portable, allowing insanely fast development of intensely complex browser-based applications.

Want to know more?

- Steve (Egesté) Regester

posted in: · · · · · · · ·



Open Source Rollup for Week #32

This week has been a quiet one for Lookout since most of the company left for Defcon later in the week.

Regardless, we managed to get a few more of our LuaJIT changes merged to ngx_borderpatrol.

We also open sourced our minor fork of the Zipkin tool, more details on that sometime in the near future though.

This week’s raw numbers:

  • 14 git pushes
  • 5 pull requests created
  • 2 repositories forked

Contributors


csanders-lookout

egeste

phrinx

rtyler

webspinner

wkimeria


posted in: · ·



Frankenwar: the Frankenjar revisited

Last year I wrote about Lookout’s use of the frankenjar, a self-contained archive of an entire JRuby-based artifact. The benefits of the frankenjar were many:

  • An entirely self-contained Ruby application without reliance on external gem or other dependencies.
  • A simple to re-use, test and deploy executable file.
  • A single trackable artifact to follow through the build and release system.
  • By using Rake as a built-in task runner, a single artifact can be used for many different, but related, functions. Such as running database migrations for the web application contained within the file.

JRuby Jay!

The frankenjar approach does have its downsides however. For running the web server, we were invoking a Rake task which would start up an embedded Puma instance. This means relying on Puma for managing request workloads itself, configuring it to use the appropriate number of request handling threads. It also means that if the .jar is running, you have a web server, if the .jar isn’t you have nothing.

At a more fundamental level, using Puma or any Ruby-based (Rack) webserver, also means that there will be additional performance overhead. A Rack-based server means lots of string handling in Ruby (not cheap) but more importantly it means that the application will be using Ruby sockets.

Ruby is not the issue, dude.

The problem with Ruby sockets is not something specific to Ruby at all, it’s just that the Socket interface in Ruby is a binding on top of the traditional BSD Socket interface. This traditional interface provides blocking I/O primitives such as select(2) or poll(2) which make building highly-scalable (read: concurrent) network servers difficult.

The JVM provides a much higher performance Socket interface with its “New I/O.” Mostly referred to as “NIO”, it abstracts some underlying evented I/O primitives provided by modern operating systems through kqueue(2) (Mac OS X/BSD) or epoll(7) (Linux). You can use NIO from Ruby with the nio4r gem but I’ve not yet seen a Rack server outside of Reel.

Servlets

A number of the short-comings of the frankenjar approach can be easily mitigated by using a Java Servlet Container such as Tomcat or Jetty. For the unfamiliar, Java has a specification similar to Ruby’s Rack for defining a server interface, the servlet specification. Many servers implement this specification often times adding additional enhancements around the servlet API itself. This allows you to take a web archive (a.k.a. “a .war”) and drop it into any number of different servlet containers.

Servlet containers like Tomcat usually use NIO by default or at least allow it to be configured. They also typically provide some amount of workload management for your applications. Tomcat for example can be configured to keep a threadpool for request handlers active with a minimum, maximum and minimum spare threads in it. Thereby allowing the server to dynamically, within bounds, scale up and down to handle varying request workloads.

You got chocolate in my peanut butter

Fortunately the same tool we use for creating the frankenjar, warbler, can create a frankenwar. The process is largely the same as the frankenjar, with some minor differences.

Like the frankenjar, the majority of the interesting configuration is in the config/warble.rb file:

# Disable Rake-environment-task framework detection by uncommenting/setting to false
Warbler.framework_detection = false

# Warbler web application assembly configuration file
Warbler::Config.new do |config|
  # Turn the `runnable` feature on to make sure that we can have an
  # executable war file
  config.features = %w(runnable)
  # Include the app/ directory in the .war
  config.dirs = ['app']
  # Ensure that we have our Rakefile available too
  config.includes += FileList['Rakefile']
  config.jar_name = 'hellowarld'
  config.override_gem_home = false
end

In the frankenjar we relied on a Rake-based runner to be the first file in the project’s bin/ directory. Warbler would then use this executable to drive the executable .jar file. Another Lookout engineer, Marc Chung, discovered some undocumented functionality in Warbler’s war bootstrap code which allows for executing binaries from within the .war file with the -S argument, e.g.: java -jar my.war -S rake.

This hidden functionality is really what makes the frankenwar feasible.


Note: if you intend to use Rake in this fashion, make sure it’s in the “default” group in your Gemfile so Warbler will include the gem in the package.


Of course, since this is intended to be a web application, we must define a config.ru file which will help indicate to Warbler that it should build a .war file:

# Add our current working directory to the LOAD_PATH
$LOAD_PATH.unshift(File.expand_path(File.dirname(__FILE__)))

require 'rubygems'
require 'app/hellowarld'

map '/' do
  run Sinatra::Application
end

With the config/warble.rb and config.ru files in place we can build and execute our hellowarld application:

% bundle exec warble war
rm -f hellowarld.war
Creating hellowarld.war
% java -jar hellowarld.war -S rake hello
Hello there
%

If we were to then drop our hellowarld.war into a servlet container, we’ll get “Hello Warld” when we access the /hellowarld URI.

(Note: a fully functional example of this can be found in this hellowarld application)


A Warbler-generated frankenwar in hand, many of the original needs for the frankenjar are met:

  • An entirely self-contained Ruby application without reliance on external gem or other dependencies.
  • A simple to re-use, test and deploy executable file.
  • A single trackable artifact to follow through the build and release system.
  • A versatile application which can run application-related tasks such as database migrations.

All with the added performance and support that Java servlet containers can offer:

  • Hot-deployment of a .war application. In many cases this is just dropping a .war file in a special directory on the application.
  • Higher concurrency through the server’s use of NIO.
  • Lower per-request overhead since much of the request parsing occurs in Java within the server.

Previously I had proclaimed that the frankenjar was “the easiest Ruby app deployment ever.” I’m ready to rescind that statement.

Not having learned my lesson about making wild proclaimations, I can now confidently say that:

The frankenwar is the easiest, and highest performance, Ruby app deployment ever.

- R. Tyler Croy

posted in: · · · · · ·