Continuous Deployment for Ruby Gems

17 May 2012

As a Ruby shop, we have created and maintain a large number of Ruby gems.

In the earlier days we set up an internal gem repository with clearly documented SSH credentials and developers were responsible for building and packaging their gems, then SCPing them over to the primary gem repository.

Over time the list of gems we create and maintain has grown dramatically and the building, testing and release management of these gems became a bit of a chore. The biggest issue with the “simple” workflow described above is that it is far too easy for a developer to package up a gem, ship it, and then never commit their changes into Git. Projects “downstream” from that gem could find themselves the unwilling beta testers for untested changes that came straight from a developer’s workstation.

(Not that this ever happened1, Lookout engineers always commit their code after running a plethora of tests. We’re supremely disciplined!)

Seamless segue to: the workflow

Usually when describing a workflow, it’s easier to diagram it out, so I’ve charted out the workflow that gems follow with this amazing ASCII diagram:

+----------------+
| Local Git Repo |
+--+-------------+
   |
   | git-push(1) for review
   |
   |    +----------+        Code review passes
   +--->|  Gerrit  +---------------+
        +----------+               v
                         +---------------------+
                         | "Gold" GitHub Repo  |
                         +----------+----------+
                                    |
                                    v
                               +---------+
                               | Jenkins |
                               +----+----+
                                    |         +----------------+
                                    +-------->| Gem Repository |
                            Tests pass?       +----------------+

Stepping through this clearly amazing ASCII diagram we can follow a change “A1” from being committed to deployed:

  1. A developer runs tests locally (of course), and creates a commit A1
  2. The developer pushes A1 up to Gerrit and adds another developer as a reviewer.
  3. (Not diagrammed) Jenkins integrates with Gerrit, pre-testing the commit. Afterward, the other developer will review (and hopefully) okay the change.
  4. The originally developer will “submit” his change in Gerrit, which will replicate into the “Gold” GitHub repository (master branch).
  5. Jenkins learns that a new commit, A1, is available for the building and testing. Jenkins fires off a job which effectively just runs:

     % rake test
     % rake build
    
  6. If the job successfully completes, we auto-promote the gem to a “release” using the Promoted Builds plugin to execute some post-build promotion processes such as:
    • Keep this build forever
    • Publish the archived .gem artifact over SCP to the Gem repo with the SCP Plugin
    • Fire off another job which updates the Gem repo’s indexes
  7. At the end of this pipeline we typically will update another project’s Gemfile.lock and then go through a similar flow to get the updated gem used by downstream projects.

Conventions we follow

To enable this fairly standard workflow across the many gems we maintain, we have a couple of conventions:


With the above workflow and these conventions in place, we’ve reaped a couple of benefits, the most obvious one has been the severely reduced time it takes for new gems to be created and incorporated into production systems.

At a higher level, we’ve gained more confidence in our gems with this emphasis on testing, traceability and code review baked into the process from the start.

That confidence pays dividends when looking at moving larger, more complex systems towards continuous deployment, which we’ll cover in later postings.

- R. Tyler Croy

get reset to zero when the minor or major versions get bumped. I am personally okay with this given the added benefit of “deep traceability” that the build number from Jenkins provides for gems referenced in production.

  1. This has totally happened

  2. Contrary to what semver.org describes, the “patch number” does not