Using Puppet Lint with Jenkins

As the Puppet and Chef developer communities have matured, there has been an increased emphasis on style and sanity checking, also known as “linting.”

In the Chef community the hammer of choice is foodcritic, while Puppet users have puppet-lint to rely on.

Puppet Lint Warnings in Jenkins

Console-based warnings are great for local development, but since we’re already running all our Puppet code through Jenkins for validation, why not let Jenkins track linting as well? This can be easily accomplished with the Jenkins Warnings plugin and a little bit of set up work.


  1. First make sure you install the Warnings plugin from the Jenkins “Manage Plugins” page
  2. Add a special task to your Rakefile which will invoke puppet-lint with a specific log format, as shown here
  3. Invoke the new Rake task in the build (rake lint:ci)
  4. Configure the Jenkins job to look for the puppet-lint warnings Warnings scan configuration
  5. Run some builds, and enjoy your new puppet-lint reports and trend graphs!

On the build pages you will be able to drill into an overview report such as the one pictured below, hyperlinked to allow you to dig deeper into the specific warnings

Build-specific warnings report

On the job page (http://jenkins/job/my-puppet-module) you will also have a cross-build trend graph to give an indication of the trend of warnings as time goes on.

Overall warnings trend

That’s all there is to it! There isn’t any more setup required to get puppet-lint and Jenkins working together nicely. The hardest part of the whole process seems to be resolving all of the warnings and the second hardest part seems to be keeping the warnings at zero as time progresses.

The Warnings plugin allows for a lot more configuration than I’ve covered in this post, so be sure to explore its more “Advanced” options once you’re up and running!

- R. Tyler Croy

posted in: · · · ·

Lookout out and about

This week is a busy social week for Lookout Engineering, and also a good opportunity for track down a real live Lookout hacker, in the wild!

You’ll be able to find a variety of engineers at each of the events with the highest density of Lookout hackers attending Google I/O.

See you around!

- R. Tyler Croy

posted in: · · · ·

Continuous Deployment for Ruby Gems

As a Ruby shop, we have created and maintain a large number of Ruby gems.

In the earlier days we set up an internal gem repository with clearly documented SSH credentials and developers were responsible for building and packaging their gems, then SCPing them over to the primary gem repository.

Over time the list of gems we create and maintain has grown dramatically and the building, testing and release management of these gems became a bit of a chore. The biggest issue with the “simple” workflow described above is that it is far too easy for a developer to package up a gem, ship it, and then never commit their changes into Git. Projects “downstream” from that gem could find themselves the unwilling beta testers for untested changes that came straight from a developer’s workstation.

(Not that this ever happened1, Lookout engineers always commit their code after running a plethora of tests. We’re supremely disciplined!)

Seamless segue to: the workflow

Usually when describing a workflow, it’s easier to diagram it out, so I’ve charted out the workflow that gems follow with this amazing ASCII diagram:

| Local Git Repo |
   | git-push(1) for review
   |    +----------+        Code review passes
   +--->|  Gerrit  +---------------+
        +----------+               v
                         | "Gold" GitHub Repo  |
                               | Jenkins |
                                    |         +----------------+
                                    +-------->| Gem Repository |
                            Tests pass?       +----------------+

Stepping through this clearly amazing ASCII diagram we can follow a change “A1” from being committed to deployed:

  1. A developer runs tests locally (of course), and creates a commit A1
  2. The developer pushes A1 up to Gerrit and adds another developer as a reviewer.
  3. (Not diagrammed) Jenkins integrates with Gerrit, pre-testing the commit. Afterward, the other developer will review (and hopefully) okay the change.
  4. The originally developer will “submit” his change in Gerrit, which will replicate into the “Gold” GitHub repository (master branch).
  5. Jenkins learns that a new commit, A1, is available for the building and testing. Jenkins fires off a job which effectively just runs:

     % rake test
     % rake build
  6. If the job successfully completes, we auto-promote the gem to a “release” using the Promoted Builds plugin to execute some post-build promotion processes such as:
    • Keep this build forever
    • Publish the archived .gem artifact over SCP to the Gem repo with the SCP Plugin
    • Fire off another job which updates the Gem repo’s indexes
  7. At the end of this pipeline we typically will update another project’s Gemfile.lock and then go through a similar flow to get the updated gem used by downstream projects.

Conventions we follow

To enable this fairly standard workflow across the many gems we maintain, we have a couple of conventions:

  • Inspired by Semantic Versioning 2 all gemspecs typically have the following set as their version:

      gem.version = "1.2.#{ENV['BUILD_NUMBER'] || 'dev'}"

    This uses the BUILD_NUMBER environment variable exposed by Jenkins to ensure that all gems built by Jenkins have a “marker” allowing them to be traced back to a particular build. This has had the bonus effect of reducing the number of superfluous commits of “Bump gem version” unless we make meaningful API changes to warrant a minor version bump.

  • All gems are expected to enumerate all their dependencies in the Gemfile in the repo. This allows Jenkins jobs to self-bootstrap using a job-specific RVM gemset for the gem build.
  • All gems are required to define the build and test Rake tasks. Naturally this means they also must include a Rakefile. More often than not the Rakefile looks just like this:

      require 'bundler/gem_tasks' # Provides `build` task
      require 'rspec/core/rake_task'
      task :default => :test

With the above workflow and these conventions in place, we’ve reaped a couple of benefits, the most obvious one has been the severely reduced time it takes for new gems to be created and incorporated into production systems.

At a higher level, we’ve gained more confidence in our gems with this emphasis on testing, traceability and code review baked into the process from the start.

That confidence pays dividends when looking at moving larger, more complex systems towards continuous deployment, which we’ll cover in later postings.

- R. Tyler Croy

get reset to zero when the minor or major versions get bumped. I am personally okay with this given the added benefit of “deep traceability” that the build number from Jenkins provides for gems referenced in production.

  1. This has totally happened

  2. Contrary to what describes, the “patch number” does not

posted in: · · ·