At Lookout we find ourselves building more and more APIs and backend services these days. Naturally we would like to be certain that everything will work fine and dandy once it has been deployed. The reality of building out a service-oriented architecture is that you not only have to expect failure to happen, you have to plan and test for it.
As of late I’ve been using a tool called
Foreman for some projects to manage their
own “development stacks.” A single service might be composed of a
redis-server instance, a MySQL database and a Rails or
Managing this with Foreman is easy enough, I would create a
web: ruby app.rb redis: redis-server -c config/redis.conf mysql: ./script/run-mysql-ramdisk
When I run
foreman start, Foreman will manage bringing all of these services
online at once, then when you Ctrl-C
foreman it will bring down all of the
That’s great for simple local development and testing, but what about with integration testing the service?
Meet Test Engineer
The Test Engineer gem builds on top of Foreman and adds some basic testing functionality. Currently it’s only been used with Cucumber but it could easily be incorporated into other acceptance testing set ups.
With Test Engineer you can use your existing
Procfile to start and stop the
entire stack with
If you’re already using Cucumber, this becomes very easy to incorporate into
existing Features with the
@testengineer Feature: Log in to lookout.com In order to find or scream my phone As a registered Lookout user I should be able to log into the user area Scenario: With a valid email and password Given I am a registered user When I log in to Lookout Then I should see my devices listed And I should see my news feed
Test Engineer will bring up the entire stack defined in your
each and every scenario listed, providing a good isolated test environment for
your integration tests.
A note about test isolation: In the example
Procfile above I referenced
redis-server and a magic script to run MySQL on a ramdisk. When doing
integration testing with services like this it is absolutely critical to make
sure that the backing data stores for these services is flushed appropriately
between the scenarios/test cases. In this example, the
should be configured to disable AOF writes and snapshots for Redis, while the
run-mysql-ramdisk should unmount its ramdisk when the process is terminated.
Test Engineer also allows you to arbitrarily turn off services during the
scenario, which allows for some interesting fault tolerance testing. You can
define a simple step which invokes
Given /^the cache server is offline$/ do TestEngineer.stop_process('redis') end
Then in my Cucumber
.feature file I can turn off the redis service
mid-way through the test to verify a fault tolerance condition:
@testengineer Feature: Survive cache service degradation Scenario: Locate my device Given I am a registered user And I have an Android device And the cache server is offline When I locate my device Then my device should attempt to locate
That’s about all there is to integration and fault tolerance testing with Foreman and Test Engineer!
Errata: Test Engineer currently relies on some goofy hacking with some Foreman internals, which is part of the reason it cannot arbitrarily start a process after it has been stopped. I am currently working on making Foreman more easily embeddable with the author David Dollar.