Jasmine and Jenkins Continuous Integration

July 28, 2013 6 Comments

I use Jasmine as my JavaScript unit/behavior testing framework of choice because it’s elegant and has a good community ecosystem around it. I recently wrote up how to get Jasmine-based autotesting set up with Guard, which is great for development time testing, but what about continuous integration?

Well, it turns out that it’s pretty difficult to get Jasmine integrated with Jenkins. This is not because of an inherent problem with either of those two, it’s just that no-one got around to writing an open source integration layer until now.

The main problem is that Jasmine tests usually expect to run in a browser, but Jenkins needs results to be exposed in .xml files. Clearly we need some bridge here to take the headless browser output and dump it into correctly formatted .xml files. Specifically, these xml files need to follow the JUnit XML file format for Jenkins to be able to process them. Enter guard-jasmine.

guard-jasmine

In my previous article on getting Jasmine and Guard set up, I was using the jasmine-headless-webkit and guard-jasmine-headless-webkit gems to provide the glue. Since then I’ve replaced those 2 gems with a single gem – guard-jasmine, written by Michael Kessler, the Guard master himself. This simplifies our dependencies a little, but doesn’t buy us the .xml file functionality we need.

For that, I had to hack on the gem itself (which involved writing coffeescript for the first time, which was not a horrible experience). The guard-jasmine gem now exposes 3 additional configurations:

  • junit – set to true to save output to xml files (false by default)
  • junit_consolidate – rolls nested describes up into their parent describe blocks (true by default)
  • junit_save_path – optional path to save the xml files to

The JUnit Xml reporter itself borrows heavily from larrymyers‘ excellent jasmine-reporters project. Aside from a few changes to integrate it into guard-jasmine it’s the same code, so all credit goes to to Larry and Michael.

Sample usage:

In your Guardfile:

guard :jasmine, :junit => true, :junit_save_path => 'reports' do
  watch(%r{^spec/javascripts/.+$}) { 'spec/javascripts' }
  watch(%r{^spec/javascripts/fixtures/.+$}) { 'spec/javascripts' }
  watch(%r{^app/assets/javascripts/(.+?)\.(js\.coffee|js|coffee)(?:\.\w+)*$}) { 'spec/javascripts' }
end

This will just run the full set of Jasmine tests inside your spec/javascripts directory whenever any test, source file or asset like CSS files change. This is generally the configuration I use because the tests execute so fast I can afford to have them all run every time.

In the example above we set the :junit_save_path to ‘reports’, which means it will save all of the .xml files into the reports directory. It is going to output 1 .xml file for each Jasmine spec file that is run. In each case the name of the .xml file created is based on the name of the top-level `describe` block in your spec file.

To test that everything’s working, just run `bundle exec guard` as you normally would, and check to see that your `reports` folder now contains a bunch of .xml files. If it does, everything went well.

Jenkins Settings

Once we’ve got the .xml files outputting correctly, we just need to tell Jenkins where to look. In your Jenkins project configuration screen, click the Add Build Step button and add a “Publish JUnit test result report” step. Enter ‘reports/*.xml’ as the `Test report XMLs` field.

If you’ve already got Jenkins running your test script then you’re all done. Next time a build is triggered the script should run the tests and export the .xml files. If you don’t already have Jenkins set up to run your tests, but you did already set up Guard as per my previous article, you can actually use the same command to run the tests on Jenkins.

After a little experimentation, people tend to come up with a build command like this:

bash -c ' bundle install --quiet \
&& bundle exec guard '

If you’re using rvm and need to guarantee a particular version you may need to prepend an `rvm install` command before `bundle install` is called. This should just run guard, which will dump the files out as expected for Jenkins to pick up.

To clean up, we’ll just add a second post-build action, this time choosing the “Execute a set of scripts” option and entering the following:

kill -9 `cat guard.pid`

This just kills the Guard process, which ordinarily stays running to power your autotest capabilities. Once you run a new build you should see a chart automatically appear on your Jenkins project page telling you full details of how many tests failed over time and in the current build.

Getting it

Update: The Pull Request is now merged into the main guard-jasmine repo so you can just use `gem ‘guard-jasmine’` in your Gemfile

This is hot off the presses but I wanted to write it up while it’s still fresh in my mind. At the time of writing the pull request is still outstanding on the guard-jasmine repository, so to use the new options you’ll need to temporarily use my guard-jasmine fork. In your Gemfile:

gem 'guard-jasmine'

Once the PR is merged and a new version issued you should switch back to the official release channel. It’s working well for me but it’s fresh code so may contains bugs – YMMV. Hopefully this helps save some folks a little pain!

Filed under Examples, javascript Tagged with , , , , , , , , ,

Autotesting JavaScript with Jasmine and Guard

June 15, 2013 3 Comments

One of the things I really loved about Rails in the early days was that it introduced me to the concept of autotest – a script that would watch your file system for changes and then automatically execute your unit tests as soon as you change any file.

Because the unit test suite typically executes quickly, you’d tend to have your test results back within a second or two of hitting save, allowing you to remain in the editor the entire time and only break out the browser for deeper debugging – usually the command line output and OS notifications (growl at the time) would be enough to set you straight.

This was a fantastic way to work, and I wanted to get there again with JavaScript. Turns out it’s pretty easy to do this. Because I’ve used a lot of ruby I’m most comfortable using its ecosystem to achieve this, and as it happens there’s a great way to do this already.

Enter Guard

Guard is a simple ruby gem that scans your file system for changes and runs the code of your choice whenever a file you care about is saved. It has a great ecosystem around it which makes automating filesystem-based triggers both simple and powerful. Let’s start by making sure we have all the gems we need:


gem install jasmine jasmine-headless-webkit guard-jasmine-headless-webkit guard \
 guard-livereload terminal-notifier-guard --no-rdoc --no-ri

This just installs a few gems that we’re going to use for our tests. First we grab the excellent Jasmine JavaScript BDD test framework via its gem – you can use the framework of your just but I find Jasmine both pleasant to deal with and it generally Just Works. Next we’re going to add the ‘jasmine-headless-webkit’ gem and its guard twin, which use phantomjs to run your tests on the command line, without needing a browser window.

Next up we grab guard-livereload, which enables Guard to act as a livereload server, automatically running your full suite in the browser each time your save a file. This might sound redundant – our tests are already going to be executed in the headless webkit environment, so why bother running them in the browser too? Well, the browser Jasmine runner tends to give a lot more information when something goes wrong – stack traces and most importantly a live debugger.

Finally we add the terminal-notifier-guard gem, which just allows guard to give us a notification each time the tests finish executing. Now we’ve got our dependencies in line it’s time to set up our environment. Thankfully both jasmine and guard provide simple scripts to get started:


jasmine init

guard init

And we’re ready to go! Let’s test out our setup by running `guard`:


guard

What you should see at this point is something like this:

guard-screenshot

Terminal output after starting guard

We see guard starting up, telling us it’s going to use TerminalNotifier to give us an OS notification every time the tests finish running, and that it’s going to use JasmineHeadlessWebkit to run the tests without a browser. You’ll see that 5 tests were run in about 5ms, and you should have seen an OS notification flash up telling you the same thing. This is great for working on a laptop where you don’t have the screen real estate to keep a terminal window visible at all times.

What about those 5 tests? They’re just examples that were generated by `jasmine init`. You can find them inside the spec/javascripts directory and by default there’s just 1 – PlayerSpec.js.

Now try editing that file and hitting save – nothing happens. The reason for this is that the Guardfile generated by `guard init` isn’t quite compatible out of the box with the Jasmine folder structure. Thankfully this is trivial to fix – we just need to edit the Guardfile.

If you open up the Guardfile in your editor you’ll see it has about 30 lines of configuration. A large amount of the file is comments and optional configs, which you can delete if you like. Guard is expecting your spec files to have the format ‘my_spec.js’ – note the ‘_spec’ at the end.

To get it working the easiest way is to edit the ‘spec_location’ variable (on line 7 – just remove the ‘_spec’), and do the same to the last line of the `guard ‘jasmine-headless-webkit’ do` block. You should end up with something like this:


spec_location = "spec/javascripts/%s"

guard 'jasmine-headless-webkit' do
watch(%r{^app/views/.*\.jst$})
watch(%r{^public/javascripts/(.*)\.js$}) { |m| newest_js_file(spec_location % m[1]) }
watch(%r{^app/assets/javascripts/(.*)\.(js|coffee)$}) { |m| newest_js_file(spec_location % m[1]) }
watch(%r{^spec/javascripts/(.*)\..*}) { |m| newest_js_file(spec_location % m[1]) }
end

Once you save your Guardfile, there’s no need to restart guard, it’ll notice the change to the Guardfile and automatically restart itself. Now when you save PlayerSpec.js again you’ll see the terminal immediately run your tests and show your the notification that all is well (assuming your tests still pass!).

So what are those 4 lines inside the `guard ‘jasmine-headless-webkit’ do` block? As you’ve probably guessed they’re just the set of directories that guard should watch. Whenever any of the files matched by the patterns on those 4 lines change, guard will run its jasmine-headless-webkit command, which is what runs your tests. These are just the defaults, so if your JS files are not found inside those folders jus update it to point to the right place.

Livereload

The final part of the stack that I use is livereload. Livereload consists of two things – a browser plugin (available for Chrome, Firefox and others), and a server, which have actually already set up with Guard. First you’ll need to install the livereload browser plugin, which is extremely simple.

Because the livereload server is already running inside guard, all we need to do is give our browser a place to load the tests from. Unfortunately the only way I’ve found to do this is to open up a second terminal tab and in the same directory run:


rake jasmine

This sets up a lightweight web server that runs on http://localhost:8888. If you go to that page in your browser now you should see something like this:

livereload-screenshot

livereload in the browser – the livereload plugin is immediately to the right of the address bar

Just hit the livereload button in your browser (once you’ve installed the plugin), edit your file again and you’ll see the browser automatically refreshes itself and runs your tests. This step is optional but I find it extremely useful to get a notification telling me my tests have started failing, then be able to immediately tab into the browser environment to get a full stack trace and debugging environment.

That just about wraps up getting autotest up and running. Next time you come back to your code just run `guard` and `rake jasmine` and you’ll get right back to your new autotesting setup. And if you have a way to have guard serve the browser without requiring the second tab window please share in the comments!

Filed under Examples, javascript Tagged with , , , , , ,

%d bloggers like this: