Monkey business value
So I was hanging out with a bunch of geeks in Switzerland, having one of those late night conversations, and an idea sort of emerged, and the more I thought about it, the more I liked the idea. And then I was thinking that a) I’m useless at following through on ideas and b) I would love someone to take this forwards. So here it is.
Our premise was that the value of automated testing is in its repeatability and low investment (in terms of human effort). However, running the same tests all the time just verifies that the system does a small number of well-known activities.
The value of exploratory testing, on the other hand, is to try all the weird combinations that people might try – aka monkey testing – in order to break the system by using it in unlikely ways. The problem is that exploratory testing requires people, so it’s slow and expensive.
Cue RMonkey (and possibly PyMonkey and JMonkey). RMonkey augments your Ruby acceptance tests, to emulate monkey behaviour. It works like this:
Boring, old-style test:
def test_user_can_login
web.navigate_to ‘/login’
web.enter_value :name, ‘bob’
web.enter_value :password, ’secret’
web.press_button :login
assert_that web.current_page, has_title("Welcome")
end
Now we monkey it up, with the keywords usually, sometimes and rarely:
require ‘rmonkey’
include RMonkey
def test_user_can_login
navigate_to ‘/login’
usually { web.enter_value :name, ‘bob’ }
sometimes { web.enter_value :name, ‘kate’ }
rarely { web.enter_value :name, random_string() }
usually { web.enter_value :password, ’secret’ } # sometimes don’t bother
web.press_button :login
sometimes { 10.times { web.press_button :login } } # bored bored bored!
assert_that web.current_page, has_title("Welcome")
end
So now you have automated monkey tests. There are default probabilities for usually/sometimes/rarely (say 80%, 15%, 5%) but that is customizable:
monkey_see :usually => 75, :sometimes => 22, :rarely => 3
The interesting part, of course, is to know what sequence of events leads to a test failure. A more fully-featured RMonkey would keep track of which events it executed and produce an informative narrative if things turned bad.
In the spirit of the infinite monkeys metaphor, this would prove most useful in a soak-testing scenario, whereby thousands of monkeys were hitting an application over an extended period of time. You would want to know that a) mostly it worked, and b) when it didn’t work, which sequence of events caused it to break.
Some obvious applications for RMonkey would be in driving Selenium or Watir, or using JRuby to drive JUnit, MarathonMan, WebDriver or any of the other Java testing frameworks.
The code for rmonkey.rb is simply:
module RMonkey
LIKELIHOOD = { :usually => 80, :sometimes => 15, :rarely => 5 }
def usually
yield if rand(100) < LIKELIHOOD[:usually]
end
def sometimes
yield if rand(100) < LIKELIHOOD[:sometimes]
end
def rarely
yield if rand(100) < LIKELIHOOD[:rarely]
end
def monkey_see(likelihood)
LIKELIHOOD.merge(likelihood)
end
end
So anyway, let me know if you would like to get involved in developing RMonkey. I think it’s a really appealing idea and I’d love to see someone do something with it.
