Monkey business value
So I was hanging out with a bunch of geeks in Switzerland, having one of those late night conversations, and an idea sort of emerged, and the more I thought about it, the more I liked the idea. And then I was thinking that a) I’m useless at following through on ideas and b) I would love someone to take this forwards. So here it is.
Our premise was that the value of automated testing is in its repeatability and low investment (in terms of human effort). However, running the same tests all the time just verifies that the system does a small number of well-known activities.
The value of exploratory testing, on the other hand, is to try all the weird combinations that people might try – aka monkey testing – in order to break the system by using it in unlikely ways. The problem is that exploratory testing requires people, so it’s slow and expensive.
Cue RMonkey (and possibly PyMonkey and JMonkey). RMonkey augments your Ruby acceptance tests, to emulate monkey behaviour. It works like this:
Boring, old-style test:
def test_user_can_login
web.navigate_to '/login'
web.enter_value :name, 'bob'
web.enter_value :password, 'secret'
web.press_button :login
assert_that web.current_page, has_title("Welcome")
end
Now we monkey it up, with the keywords usually, sometimes and rarely:
require 'rmonkey'
include RMonkey
def test_user_can_login
navigate_to '/login'
usually { web.enter_value :name, 'bob' }
sometimes { web.enter_value :name, 'kate' }
rarely { web.enter_value :name, random_string() }
usually { web.enter_value :password, 'secret' } # sometimes don't bother
web.press_button :login
sometimes { 10.times { web.press_button :login } } # bored bored bored!
assert_that web.current_page, has_title("Welcome")
end
So now you have automated monkey tests. There are default probabilities for usually/sometimes/rarely (say 80%, 15%, 5%) but that is customizable:
monkey_see :usually => 75, :sometimes => 22, :rarely => 3
The interesting part, of course, is to know what sequence of events leads to a test failure. A more fully-featured RMonkey would keep track of which events it executed and produce an informative narrative if things turned bad.
In the spirit of the infinite monkeys metaphor, this would prove most useful in a soak-testing scenario, whereby thousands of monkeys were hitting an application over an extended period of time. You would want to know that a) mostly it worked, and b) when it didn’t work, which sequence of events caused it to break.
Some obvious applications for RMonkey would be in driving Selenium or Watir, or using JRuby to drive JUnit, MarathonMan, WebDriver or any of the other Java testing frameworks.
The code for rmonkey.rb is simply:
module RMonkey
LIKELIHOOD = { :usually => 80, :sometimes => 15, :rarely => 5 }
def usually
yield if rand(100) < LIKELIHOOD[:usually]
end
def sometimes
yield if rand(100) < LIKELIHOOD[:sometimes]
end
def rarely
yield if rand(100) < LIKELIHOOD[:rarely]
end
def monkey_see(likelihood)
LIKELIHOOD.merge(likelihood)
end
end
So anyway, let me know if you would like to get involved in developing RMonkey. I think it’s a really appealing idea and I’d love to see someone do something with it.
This looks formidable! I’ll give it a go on my little pet project :)
I’d love to get involved – monkey testing. I love monkeys.
I’ll give it a whirl…..
Dan,
Testing your code to prevent code entropy and to scope the development task is valuable. Writing tests to replace exploratory testing is not.
Monkey testing would give another excuse for developers to code for functionality which has not really been asked for. Also, isnt it faster to just do the random exploratory testing than writing tests for it ? And if those tests are really important, I would argue that you would need them in your story acceptance criteria as main tests.
Just my $0.02 for what its worth !!
regards,
Prashant
I agree. The purpose of RMonkey isn’t to replace exploratory testing, it’s to simulate realistic soak testing. The randomness allows you to run a large number of similar-but-different, authentic paths through an application. You would be looking for things like deadlocks or timeouts – things that are difficult to detect without load or soak testing, but where lots of instances of identical paths might not show up an error.
It’s then up to the testers to determine whether it’s a defect that should be fixed or whether it’s acceptable behaviour and enough of a rarity not to care. So the programmers wouldn’t invest effort unless it was deemed to be worth it. Or something like that anyway – as I said, it was late at night :)
Have you checked out OpenSTA and other load test apps? If I understand the philosophy correctly, I think some percentage of monkey factor is usually baked into these kinds of tests (eg. 83% of users will follow the defined path, while 17% will go off and do something wierd) to give the load testers a more realistic scenario. Cool stuff, there might be some good ideas in there that can be aped (tee hee!) for acceptance testing.
[...] The developer tuts, “it’s your own fault” he says, “you broke it with your impatience”. And that is why Dan North’s recent blog about Monkey Testing fills me with happiness. Testing software for my monkey behaviour, so that it doesn’t break when I do things that I’m not supposed to do – because I am human. [...]
carnival of the agilists, 1-mar-07…
This issue of the carnival consists entirely of Britblogs!...
This looks like some of the work being done on testing at Microsoft, although they have gotten much further than you suggest. http://research.microsoft.com/Pex/
I suppose stochastic monkey testing is good and effective. But giving it a sense of direction is also good with the help of model of the application. I am planning for a model based testing tool using Ruby. I have already tried this with Python and was succesful.
Stochastic monkey testing is good but it also better to give it a sense of direction using model. I am currently working on the model based testing tool using Ruby. I had tried this with Python.