All oracles are failable, when are you confident enough?

Recently I had the opportunity to participate in Rapid Software Testing (game master Michael Bolton) and the acclaimed dicegame. I also had the chance to be game master on a variation of the dicegame session for a small test team. Reflecting on the experiences I had two considerations: (Some spoilers apply) 


When are you confident enough?

The dicegame is played by a loop of theories/ideas and tries/tests on the idea. The goal is to produce a theory/algorithm that can succesfully predict the number that the game master presents. How many tries/tests/checks would give you confidence in the theory you have in mind? Options:

  • When you succesfully predict one throw. Ie you say 7, game master say 7. Do you yell "LOOSERS, Sea ya"?
  • When you have 7 succesfully predicted in a row? (why 7)
  • All 7776 combitations of 5 pieces of 6-sided one-colored standard dices?
  • Do you try every throw just once, or more times?
  • Would you know if every trial number divisible by 100, the game master would say "pi" (think leap years)


The dice game seems simple, but the problem domain of even the dice game is infine. Or at least practically infinite (7776 is practically infinite in the dice game IMHO). James Bach replied to my tweet that "The number of tests doesn't matter, but the character of the test, relative to your mental models, matters a great deal.". My purpose is not to find a fixed number of tries, but to make you consider the underlying assumption on confidence levels. That is you have a confidence in your model until it fails. You are confident to the level of x succesfull predictions, where the x+1 prediction fails. All you know at that time is that your theory is "incomplete" (not wrong, not right) - and this calls for more learning and more ideas...


All oracles are failable

The oracle in software projects is the ressource of answers - the documents, the mindmap, the subject matter expert. In the dice game the game master is the oracle (ie Michael Bolton, James Bach or yours humbly). We are humans hence failable. The physical oracles (docs, ...) even more. This made me ponder:

  • Would you approach the dice game differently actively knowning that the dice game master is failable?
  • If the game master (aka oracle) made a deliberate error ever once in a while - would you know?
  • If there is a bug (non deliberate) in the game masters algorithm would you know?
  • How would you test for the oracle making mistakes?
  • Do you test the dice game different if it was a human or machine oracle
    • First think about the dice game being computer based
    • Secondly what if it was a human behind a computer based interface
    • Consider the implications of the Turing Test
  • Oh, did you forget that I could make mistakes? - was that a rule, or an assumption?

Michael Bolton replied to my tweet that "In the dice game, we need to be prepared to deal with any test that a tester offers. Is it a bug? That depends on the framing." The key framing of the dice game is usually a lesson in learning, in theory setting and trials of that theory - still under the underlying assumption that the game master can deal with any test that is offered. What would happen if the game master was blindfolded? What would the case if the algorithm was more complex - less humanly processable in a short time. There will always be a level of capability of the oracle, and it will fail - eventually.


/Jesper @jlottosen

Views: 424

Add a Comment

You need to be a member of Software Testing Club - An Online Software Testing Community to add comments!

Join Software Testing Club - An Online Software Testing Community

Comment by Jesper Lindholt Ottosen on November 4, 2011 at 22:05

A paper article (!) I read  compared the tools of the judges in American football and European football. The later (EU) have three subjective people on the field to view the charades and unpredictability. The former (US) objective cameras, expert panels to watch the tactics and handle the complaints. Yet for all the data analysis the data doesn't always provide the complete information to make "final" decisions. So in both sports everyone complains about the judge. Objectiveness is approximated by the sampling method - and as above the oracles are fail-able to confidence level.


Any system, model or solution is an abstraction. We must do our best to dig into it, go around it and go above it - to understand how it works and how not. Being aware is half the journey - knowing is 75% (Credits Elisabeth Henrickson).


© 2017   Created by Rosie Sherry.   Powered by

Badges  |  Report an Issue  |  Terms of Service