From test cases to mind maps: a personal experience

The trigger of this post was a comment on my other blog, asking for the results of a research I did on test case management tools at the beginning of 2011. This research was stopped after some months, as I progressively realised that a test case management tool wouldn’t fit my current context, but I never officially concluded it. I am not concluding it with this post, maybe I resume it in the future, but two years after this research began, the fact is that we’ve lived and tested properly without test cases or test case management tools. This is how we did it.

Before I start, I am not trying to say what’s best or not regarding this topic, this is just a personal experience, a case study if you want. Just food for thought, something to consider if you are in this situation too. Hope this helps :-)

 

Our context

I’ll try to summarize our context in a few lines, contact me if it is not enough to get the whole picture of the situation:

  • 2 web-based applications to test

  • Both are mature applications, with more than 3 years of existence each

  • +500 use cases (and growing!), ranging from simple CRUD ones to huge and complex internal system integration cases with several conditional interpretations and business logic rules implied

  • 2 lone-wolves testing department, paired up with 4-5 supersmart developers

  • An average of 2-3 releases per month

  • No automation available (at the beginning of that research)

  • ...

The first number that came to my mind was that we needed a tool that managed at least +500 test cases (say, the happy path of each use case), so we were supposed to develop +500 test cases, and we were supposed to maintain those +500 test cases too, without hiring new teammates or slowing down the release velocity we had.

We couldn’t take that investment, our focus would have switched from testing to actually test case creating and maintaining, feeling also this would become an endless task, as there are new use cases every month, plus revisits of older ones, implying new test cases to write and old test cases to maintain, plus at some point exploring unhappy paths if time was available.

 

What were we doing at that time?

At the very beginning, two years before the research started, I tried to create the test cases in Excel format at the beginning of the testing phase (we were doing waterfall at that time) before testing hands-on, and got the following learnings:

  1. Test cases evolve while testing, so your mindset and the tool you use to manage those cases has to be ready for that.

  2. Test cases maintenance and reutilisation is a hot topic, hard to handle in a fast-changing environment like ours, for reasons stated above.

With this approach, I found that I invested more time creating, editing and adapting to current context my Excel test cases than testing. Progressively, I abandoned ideas; first, I abandoned the idea of reusing test cases (so the maintenance issue disappeared as well); and then, I abandoned the idea of preparing a whole set of test cases before starting testing, embracing a more dynamic test design and execution. I was doing exploratory testing without knowing it, but felt uncomfortable with the idea of having no test documentation. That’s why I started researching for a test case management tool.

 

And... Mind mapping enters the building!

While researching for specific tools, I also tried to find out what others in similar situations were doing, then I discovered that exploratory testing was a real thing, even highly respected by several influential testers; and then discovered mind mapping as well. Right after that, I tried to switch the use of Excel and Notepad in my unofficial exploratory activities to start using Xmind as a support tool to perform those.

I had an “ah-ha moment” the very first time I did this, as mind mapping is the perfect tool to develop and execute tests while testing, evolving your test ideas while actually testing, having lots of fun and empowering the creative side of testing, which also felt rewarding. In addition to this, using mind mapping allowed us to start developing a “test ideas mind map” at the very beginning of new projects and iterations and evolving it while we dug deep into their content, sharing it with the development team and the stakeholders, achieving a huge test ideas compilation just before starting testing hands-on, that could evolve as well as the testing took place.

 

The research for a test management tool ended de facto...

...the moment I asked myself ¿why do you want to invest such amount of time in scripting those test cases when you are delivering reasonably well without them? The main answers that came into my mind referred to test execution records, coverage and reutilisation. So I analyzed these answers deeply.

To have some test execution records is not enough reason to implement a test case management tool, so I kept digging in the other answers.

About coverage... we are novice in exploratory testing, so I was unsure about leaving important areas untested in such a dynamic approach, but the idea of getting a test case management tool was not enough for that. At that point I preferred a simple “then get better at exploratory testing!”, and, on the other side, feeding those tools would have left us less time to test effectively, so coverage would be hurt as well. Also, what coverage offers you a test case management tool? The coverage over the test cases you have been able to get into it, which is a biased way of looking into coverage. I felt more confident in growing coverage on important areas as we tested.

Reutilisation has been a recent milestone for us, deciding to not reuse anything so far, in order to recreate and review test ideas when needed, sharpening our senses and exploratory skills at every iteration, but we will probably enhance this conclusion soon...

As you can read, we found no reason to move our current fruitful effort to a scripted approach. Also I was afraid of us falling in the “passed test cases false comfort”, confusing “scripted test execution finished” with “testing finished”, and decided not to be tempted about it.

 

Next steps (some of them already executed)

Once we officially decided that we were doing exploratory testing with mind maps and that we were (and are!) happy about it, we started to iterate this idea to improve our performance. Things we have done already:

  • Investing some time in automate validation checks, to combine our exploratory in-depth performance with some general and light functional checking coverage in other areas.

  • Scripting some test cases for the most critical use cases we have (I am talking about 5 or so), creating a mini scripted test suite to be executed in every testing cycle, before installing a new version. This suite won’t be ever automated, as it takes 15 minutes to be executed and I want the 7 senses of a thinking tester into it, and a robot does not offer me that safety. Smoke tests are scripted too. These are simple proofs that exploratory and scripted testing can coexist.

Things we are bound to do in the near future:

  • Thinking again about reutilisation. Maybe we could develop some mind map checklists to start the test ideas compilation, updating these documents as we use them, accepting they have to be updated before evolving them, but taking advantage of the knowledge and mental effort already executed.

  • Developing some sort of testing low-tech dashboard, in order to communicate easier the status of the testing effort to the team et altri.

     

Finale

This is it. A 4-year process summarised in less than 1500 words. I hope to be able to write some more about this in the following 4 years, let’s see if we get any better on this endless and fantastic subject!

Views: 3489

Add a Comment

You need to be a member of Software Testing Club - An Online Software Testing Community to add comments!

Join Software Testing Club - An Online Software Testing Community

Comment by Mauri Edo on December 19, 2012 at 17:57

@Margaret:

To me TestManagement is not just about creating and managing test cases, it is about how they can support the other artifacts of the test process.

Superb! Completely agree!

Also, I am starting to think as well on this "two worlds", that was the original idea of the writing, to get some different views, and yours and Kobi's made me get a bigger picture of the thing.

Besides, I confess I've never linked before mind-mapping and automation frameworks in my head, but it makes a lot of sense, thanks for pointing this out and for commenting again,

Cheers!

Mauri

Comment by Margaret Thomas on December 18, 2012 at 15:17

Ok I haven't read through the latest comments.  But I reread your blog and now have a better understanding.  I always enjoy how people "solve the equation" sort of speak :).  I always believe there is no one answer and it truly depends on the challenges before them.

During my interview process, I was introduced to test case modeling which is kind of similar to "mind mapping" the tests that you are doing.  I saw how it could fit into the overall testing process. 

To me TestManagement is not just about creating and managing test cases, it is about how they can support the other artifacts of the test process.  If you want to graduate to a ALM System then you can support all aspects of the product develop and release process.  So to limit the scope of using TestManagement for just creating and maintenancing tests I agree would not provide any value. 

One idea I can see is linking those "mind mapped" test to a test automation framework.  This is something I have seen at other companies.  These diagrammed tests can allow for exploratory testing and formulating the ideas.  Once you have concluded your diagram I see no reason why they cannot be managed by a ALM/TM system where they can easily share information between other artifacts such as configurations, releases, sprints, defects, etc.  Test run results are always available to the entire team on demand.  If you are automating, then you would have your TestFramework translate the diagram into usable test scenarios that can be done finitely or infinitely as well as serialized or randomized. 

I think it is like having the best of both worlds ...I won't go further as I typed/talked enough :).  However thank you again for sharing.  I think it is fascinating how you solved the problem for your challenge and it is a technique I see more people adopting.

Comment by Mauri Edo on December 18, 2012 at 14:42

@Kobi,

Thanks for enhancing the discussion, I am enjoying it very much :-)

Ok, I get your example on exporting, but when I talked about a dashboard, I didn't mean a highly specific one, with perfect numbers on completion of test cases, executions, and so on. I was thinking more on a low-tech dashboard, as put up by James Bach (http://www.satisfice.com/presentations/dashboard.pdf). I said yesterday that we don't need to report in high detail the progress of the testing tasks, and something like this will do perfectly, no databases needed ;-)

But I understand what you say, at some point, if you need to read data of test cases and plans and "do something", mind maps aren't the best option, but it's all about context!

Regarding your comment on XStudio, I feel we are at the same sort of point, but looking into it from opposite directions, mindmapping and TCM tools can be complementary, maybe they have to be so, as there are "things" that you can't cover just with one approach.

Thanks again for your thoughts,

Mauri

Comment by halperinko on December 18, 2012 at 7:25

@Mauri

I am playing the "Devil's advocate" to enhance this discussion :-),

As to exporting - what I meant was that in some case, sooner or later you might need to extract the data to another tool,

For instance as you mentioned - putting-up a dashboard.

When using a DB based system - it is normally much easier then extracting data from MM, Word or Excel documents.

I fully agree with your "light-weight" process demand - just saying that it can be achieved in means which might be easier for later adaptation and growth.

I am also trying to push to this direction on free tools such as XStudio (and some of it will probably come in its next 1.8 release).

Things like tabular "excel like" means of writing tests highlights, and some more colorful visualization on trees might allow us to enjoy best of both worlds.

@halperinko - Kobi Halperin 

Comment by Mauri Edo on December 17, 2012 at 19:59

@Kobi,


Thanks a lot for challenging my approach, that's where real learning lays! :-)


I agree with you that a Test cases management tool isn't equivalent to perform Scripted testing but... Isn't that its main goal? Okay, you can use a tool like this to just store test ideas and not use all details of test cases, taking advantage mainly of the tree view, but I preferred a lightweight approach that didn't need the installation, learning and feeding of a test cases management tool that we won't use at 100%.


Also, I tried to highlight that we are a small team with a huge workload above our heads, I chose the fastest option to have a "system" that was already proven to be working for us. Maybe if our context would have been different, my choice would have become another one... By the way, I am not against test cases management tools, I just explained our reasoning process taking into account our situation, maybe I reconsider this choice in the future, when I feel the context we are in adapts to it.


Again on the context, we don't need to export or report our testing as is, that's why I have the choice to develop some dashboard or not, and we can do it as we want, over our current "system". Maybe using some built-in features in a TCM tool will be more stable and full of features, but a lightweight approach will be adjusted perfectly to our needs at that time, and evolve with them as well.


I am looking forward to hear from you again after you sharing my thoughts with your local group:-)


Cheers!


Mauri

Comment by Mauri Edo on December 17, 2012 at 19:57

@Margaret:

Thanks for your comment, I'll be glad to hear your thoughts after rereading my words if you want to share them.


Cheers!


Mauri

Comment by halperinko on December 17, 2012 at 14:37

I must say I can't figure out why you use the term "test case management tool" as a way to say "scripted testing"?

You may as well use a free (or not) test management tool to record just the test ideas.

No one binds you to fill in all details, just like no one binds you to select different colors for mindmap objects...

It brings me back to a couple of discussions I raised before in that matter:

What made MindMapping better then just using the test management tool Tree structure?

When you do wish to export and/or elaborate test scripts - MindMap has less abilities than a Test Management system.

So why not use these tools in the 1st place - just with a Mind Set of evolving test cases?

Now you seem to be seeking means for dash boards, and eventually you are "wasting time" building your own test management system in stages, instead of benefiting from an existing tool, which will probably have more features and higher stability.

(There are some interesting ideas in this article - I hope to discuss that further on our local forum:  http://www.tapuz.co.il/forums2008/forumpage.aspx?forumid=936 (Sorry - not in English), and will try to write results back here... :-) )

@halperinko - Kobi Halperin

Comment by Margaret Thomas on December 16, 2012 at 20:35

I enjoyed reading your article and I have to reread it again.  I must admit, the technique is eluding me and it is hard to envision using this method for testing.  Thank you for sharing.

Adverts

© 2017   Created by Rosie Sherry.   Powered by

Badges  |  Report an Issue  |  Terms of Service