Welcome to my eleventh blog post. I’d like to turn our attention to test cases for a moment. Of late I’ve both read and heard many people within and outside the test community ponder whether these are in fact required (in this context, think ‘manual’). Well, this is my personal blog and I’ll give my answer to you straight. Yes. Of course you do. Why? Allow me to explain.
At the start of the year I joined a new team. Like all new teams, there are established ‘artefacts’ that routinely need to be tested. I could have said components. I could have said products. I could have said systems. Whatever. Let’s generically refer to items under test here as an artefact.
These could be legacy, well established artefacts that continue to be maintained/improved upon. There maybe new versions to be upgraded to, which, in turn may bring new features along with it. If you’re lucky the shiny new version may even address a few niggling long-standing bugs that have remained open for longer than originally anticipated.
Equally, you might be asked to test something completely different. Something entirely new. Something of which you’ve never clapped eyes on before.
Now here’s the rub. As a new team member there were several things I hadn’t clapped eyes on before. Whether it was something which had existed for a while or not. It may as well have been brand new to me. Since, as far as I was concerned, it was. What is it? What does it do? How do we go about verifying it’s behaviour? Are there any pre-requisites as far as the test environment or test data for example are concerned? I had lot’s of questions.
In fairness, the team have been great. Very supportive and parachuting in has given rise to plenty of verbal communication. However, we have a job to do. There’s work to be delivered. Time is always of the essence.
Thankfully, when it comes around to testing several of these artefacts, particularly when performing regression testing (since many of these have been around well before my arrival), I have the benefit of being able to reference their respective test suite(s) within our Test Management tool. This has proven to be an invaluable resource. These tests have allowed myself to become familiar with expected behaviours for a variety of scenarios and have provided a practical opportunity to exercise these tests with minimal support. For want of a better turn of phrase, they’ve enabled me to hit the ground running.
Before we go any further I am not saying you must always have a test case with definitive steps to follow. I’ve found a successful test strategy employs a proverbial smorgasbord of different elements. There has been many a time I’ve found a bug that was never originally conceived at requirement gathering stage or has been found as a direct result of running a specific test case. That said, I’ve also identified many bugs as a direct consequence of executing a set of test cases – I often find these either fail outright or you find something out of the ordinary as a consequence of running a particular test case. The test happened to have instructed you to perform x, y, and z which as it happens passes but you also found something unexpected happens if you slightly modify the steps. In which case, you might have been the first to realise this happened and you might choose to update the test case so this would always stand to be caught by others running the same test again in future.
I mentioned ‘others’ there. By others I mean your fellow compadres, comrades, colleagues, team members. If left to our own devices and regression test an artefact in an exploratory manner, there’s a chance that your test approach may fall foul of inconsistent test coverage. Somebody might simply forget to regression test a particular scenario or aspect of functionality. For my money, if you are at least executing the same set of test cases then this would go a long way to mitigate any such gaps in your coverage. Though, exploratory testing is very powerful. You or somebody may find something worth fixing as a direct consequence of not following any pre-defined steps. In my experience you need both.
I’ll be coming on to granularity next but just to recap – I would recommend you look to maintain a test suite but this must not blind you to thinking all you have to do is to run those tests and bingo. Testing as we all know is an art form. A skilled tester will always be on the look out for other scenarios to explore, etc along the way as part of their test execution cycle. The balancing act is ensuring you verify the high value stuff first and foremost. Don’t waste too much time and disappear down a rabbit hole trying to break something and supposing you do, there’s a real chance the steps to reproduce are so far out there on the edge of the universe that nobody cares.
So granularity then. I’ve opened some test cases, looked at the convoluted pre-requisites, glanced at the one billion steps, the way it’s written, language used, etc and sighed for about ten minutes. Not a good sign. Test cases ought to be easy to digest. Easy to understand. Preferably easy to run but that’s always in the eye of the beholder (only easy if you know how but a well written test case should make it easy to learn from). Then I’ve had test cases at the other end of the spectrum that literally contain one step or duplicate something you already did in the previous test case. It can be infuriating. So if you are not careful you could have a test suite which has a few enormous test cases within which take forever to plough through or have a test suite which has hundreds of tiny ‘baby’ test cases which only serve to inflate the number of individual tests needed to be run and also throw up challenges for how to maintain them.
Stop and think before putting your test suite together. Can you afford to consolidate a few tests where it makes sense to do so? Remember you can still do this and test exactly the same sorts of things that’d normally be executed individually but you’d stand to verify these in a much more efficient manner. This in turn makes it easier to manage from a maintenance perspective e.g. you may only need to edit one or two test cases as opposed to several. Care needs to be given to not end up with too much in a single test. Use your judgement and common sense when deciding.
Personally I like using bullet points. These have a nice way of making it clear what to do without feeling like you are shackled to them. You should still have that sense of freedom to explore further. Add notes of interest. Any ‘gotchas’ to be on the look out for. Nothing must detract from the whole point of running the test in the first place. It must verify expected behaviour and flush out as many issues as it possibly can. Some of these might require fixing. Some might not. Surface the information. Make your recommendations known.
Lastly, I find using test cases are a good way of radiating to the wider team what we intend to cover as part of our regression test effort. It’s proven to be of enormous interest to the development team to have visibility of this and offer their support when things invariably change and we are all looking to mitigate any perceived risk before release.