Hello again. Welcome to my third blog post. I’d like to muse about test automation. Yep, that old chestnut. I’ll be honest and say I fell into the world of testing. At the outset I was given a choice, a proverbial fork in the road, as to whether I wanted to pursue a career as a developer or as a tester. I didn’t deliberate for very long. Though I have a curiosity about programming and understanding how things work, my passion for testing and general aptitude for breaking things far outweighed any possibility of becoming a developer. I recall my dad telling me as a kid “Steven. You could break an iron ball.” implying I broke the sturdiest of toys with considerable ease and/or had a natural propensity for identifying problems. The stage, as they say, was set. Little did I know, I’d be using such skills as a part of my future career in software testing.
Oh yes, the title. I need to explain that one. By now you’ll have realised I like to use attention grabbing titles for my blog posts. This one is for all you Doctor Who fans out there. I often hear cries to automate something. Sometimes this can almost feel incessant, much like the motorised dustbins you see chasing the good Doctor and his faithful companion. Not even a set of stairs can fool them nowadays. That’s progress for you. Well, with the onset of continuous delivery, the cries for automation seem even louder than usual. You’ll also be familiar with the usual bun fight over how long you’ll need to undertake regression testing and vocal members of the team saying “Can’t we just automate this and save time?” with their eyebrows at 45 degrees. We manual testers are soooo slow aren’t we. There’s been times when no sooner have I started test execution, I’ve been asked whether I have finished yet. Face palm. You either want the confidence to know your software is behaving as expected before releasing to your audience or you don’t.
I’ll try not to rant here, but ‘Testing’ is a discipline like any other. Take ‘Programming’ for example (or ‘Coding’ or whatever else you wish to call it) as another discipline. Or ‘Business Analysis’ as another. I’ve rarely witnessed anybody pressurising for a developer to finish coding something or for a BA to finish writing user acceptance criteria in double quick time. Yet us testers sometimes get a raw deal. Whether following a waterfall methodology or not, we’re usually the penultimate ones who need to look at something before a stakeholder presses the big red button to deploy into the live environment. Typically, it can feel like the stakeholder is looking over your shoulder tapping their watch in an ever-so-unsubtle way (and coughing at the same time as muttering the words “let’s ship it”). We often feel the squeeze.
So yes, we want to release more frequently. Yes, we wish to develop and ship software like a well oiled machine. Yes, we want the audience to benefit from new and exciting features as soon as possible. We all do. Everybody in a software development team should all share these goals. Some of this is easier said than done however. Automation is not for the faint hearted. Another cliche for you would be to say ‘fools rush in’. Your team needs to give this a lot of thought. Think about what you say?
I’ll cut to the chase and say you are more than likely going to need both manual and automated test effort. Well, you can’t automate everything. Just as you can’t realistically test everything either. You could certainly try but you’ll soon come to realise that the effort employed far exceeds the value in trying to automate certain tests. Then there’s the whole human element to consider. As I explained earlier, some testers are naturally pre-disposed to identifying problems and breaking things. You can’t automate that. Machines (Daleks?) are great aren’t they for some things but they certainly have their flaws. You can’t automate experience, instinct, intuition, etc. These traits are what help testers identify defects a machine couldn’t possibly uncover. If I had a penny for every time I demonstrated a defect to somebody and be asked “how on earth did you do that?”. Writing scenarios, coming up with acceptance criteria, etc is great but there will be user journeys that haven’t necessarily been considered upfront. Then there’s the subtle nuances which affect behaviour across different platforms and devices. Something may work just fine using one browser/device but be completely screwed on another – would your automated tests always capture these instances or possibly give you a bumsteer if everything is showing as passing I wonder? Oh look everything is green. So false positives are also something to think about. What else is there to think about then? How about setup and maintenance. This is a real doozy. You’ll need to carefully consider which automated solution you opt for. Not only for expense in the conventional sense, but the cost of maintaining this for the forseeable. Automated tests are invariably brittle (particularly when testing at the UI level). Are you going to use live data or canned data (this can also lead to false positives)? What if something somewhere changes with or without your knowledge and breaks your valuable tests? Who is going to pick things up when they fall over? What if there are only a chosen few in the office who really understand how it all hangs together and they aren’t available? I mean they could have been taken ill or have left the company altogether – taking that precious knowledge with them.
Don’t get me wrong here. There is absolutely a place for test automation. There are benefits to be had for sure. Imagine having those mind numbingly tedious tasks removed by an automated solution freeing your time up to manually test the more complex stuff (the good stuff). In my experience, you need to weigh everything up and agree with your team what you definitely need to automate and what you definitely do not need to automate. The caveat here would also be to say it’d be wise to identify a middle ground as well. So maybe there are tests you’d like to automate but at a later date maybe. Start with the high value, straight-forward to automate type tests. The ones which are going to serve you by being run repeatedly against each new build version. My previous blog talked about high value smoke tests and these would be a good place to start.
Whether automated or manual. The whole point of testing is to find defects and exterminate them! If you’ll excuse me I need to find somewhere to park my TARDIS (I’m actually heading out for some lunch).