Deploying Metrics for Measuring Software Quality in Automation

Test automation has become the bread and butter of the testing industry. And even though its implementation is crucial, it can be difficult to measure its true value sometimes.

The reason why test automation has become such an instrumental driving force in app development is that it helps prevent testing from being a bottleneck in your software development lifecycle. Today, we’ll be discussing how certain metrics allow you to measure the impact of test automation. We’ll also be discussing how you can measure the costs of your testing and with the two metrics taken together, you’ll be able to do a cost-benefit analysis of overall test automation.

What’s wrong with traditional test metrics?

Trying to measure the effectiveness of testing is not something that’s new. Businesses have been trying to accomplish this for years at this point. The problem with their efforts till now is that their metrics have some significant downsides when trying to measure the impact of test automation. Let’s go over a few of these:

The number of tests. This one’s a classic. The easiest and most common metric for how much testing is being done: all you need to do is count how many tests you have. The more the better, right? Not always. Firstly, your tests will always need to evolve along with your software. Secondly, it may be better to join and mix several tests into one larger one. Thirdly, you’d want to avoid generating a false incentive to create tests purely for the reason to increase a metric.

The number of test runs. The number of test runs is actually good at keeping track of the number of testing you are doing. But the caveat with this is that it ignores whether that testing is effective, necessary, or efficient. Take this as an example, your teams could be performing smoke tests continuously and while that shows that a lot of testing is happening, it masks the fact that large portions of the application are getting no share of testing.

Defects found. The general consensus regarding this metric is that the more defects your team find, the more the team is good at testing. This is a really misplaced metric. Defects aren’t just reliant on the quality of the job testers are doing, but developers too. If your developers are producing good code, the number of defects should minimize with time as your code evolves. Another reason why you might not be finding any defects may be that your code is stable and mature.

Code coverage. This one’s a favourite among project managers. Code coverage measures how much of the code base is associated with unit tests. But this has two big problems. Trying to achieve 100% code coverage is impractical most times and it may also introduce a bias in developers to write tests that are designed to pass.

Automated test coverage. Today, testing companies aim for complete test automation but that may not be the right approach. While some tests do benefit from automation, others don’t. Take exploratory testing for instance or any other test that is run rarely. But that’s not to say that low test automation percentages are better than high. Ideally, your actual target should probably be nearer to 75% than 100%.

Measuring the impact of test automation

So, if these metrics alone are not any good, what do you have to be measuring instead? Well, there are two distinct sorts of testing to consider: regression testing and progression testing. In other words, one is about running your existing tests, the opposite is about creating new tests.

Measuring the effectiveness of regression testing

Regression testing involves running a subset of your existing tests to see whether the new code has introduced any bugs. It’s important to note that regression testing shouldn’t seek to rerun every single test every single time. Some tests will include duplicate steps, for example when a test must put your system during a certain state. More of a problem is how long it takes to run tests. Most test suites within the world contain thousands of tests. There’s no way you’ll run a lot of tests whenever you release a replacement feature. And in fact, an outsized proportion of your tests should be manual.

Which metrics are useful for measuring regression testing?

Test coverage: Sound familiar? It seems we will adapt this metric to make it more relevant. The key takeaway here is to identify where your tests are duplicating efforts. What you may also like to know is what proportion of your functionality comes under tests. Of course, this isn’t as simple as testing each functional element once, because you would like to check all the necessary conditions, like using different combinations of data or deliberately using bad data, etc.

The time needed to test: Also referred to as Speed of Testing, this metric measures how long it takes you to finish the specified tests for every release. The more tests you automate, the quicker you’ll get through them. Especially, if you employ a cloud-based system that permits you to run large numbers of tests in parallel.

Measuring progression testing

Progression testing happens for 3 important reasons:

When you have to create tests for the latest features.
To highlight steps to reproduce a reported bug.
For unplanned testing to be undertaken and find new bugs.

Measuring progression testing is tough. But here are a few of the metrics we’d recommend if you would like to know the impact of test automation.

Time to make new tests. One of the foremost important things is having the ability to make robust tests fast. Ideally, you’d want to automate these, especially for testing new features or reported bugs. So, you ought to measure how long each test takes to make. Your aim should be to scale back the typical time, without sacrificing test quality of course!

The ratio of bugs found vs bugs reported: The aim of progression testing is to seek out bugs before they’re actually reported by the end-user. This ratio measures how well you’re achieving this. If you discover more bugs before release than get reported by users, you’re doing well. It’s worth contrasting this with Defects Found. During this case, we aren’t trying to seek out more defects, we try to seek out those defects earlier. A stable and reliable defect tracking software can be the saving grace.

Although the jury is still out on the effectiveness of certain metrics used to gauge the efficacy of test automation, we believe that the ones mentioned in the article make for a solid baseline. With the problems plaguing the approach of metrics today, having a few surefire alternatives can benefit your automation testing greatly.

And if you’re looking for a surefire alternative to your existing test management solution, Kualitee is the tool for the task. Test management tools are used in tandem with automation tools to ensure that all your testing (manual and functional) can be easily managed from one place.