Having a solid test suite is instrumental to having confidence in your system, but how do you know if your test suite is good?
All metrics for evaluating the quality of a test suite have to be applied carefully; if taken to an extreme blindly, they all can cause harm. Common metrics include:
This last metric, called coverage, is what we will look at today.
Test coverage counts how much of the application code is used by the test suite. Consider the following example code:
def application_code(animal)
if animal == "cat"
puts "felines rock!"
elsif animal == "dog"
puts "canines are better"
end
end
def test
raise "Test failed" unless application_code("cat").include?("feline")
end
In this example, the test only tests the cat
side of things, the dog is left out. In terms of test coverage, this means that of the 7 lines of application code, 6 are being run by the test, giving 86% test coverage.
Test coverage gives you an easy, understandable metric for your test suite. I can easily say that a test suite with 1% coverage is almost certainly not well tested, whereas at 99% coverage, it almost certainly is. In addition to how much coverage you have, where you have coverage makes a big difference. If you are a bank, having 100% coverage on the finances section is much more valuable than 100% coverage on a “where’s the closes branch” feature.
When you have high coverage and a quality test suite, you can be more confident that the system works as specified. This increases the speed at which new features can developed and deployed.
Test coverage can easily be misleading:
def divide(a, b)
a / b
end
def test
raise "Test failure" unless divide(10, 5) == 2
end
In this test, we are getting 100% code coverage. Congratulations!
Just don’t pass 0 in for b
.
Or do 11 / 5 cause that is 2
instead of 2.2
And good luck if you pass in something that isn’t a number. Currently, the code will happily try to compute 11 / 🍎
.
Poorly written tests can increase the coverage metric without testing the code well. This leads to a false sense of confidence.
I typically recommend testing the critical sections of your code first. This is the checkout path for an e-commerce site, posting on social media, or searching on a knowledge-sharing service.
Having high coverage of this critical code means you can deploy with confidence, and make updates without worrying about spooky actions infecting your code.