Four Questions to Ask Yourself for Better Bug Reports
What makes a good bug report, how to go beyond that and why.
An important side-effect of exploring, experimenting with and learning about the software under test is the discovery of possible bugs. In most cases, these findings must be documented in some way to be discussed, analysed, prioritised and solved.
This post provides questions you can ask yourself to make your bug reports as useful as possible and other questions that can help you to learn and improve.
Can other people reproduce the bug?
The more information about your actions and observations you provide, the higher the chance that somebody else can reproduce the problem you ran into. Additionally, when the bug is fixed, another person can re-test it.
Well documented bugs enable developers and testers to work more independently.
Here is information you might like to add to help you with that.
Environment
It may be useful to document the environment you tested on if you have multiple backend environments. The problem could be related to test data or infrastructure, which can be environment-related.
The same is true for client-side applications like mobile and web. Problems could be related to the operating system, browser, or anything else the client runs on.
Version
It can take some time from the moment you discover the bug to the moment the bug is fixed. Days, weeks or months. This may lead to a different behaviour. Documenting the version or commit helps to understand what changes were introduced since the bug was discovered and if these changes may have fixed the problem or changed the behaviour.
Test Data
The problem could be related to the specific test data you have used. Document how to create this data, where to find it or how to use it.
You can also try to reproduce the problem with other test data. This can be a first opportunity to narrow down the possible cause.
Steps
Write down what you did. Step by step.
It is not the most exciting thing to do. But sometimes the small details matter. By documenting the actions that led to the problem, you increase the chances that someone else will understand what you did and be able to do it as well.
You can then try to reproduce the problem by going through your description to see if you have missed anything. And you also make sure that the issue you have found is reproducible.
Evidence
You can also attach screenshots, GIFs or videos. This is especially useful for GUI-related problems. For non-GUI-related problems, attach logs from the browser’s web developer console, requests and responses from HTTP requests and server logs.
It is important not to overdo it. Ask yourself: What kind of attachment is really helpful? Which of the thousands of log messages are related to your documented steps? Too much information can be overwhelming and distract from the actual problem. Cut it down.
Documenting the environment, version, test data and steps, and attaching evidence are the boring but necessary basics. Let's move on to more exciting things.
Is the impact clear?
It is common to document the actual and expected behaviour in bug reports. I am no fan of this, for reasons that I might elaborate in another post.
Instead, I document my observations, potential impacts and risks. For impacts and risks, I focus on customers, the business and the technical side of things.
It is also a good idea to check for workarounds. Is important functionality completely blocked, or are there ways around the problem?
This information helps other people to prioritise your findings and make decisions.
Can I learn something?
Finding a problem is an opportunity to learn something new about the software, the business and the technical aspects.
There was a time in my younger days when I was really into stories about detectives and even considered doing this professionally. Bugs give me a chance to put on a detective hat and investigate. For me, this is where the fun begins.
You can take a closer look at the server’s log message. This will help you to better understand what the backend code is exactly doing. The same applies to HTTP requests and responses.
By doing so you might be able to find information like stack traces or exceptions that give the developers a first direction. Not only will you find the problem, you may also find the cause.
You can also check the commit history’s messages and see if any recent changes might have introduced the problem.
You are already interrupted in your testing. You are already familiar with the problem. Keep going as far as you can. This is much more efficient than stopping and leaving it to the next person who has to start from zero.
If you have the skills or the willingness to learn it, you can attach a debugger and check which lines of code the software is running and what the values of the different variables are.
Everything you learn during your investigation can be helpful information that can be added to your bug report. And it can inspire you to find new test ideas - now and in the future.
Can I fix it myself?
If you go as far as looking into the code and attaching a debugger, you might as well attempt to fix the problem yourself.
When there is a trustworthy code review process and valuable automated tests in place, there is little to worry about. You will make mistakes. But you will learn from them. And in the long term, you can contribute more.
This is not really about the bug report itself. But I would like you to understand that your contribution can go much further than finding the problem, because I think these extra steps will make you a better tester.
After some time off I am happy to be back and bring new posts to you.
Happy investigation,
Florian