Bug Reporting is a Science
•
Bronwyn Achemedei
I’m a former molecular microbiologist who made a career change into tech and landed a job as a mobile application software tester. When I began my job I knew nothing about app development or how to test. But, little did I know, my skills as a research scientist would help me be a better bug finder and reporter. Today, I’m going to explain how bug reporting is a science and why testers should consider adopting a scientific thought process.
1. Scientists are curious
Working for a company that creates various and diverse app technologies presents an amazing opportunity to be exposed to many different products, but the need to quickly switch between projects can be tricky to manage. The time we can devote to developing extensive test suites is limited, so it’s important to prioritise and know when to leverage exploratory testing. During testing, if I see behaviour that isn’t exactly what I expect, I drill down to find out more. Our team may not have time to write out every negative test and edge case that exists, so during testing I think of creative angles to cover all the weird and wacky ways a user may use the product.
Original comic by Corentin Penloup
You’d be hard pressed to find a scientist who isn’t interested in the ‘why’, and this is an attribute that will set testers apart from the pack. Finding bugs always excites me — I’ve found something that the developer missed and this is what I’m employed to do. But a tester’s job shouldn’t end there. When I find a defect, I then think about the why — why has this bug occurred? Is it due to a newly developed feature, or has it been flying under the radar in previous releases? Does the problem lie with the backend or frontend? Does it affect multiple operating platforms and devices?
If a tester is able to raise a bug report with answers to questions like these, not only will it save the developer time, but it will direct them where to start, and they will therefore be more likely to pick up the bug report and resolve it.
2. Scientists understand and apply the scientific method
Most of us learned the scientific method, or experimental methodology, years ago in high school, and that’s where we left it. But for scientists it’s a process ingrained in our day-to-day work. The scientific method can be boiled down into a few steps:
Make an observation
Ask a question
Form a hypothesis
Test the hypothesis
Iterate — if the hypothesis fails, form a new one and test it. (Wash, rinse, repeat)
Image credit xkcd
So, how can we apply this to testing? Imagine you’re testing a mobile app where users are able to make a booking. But when the Submit Booking button is tapped, the app crashes.
Some testers may immediately write up a bug report, “Submitting booking causes crash” but if we apply the scientific method we may be able to add valuable information to that report. Let’s break it down.
Observation: App crashes when Submit Booking is tapped
Question: Does this occur for all users? The app allows bookings to be made as a logged-in user and a guest user
Hypothesis: Given this test failed as a logged-in user, the bug will only occur for logged-in users
Test: Complete a booking as a guest user. This should not cause the crash
So we run the above test and discover that our hypothesis is correct. Great! Now our bug report can be more specific, “Logged-in user — submitting bookings causes crash”. Okay. Can we provide more information? Of course! There are many other questions that we can ask, running through the above process to narrow down the exact cause of the crash.
3. Scientists understand that correlation ≠ causation
Image credit xkcd
As I mentioned earlier, scientists are naturally curious and constantly ask questions in an attempt to answer the why, but they also know what kind of questions to ask. We’ve all heard it before, but it’s important to understand exactly what is meant by ‘correlation does not imply causation’. In terms of testing, this means not raising a bug report for “Booking — logged-in user submitting booking causes crash” without first confirming that this defect does not also occur for guest users.
To determine the root cause of an observation, the scientist (or tester) should have a good idea of all the variables involved. Having a thorough understanding of the product is necessary, and knowledge of the underlying code is a huge benefit. At Adapptor, the test team works closely with the developers and are able to brainstorm possible causes when testers hit a roadblock.
Let’s examine some potential variables in our booking example.
First consider the main system variables:
Is it operating system related? If the bug is occurring on an iPhone, run the same test on an Android device
If it is operating system specific, could it be related to the OS version? Run the test across devices with varied OS versions
Now think about the app:
Does the issue occur for all user types?
Could it be related to the type or place of a booking?
If it is only occurring for logged-in users, does it occur for users with different booking histories?
Might the cause be the backend fetching previous booking data to prevent overlapping bookings?
Is the booking still processed? Does it appear under upcoming bookings?
These are just a few of the many variables that could be involved. When working my way through them, I always start with the simplest explanation and then proceed down the list.
4. Scientists know when to call it quits
A situation familiar to many scientists: you’ve dedicated a large amount of time to an experiment but it won’t work. There comes a point in time where regardless of the effort applied it’s not viable to proceed. The concept of sunk cost highlights this perfectly. A good scientist will be continually evaluating if their methodology is worth continuing, or if it’s time to explore a new process.
I’ve explored the importance of investigating and gathering sufficient data to aid bug fixing. But testers need to be aware of the time spent doing this. The simplest way to do this is time-boxing—allocating time for exploratory testing and bug investigation.
Exploratory testing should always have purpose and scope, also known as a charter. Once a charter is designed, testers are able to assign it priority, based on how important the features involved are, and then allocate a time limit.
When investigating a bug, either within an exploratory or scripted test, a similar process can be applied. First, determine the severity of the bug. Then, consider the complexity and diversity of variables. Finally, decide how much time should be spent on investigation.
5. Scientists understand that no result is still a result
I recall a research project where I was certain I knew the reason behind my observation, but after more than a year’s worth of work, my research did not support my original hypothesis — it couldn’t tell me what had caused my original observation. Disheartened, I told a fellow researcher that I’d wasted a year with nothing to show for it. They told me I was wrong, I had stacks of research, it just told a different story to the one I’d set out to prove.
Original comic by Mr. Lovenstein
I see the same scenario repeated in my current work. I’ll find a bug, identify and test all variables I can think of, but none of them will explain why the bug is occurring. It’s times like that you must remember, providing the developer with a bug report which has ruled out all these variables is leagues better than just identifying the bug and passing it on.
6. Scientists are trained to report in a concise, yet detailed manner
Scientific writing is tough but once you grasp it, it becomes second nature. When reporting findings, it is crucial to present only the information which directly relates to your research. Repetition, unrelated details, and wordy phrasing are beaten out of you as supervisors return red-marked drafts over and over.
Using these learned skills means when I raise a bug report, I aim to create a title that conveys the issue in as few words as possible, outline how to reproduce the bug in terms developers will understand, and explain the different variables examined and the results of those tests.
Tester and developer relationship in a nutshell. Meme based upon @shenanigansenpic.twitter.com/B8Te67kOIA
— The Best Linux Blog In the Unixverse (@nixcraft) December 4, 2019
https://platform.twitter.com/widgets.js
It’s great to practise thorough testing in an effort to determine the root cause of a bug, but there is no point if that information is not conveyed to the developer. Say you’ve now determined that the booking bug occurs only for logged-in users with a booking history, on iOS only, and the booking is not processed as it does not appear as an upcoming booking. Make sure this information is clearly laid out in your report. It is also critical to convey to the developer that you tried the alternatives and they did not cause the issue. Simply stating, “System information: iOS”, does not convey that the bug occurs across different OS versions, but not on Android.
Don’t make developers sift through unnecessary details
On the flip side, don’t write up a bug report with so much information that it’s difficult to understand the actual issue. Testers can end up running hundreds of scenarios to determine the cause of a bug and it is a skill to determine which results are important to include in a report, and what is unnecessary.
Using precise language is also important. When writing titles, avoid using full sentences and stick to key words. The titles below contain the same amount of information, but the latter is more concise and easier to read:
“Logged in users on iOS with previous booking history causes the app to crash when making a booking”
“[iOS] Booking — Logged in user with booking history — submission causes crash”
Think about how you’re presenting the data
Screenshots are an incredibly useful tool and at times can mean the developer doesn’t need to reproduce the bug to visualise the problem or be sure it has been reproduced correctly.
Sometimes it’s useful to provide comparisons of different builds. Rather than simply adding the screenshots as an attachment to your report, think about how you’d want the reader to view these images. It’s far more meaningful, and easier to view, if these images are presented side-by-side, rather than as two separate attachments.
Utilise text formatting, tables, and lists to present your data so that developers can quickly comprehend the bug.
In summary
Testing goes far beyond writing and executing scripts. As testers, we know our products inside and out and therefore are the best people to be diagnosing the cause of the defects we identify. I enjoy breaking things. There is always a thrill in finding a bug, but let’s be accountable for finding the cause. That’s what working in a team is all about, right?
