There are many guides on how to write good bug-reports (or even, illogically, on how to write good bugs), including http://www.chiark.greenend.org.uk/~sgtatham/bugs.htmle and http://www.softwaretestinghelp.com/how-to-write-good-bug-report/e. Something that is regrettably lacking is a good guide on how to read and handle bug reports—at which many developers and most product managers, project managers, first-level supporters, and similar are very poor. This page is an attempt to provide a partial remedy to this problem based on my extensive experiences with bug tracking over the years, where I have both used it and seen it used from a number of perspectives, including the developer’s, the user’s, the tester’s, and the release manager’s.
With some points below, the objection “But we don’t have the time!” might apply. While this can indeed be an insurmountable obstacle, do see an excursion on this.
The exact terminology to use is somewhat disputed, with other alternatives being e.g. “error” and “issue”. The most commonly used term in this context, however, is “bug”, and I will stick to that for now. I note that the runner-up, “issue”, has a wider meaning and includes e.g. requests for assistance that are not caused by software errors. Parts of the below will apply to this wider term; other parts will not.
Be grateful for all input. Do not assume that you are doing the user a favour by fixing “his” bug—he might very well take the view that he is doing you a favour by reporting “your” bug, having spent fifteen minutes writing a bug report on something that he might never encounter again (could easily work around, could live with, whatnot). If in doubt, he is the one who discovered the bug, not the one who caused it. Things will go smother, and your product will benefit, if you adapt the same attitude—he is helping you, not the other way around.
Making unreasonable demands of the user, using rude language, or otherwise being unappreciative, will lead to an alienated user and a worse product.
Note that a paying costumer might rightfully and righteously take the view that your employer is wasting his time and money, and might be in a less than good mode to begin with. Playing the “be grateful” card (even by implication) will not go over well.
Also note that many companies make odd impositions on a user who tries to register a bug: I have, e.g., been made to register online with a dozen mandatory and irrelevant fields (that I duly filled with dummy information), had to wait half an hour for a confirmation email with my login data, and only then been able to actually enter my bug report. If a user is somewhat lacking in gratitude at that point, this is entirely your employer’s fault. (To make matters worse, even some FOSS projects do similar things today.)
Here another danger is obvious, namely that users will be reluctant to jump through all these hoops in order to report bugs in the first place. Revisiting this page in 2023, I cannot recall the last time that I bothered with a bug report. (Excepting programs that I have been paid to develop or test.)
Why cut off potentially valuable feedback by introducing arbitrary hurdles?
Similarly, cf. the following side-note, a user who does enter a bug report and then suffers e.g. a “we don’t care—closed; won’t fix” is unlikely to ever bother again—and he might well give up using the product in frustration.
A related problem is decision makers (note e.g. the infamous attitude of Mozilla) who take an attitude of “we know better—closed; won’t fix”, “we don’t care—closed; won’t fix”, “we decide what the user should want—closed; won’t fix”, “too few users are affected by that—closed; won’t fix”, or similar.
(This is partially off-topic, especially as it might be more likely to relate to feature-wishes or complaints about new and poorly thought-through features than to regular bugs, but it is still something to guard against. If in doubt, even behavior which conforms to the specification can be harmful to usability and long-term product success—as with Firefox)
Note that it is the attitude that is problematic. Coming to a similar conclusion with another attitude is not necessarily wrong, especially when resources are limited. For instance, moving a bug report to a list of feature requests with the motivation that it actually is a feature request is far better than a “closed; won’t fix”—even should the feature request ultimately never make progress. (Provided that it never makes progress because other features give better bang for the buck—not because the move was a “closed; won’t fix” in disguise.)
Also note that there might be some overlap between this attitude and e.g. the “negotiations” mentioned below, in that someone might dishonestly go down the “closed; won’t fix” road in order to keep the number of open bug reports down, regardless of the validity of the report. In particular, while a status like “closed” is an attribute of the bug report, not the bug, closing a bug report for a bug that actually exists (and which has not been fixed) can give a very misleading impression and great care should be taken.
Be kind to newcomers. Consider that the bug reporter might know considerably less about computers than you do, have less experience with bug-tracking systems (in particular, your local one), and will almost certainly know less about the application that the bug occurred in than you do.
Handle e.g. non-native speakers similarly. You are not perfect either.
Conversely, the user can be a highly competent developer with twice your experience and an IQ two standard deviations above yours, so make very sure not to be a condescending medium fish.
When requesting information, tell the user why. If you do, he is much more likely to give it to you. Example: The first link above states
Somebody reported a bug to me once, and I asked him to try a command that I knew wouldn’t work. The reason I asked him to try it was that I wanted to know which of two different error messages it would give. Knowing which error message came back would give a vital clue. But he didn’t actually try it - he just mailed me back and said "No, that won’t work". It took me some time to persuade him to try it for real.
The author sees this as a user error. I see it as a developer error: Had he stated upfront “This command will give one of several error messages. Knowing which one will tell me [...].” then he would almost certainly have received the cooperation that he wanted.
This principle applies much more generally: As good as always, it is appropriate to tell a counterpart why some particular piece of information is needed. (Off the top of my head, the only exceptions are when the reasons are obvious, respectively the requestee is just an aidee of some sort to the requester.) Doing so increases the probability of cooperation, is more polite, and can help the counterpart to help the requester, should his request be incomplete, irrelevant, whatnot. This applies even when he, theoretically, has the right to unconditionally order the information to be given—which you do not.
Do not demand that users perform extensive extra tests of e.g. other versions or platforms, that they try to re-install something before reporting a bug, or similar. A user who has given a correct report on how the bug occurred on his system, as it was at the time, has done his job—the rest is your job.
(The above is not to be confused with clarifications that are necessary in order to understand the bug report, when/where/how the bug manifested, etc.)
Asking for his further help, without making demands and without making your work contingent on this help, is perfectly in order, but if such help follows, especially help that requires considerable work, then the user is doing you a favor.
Note, in particular, that many bugs can be found in thirty seconds by actually looking at the code; sometimes a developer can even know what is wrong upon hearing the problem description. Forcing a user to spend twenty minutes (sometimes even considerably more) doing research for such a bug is extremely user unfriendly.
Similarly, do not demand that the user provides a patch himself, even if you do work on a FOSS software. He already has helped by reporting the bug, and it is unreasonable of you to demand further help from him in a rude manner.
Again, note the importance of attitude: A “Sorry, we appreciate your report, but we do not have the resources to handle that right now. If you write a patch yourself, however, we will be happy to integrate it.” is perfectly in order; a “We are not here to do your work for you. If you want it fixed, fix it yourself.”, OTOH, is not in anyway acceptable (and, yes, I have seen such formulations on repeated occasions—fortunately, always directed at others, else there would have been a word or two said in response).
Gathering detailed information on platforms and similar might take considerable effort on behalf of the user—and is not always necessary. Often, an experienced developer, who happens to be a reporting user, can rule out the value of some additional information in advance. It is better to give basic information about the system, with the indication that more information will be provided upon request—in particular, as the user cannot reasonably foresee all information that can be relevant.
If some information is mandatory, this should be stated in advance and in a sufficiently clear and obvious manner that the user can be expected to know it. (For instance, if such claims are hidden in a ten-page “how to file a bug report” document specific to a certain product, this would typically remove any blame from the user.)
Care should be taken not to include too much in the mandatory category.
If you always want particular informations, regardless of the
circumstances, write code to check for it yourself. The user can then be
given instructions along the line “Please always append the output of
[application name] –environment.” (or similar).
Be grateful for all input. (Yes, I said that already.) All products have bugs, and it is in your best interest to know of as many as possible (in your product). You might be happier not knowing, not being bothered by complaints, or with your head in the sand; but you will be better off knowing. Foster a mentality of “Every bug report is a gift.”, and your software will be better off for it.
Do not reclassify bugs for better appearances: I have repeatedly seen major revisions of e.g. bug severities when a deadline approaches. Typically, the organisation has had some rules about the respective number of critical, severe, major, whatnot bugs that are allowed before a release can be made—for a good reason. When these numbers are not reached in time, then the release should be delayed, in order to ensure a sufficient quality of product, to avoid excessive bug fixing post-release, to not ruin the opinions of customers and users, etc. However, in reality, a game of “How can we re-classify this bug?” starts, where the bugs are deliberately moved into the wrong categories to enable the release to go ahead as planned.
This should not be confused with a limited correct re-classification that can validly occur at any time. Warning signs for the wrong kind include negotiations about severities with stakeholders (including giving promises in return for being allowed to re-classify), the opinions of stakeholders or developers being over-ruled without logical and factual arguments, decisions made by managers and project managers who cannot truly judge the situation and stand to benefit from the short-term keeping of the deadline, and obviously the sheer quantity of bugs involved and the closeness of the deadline.
But should there not be some amount of leeway and discretion here? Getting the product out on time is important too! (A similar protest, with an analogous answer, is that a developer might over-focus on the technical side and that someone else might have a legitimate point concerning e.g. business considerations.)
True. That discretion, however, should be built into the rules, so that cheating the system is not needed, and the bugs can retain their correct states, classifications, and whatnots. If the rules are too inflexible, then lobby to have the rules changed in an orderly manner. Also beware that the rules, as they are, already might have considerable flexibility, and that they are now being broken after this flexibility has already been used up. Consider e.g. a rule made according to a reasoning of “Ideally, we should have no severe bugs. This, however, might be too restrictive, so we give [someone] the discretionary power to let through one (known) severe bug per 50.000 lines of code, with an upper limit of three severe bugs for the entire application.”. Now assume a situation where we have four severe bugs: As likely as not, someone will complain that, “It is just one bug more, the rules should have some leeway!”—contrary to the purpose of the rule and ignoring the original leeway.
(In addition, my personal opinion is that deadlines are changeable, scope of product and features is changeable, but quality concerns are not. The opposite is short-sighted and, often, unethical. That many others see it differently is demonstrated by the fact that there are rules that allow releases with any severe bugs, at all.)
Do not throw out old bug reports in a blanket manner when moving on to a new version—even if a substantial re-write! Even with a re-write (and the more so for a lesser version change), these bugs could still be present for several reasons, including that the offending code might still be there (alleged re-writes often contain considerable copying...). Even if the code is changed, similar algorithms, patterns, and other solutions might be present and have re-introduced the bug. The bug might be a result of another bug in a third-party library or of a misunderstanding of the interface to this library. Etc.
Correspondingly, each and every (open) bug report present for the old version that could conceivably, even in a thousand-to-one shot, still be present, should be explicitly tested. Ideally, serious closed bugs should be checked too (unfortunately, this will almost always be unrealistic).
Do not scorn minor or “trivial” bugs: Not only should every bug be at least considered for fixing (even if it eventually is not), but a great occurrence of many minor bugs can be a warning sign of an underlying problem. It is, in particular, possible that even a single apparently minor bug is just one minor manifestation of a major one—even a show-stopper.
Do not refuse treatment with the view that the bug occurred in an older version, without having explicitly checked whether it applies in newer versions.
Note that restrictions to e.g. platforms and versions need not indicate the true scope. If the user reports that V1.32 on Windows NT 4.0 shows a certain behavior, then this implies that the user encountered the behavior using V1.32 on Windows NT 4.0—not that he has extensively verified that the behavior only occurs with specifically V1.32 on specifically Windows NT 4.0.
Assuming the opposite is absolute amateur hour. Adding to this error by ruling that the platform or version mentioned is so rare that the bug can be neglected is inexcusable. Until otherwise proven, assume that the bug is present everywhere.
Beware of the impact of different platforms, libraries, etc., and do not throw a bug report out with the claim that you cannot reproduce it. Unless the bug is trivial or the user gives signs of not having done his homework, take the bug sufficiently seriously to at least think possible causes through, have a look at the source code, etc.
If this turns up nothing, then ask the user for more details to find out what you are doing differently. If, and only if, he cannot reproduce it anymore, then mark the report as invalid.
Beware, in particular, of misconfigurations on your side: At E4, I originally had a bug report for a webpage rejected, because the internal QA team could not reproduce the JavaScript error that I reported. As it later turned out, the nit-wits actually had the display of JavaScript errors turned off in their Internet Explorers... (I note that this team tested more or less exclusively through access over the web and using IE. Not activating these error messages was about as professional as a near-sighted health inspector going about his work without glasses/contacts.)
Similarly, beware of sporadic bugs and heisenbugs. That a bug cannot be reliably reproduced, does not automatically mean that it is not there. Notably, if a bug occurs only occasionally, e.g., due to a threading issue, then chances are good that other bugs will be caused by a similar mechanism—one, possibly, that once in a life-time freak incident that costs your employer two million dollars.
However, note the difference between rejecting a bug report outright (bad) and putting it on a wait-and-see list (often acceptable): Wait some reasonable amount of time and see whether the bug is observed by someone else. If not, then the report may be rejected. (A further observation would, obviously, serve as verification and move the report from the wait-and-see list to a more active status. What constitutes “reasonable amount of time”, I leave unstated, as it depends on wildly varying factors like size of the user base.)
Not every bug must be fixed (although this is certainly the ideal), and sometimes circumstances (lack of time, money, developers, control over third party products, backwards compatibility, ...) can make a bug fix impossible even in the long term; however, no bug must ever be swept under the carpet.
A few general, semi-related notes that will likely be moved elsewhere later (possibly to a page dealing with debugging):
If a bug (pertaining to logic) is verified, always write an automatic test for it. This allows to automatically verify that the solution is correct, that the bug does not recur later and is not present in other versions/on other platforms, and may help in detecting similar bugs. In many cases, it can make sense to write a test case immediately, even based on just a report. Notably, more test cases is almost always better.
When many bugs come from one single method/class/module, consider a complete re-write: Where there is many, there will be more.
Read up on debugging techniques and experiences elsewhere, e.g. http://c2.com/cgi/wiki?HeroicDebugginge.
Some above items might not always be realistic due to a lack of time and/or resources. If so, a compromise might be needed; however, it is the more important to ask why and to take corresponding actions for the future.
For instance, if a development team is unable to keep up with incoming bug reports, then questions apply like:
Is the team understaffed or lacking in competence?
Are the quality standards for the product too low?
Are the quality measures (e.g. regarding testing) insufficient?
Has too much past time been spent on evermore features and too little on maintenance?
(On a more detailed level, questions like “Is there some individual component that is particularly error prone?” can be valuable.)
Chances are that one or more of these and similar questions will be answered with “yes”. Note, especially, that such issues can have a negative effect on each other and cause an ever-worsening spiral of problems and a lack of time and resources that would not have existed, had quality been taken sufficiently seriously sufficiently early. Of course, to reason that “Due to lack of time, we have to cut back on quality!” in such a situation is very dangerous, as it will only make the problems worse the next time around.
We then have the meta-question: Are we forced to sacrifice X because we lack Y, or do we lack Y because we have sacrificed X in the past?
(Similar remarks can apply elsewhere. For instance, above I discussed “negotiations” about bug classifications in order to allow a release. Chances are that the bugs in question have been around for some time. Why were they not actually resolved when there was time enough to do so? If the bugs were new, OTOH, it is even worse: discovering even one severe bug a few days before release is very, very bad, as it points to poor quality control and an increased risk that other severe bugs are discovered after release.)
Some of the more time-consuming items above point specifically to the benefits of automated testing. For instance, if a bug is fixed today and verification takes place solely with manual testing, re-testing the problem can be both time consuming and error prone—and if the bug was not fixed in the first place, verifying that it is gone in the wake of greater change (e.g. a rewrite, cf. above) is hard. Write an automated test, and do so before fixing the bug, and much less effort will be needed down the line.