Debugging

Don't skip this step

Programmers are sometimes tempted to skip the replication step, and to go straight from the bug report to the fix. However, failing to replicate the bug means that it is impossible to verify the fix. The fix may turn out to be for a different bug, or it may have no significant effect at all. If the bug has not been replicated, there is no way to tell.

Failure to replicate the bug is a real problem which I've seen happen many times. In a complex program, it's often quite easy to find something to fix--perhaps a real problem, perhaps merely something which looks like a problem. It's human nature to assume that any particular fix solves the problem at hand. Without a means of verification, any plausible fix will be accepted, whether it is right or wrong. An incorrect fix leads to future problems. On average, skipping the replication step wastes more time than it saves.

Difficult replication situations

By far the best way to replicate the bug is on a system wholly under your own control, with a copy of the program you built yourself. Replicating the bug on your own system means that you can easily test a patched version of the program. Unfortunately, in some cases this is impossible.

One reason that replication on your system may be impossible is that the user reporting the bug may have a unique system configuration, or the user may be using private input files which you can not obtain. In these cases you must try to ensure that the user can reliably replicate the bug, and that the user can test the patched program. Without this help from the user, debugging is reduced to little more than guessing. It's generally not worth proceeding. If that is not an option, then you should at least try to construct some sort of test case for any possible patch, to confirm that the patch does do something useful even if you can't verify that it fixes the original bug.

A more common problem is that the user can demonstrate the bug, but you can not replicate it on your own system, and you don't know why. There is no definitive way to solve this problem, but here are some of the possibilities to consider.

• The user may simply be misreporting the problem, or the way in which it is replicated. Go over it again. Confirm what the user actually types and where the user actually clicks. Confirm what the user sees on the screen. Try not to make any assumptions about how the program is being run--the user may be doing something which is so off the wall that you would never consider it.

• Some programs have odd behaviour under unusual conditions. Check whether the user's system is very low on disk space, or has bad network connectivity, or is very overloaded. Check whether any other programs are failing unexpectedly--hardware problems are rare, but they do happen. Check for messages in the system logs.

• Check the version of the software the user is running. Check the versions of the operating system and any system libraries. Check which system patches have been applied. Check the exact system type, monitor type, keyboard type, etc.

In some cases a bug may happen rarely and apparently randomly, so that it is very difficult to figure out what triggers it, and how to replicate it. In this frustrating situation, I only know of one reliable approach, which is to automatically log all potentially relevant events, and save the logs when the bug occurs. In some cases it's particularly useful to be able to replay the program using the log, which can be a powerful mechanism for replicating a bug. Of course this requires a logging facility, and adding one to a large program can be a large undertaking. Nevertheless, it's probably worth doing the second time you find yourself searching for a bug which appears to occur randomly.

Debugger is a misnomer

As a practicing programmer, you are probably familiar with the tool called a debugger. The name of this tool makes it seem like the tool of choice for debugging--it seems logical that in order to debug you would use a debugger. However, this is inaccurate. As I said initially, debugging means removing bugs from programs; while a debugger is a powerful tool, one thing it does not do is remove bugs. A better name for the tool would be "inspector". (As it is unfortunately too late to rename the tool, I will continue referring to it as a debugger.)

In order to fix a bug, you must understand it. A debugger can sometimes help you understand a bug, but it is only one of several tools for that purpose. Only in certain specific circumstances should it be the first tool you reach for.

Understand the program

In order to understand a bug in a program, you must have some understanding of the program.

If you wrote the program, then you presumably understand it. If not, then you have more serious problems. [1]

If you didn't write the program, you need to grasp its general structure. Most programs are organized in a fairly sensible fashion, once you know the general approach. If you are lucky, the general approach is documented, or you can ask the original designer.

More commonly, you need to pull the structure out of the source code. The best approach is to start looking at the source code from the start of the program (e.g., the main function in a C program). Skim through the program, stepping down through functions, until you find the main center of action--in most programs, some sort of loop. This can normally be done fairly quickly. The nature of this center of action should tell you where to look in the source code for any particular activity. It should also tell you the general way in which the program acts.

The worst cases are large programs written over many years by many different people. These often become a hodge-podge of different ideas with little consistency. The situation is depressingly common. You must simply do the best you can. At least try to avoid making the mess worse.

When examining a large program, some people find it helpful to use a source code analysis tool, such as Source Navigator. Personally, I normally rely simply on Emacs tags tables.

A debugger can also be helpful when trying to understand a program. By running the program under the debugger and setting breakpoints, you may be able to see the dynamic behaviour of the program. When you reach a breakpoint, look at the call stack to see how you got there, and look at key variables. Or if you don't reach a breakpoint you expected to reach, you've also learned something.

Locate the bug

The next step is to locate the bug in the program source code.

There are two source code locations which you need to consider: the code which causes the visible incorrect behaviour, and the code which is actually incorrect. It's fairly common for these to be the same pieces of code. However, it's also fairly common for these to be in different parts of the program. A typical example of this is when an error in one part of the program causes memory corruption which leads to visible bad behaviour in a completely different part of the program. Do not let your eagerness to fix the bug mislead you into thinking that the code which directly causes the bad behaviour is actually incorrect.

Ordinarily you must first find the code which causes the incorrect behaviour. Knowing the incorrect behaviour, and knowing how the source code is arranged, will often lead you quickly to the part of the program which is at fault. Sometimes a quick scan of the source code is enough to identify the problematic code.

Otherwise, narrowing down the bad behaviour to a particular piece of code is where a debugger can be very useful. If you are lucky enough to have a core dump, a debugger can immediately identify the line which fails. Otherwise, judiciously setting breakpoints while replicating the bug can quickly hone in on the code you are after.

Modern debuggers, such as gdb, have powerful capabilities to make this process more manageable, such as conditional breakpoints, data watchpoints, ignoring breakpoints certain numbers of time, and support for executing simple code at breakpoints. These features are very useful when locating bad behaviour in code which is executed many times, and only fails under particular circumstances. It's a good idea to learn the features which your debugger provides.

Of course, sometimes a debugger does not help. In some cases, a bug will not occur when the program is run under a debugger, even though it can be reliably replicated without the debugger; this typically indicates a problem which depends upon precise timing or memory layout. In other cases, you may not have access to a debugger, or the debugger may not be very powerful; this can happen when working on embedded systems or other impoverished programming environments, or when using programming features which are not supported by the debugger (for example, in some environments, threads).

In such cases simple print statements can sometimes help locate the source of the bad behaviour. Add print statements to relevant locations, rebuild the program, replicate the problem with the new program, and use the print statements to hone in on exactly what code is being executed when the problem occurs. Ideally the program already has some sort of logging facility which you can reuse. Even if it doesn't, you should consider a systematic approach to the print statements you add, so that you can build on that when doing future debugging of the same program. In particular, every print statement should clearly indicate where it is located in the program, so that it can be quickly found again later.

Another useful approach is to add check routines to the code to verify that data structures are in a valid state. Such routines can help narrow down where data corruption occurs. If the check routines are fast, you may want to always enable them. Otherwise, leave them in the code, and provide some sort of mechanism to turn them on when you need them.

In the specific case of a memory corruption bug, you may be able to replace the standard memory allocation routines with ones that perform various checks. For example, on GNU/Linux systems, read the malloc documentation to see how the environment variable MALLOC_CHECK_ can be used to do this.

The final fallback for locating the source of the bad behaviour is simple source code inspection. This is the only option if you can't replicate the problem. A clear understanding of the overall program source code is an absolute requirement for this to work. Unfortunately, a complex problem is nearly impossible to isolate by simply reading the source code. You will have to guess at likely possibilities, and try to trace through the code carefully to see if they are really problems.

If you are very unlucky, the bug may not be in the program source code at all. It may be in a library routine, or in the operating system, or in the compiler. These cases are rare, and it is a mark of an inexperienced programmer to suspect a compiler bug too quickly. However, they do happen--they've all happened to me--so when all else fails, consider these possibilities. Verify a bug in a library routine or the OS by writing a check program, and verify a bug in the compiler by examining the machine code directly.

Locate the error

Now that you have found the code which causes the bad behaviour, you need to identify the actual coding error. Often they are the same code--that is, the coding error directly causes the bad behaviour. However, you should always consider the possibility that the actual error is elsewhere.

For example, the routine which causes the bad behaviour may be behaving correctly, but be called with bad input, or at the wrong time. A coding error elsewhere may cause a data structure to hold unreasonable values. Another possibility is bad user input.

The fix in such cases may be two-fold. You should, of course, fix the code which called the routine incorrectly or otherwise created the bad input data. In the case of bad user input, you should validate the input. In addition, however, you may want to add checks to the code which used the values. It should check for unreasonable input, and report an error or otherwise handle the error without causing invalid behaviour.