* DEBUGGING

Tools:

1. interactive symbolic debuggers -- gdb, also integrated into many IDEs

2. code testers and verifiers:

- tools that can analyze raw source code for problems: coverity, even a
  modern gcc w/ lots of -W... warnings flags will catch lots of possible
  issues -- I like to use "gcc -Wall -Werror".  Some tools would instrument
  the source code w/ extra checks, then you compile and run a "debugging"
  version.

- tools that run the binary code directly w/ some checking, possibly
  instrumenting the machine code.

- many more tools both open source (e.g., valgrind) and commercial.

Note: there's still few tools to debug operating systems, embedded systems,
and distributed systems.  Also very hard to debug any timing based bugs,
like races and deadlocks.

3. regression suites: long list of small programs and scripts that run on a
compiled binary, and lists which programs passed/failed the test.

4. tracers: system call tracers like strace(1), library tracers like
ltrace, and more.  For networking, often useful to use packet sniffers like
tcpdump, ethereal, etc.

5. built in debugging and/or tracing.  Example, add lots of optional printf
messages, that can be enabled/disabled on demand.  Helpful to debug b/c such
tracing is custom to the program itself, also easier to convince customers
to enable it and send you logs.

Enable debug: run a different program, pass some command like option like
-D, send a signal to the program (kill -USER1 <pid>), or send a msg to the
program by another mean (e.g., unix sockets).

Where to log:

- easy -- print on terminal e.g., to stderr (the unix "error" channel).

- send log message to system wide logging server (e.g., syslogd).  syslogd
  can then be configured what to do w/ log messages: ignore some, forward
  some to some other central logging station, save to a file, etc.

- sometimes you can write the trace o/p to a file directly.  File name may
  be configurable.

What debugging architecture to use: problem is you can't log too little
(then hard to debug) or too much (hard to sift through many messages).

- define debug LEVELS: 0 (off), 1 (log some), 2 (log more), 3 (log even
  more), etc.  Easy to use and setup.  Problem: often we want some messages
  from different levels.

  You can pass a higher debug level to a program using command line
  switches.  You can also use getopt(3) such that each time you use a debug
  switch, it raises the debug level, for example

  $ ./myprog -d args # debug level 1
  $ ./myprog -d -d args # debug level 2
  $ ./myprog -ddd args # debug level 3

  Alternatively, you can design signal handlers so each time you send a
  SIGUSER1 to the program, debugging level is increased; and each time you
  send a SIGUSER2, debug level is decreased (or perhaps turned off).  Note:
  it's useful each time you change a debug lebel, or turn it off, to log a
  message saying so.

- define debug SUBSYSTEMS: each subsystem has its own boolean code that can
  be turned on/off to list messages just from that subsystem.  There can be
  many subsystems: net, gui, crypto, process, etc.  Each subsystem can be
  defined symbolically (e.g., a string), but internally, they're represented
  by a "bit map" where each bit is mapped to a subsystem to debug.  This
  allows you to enable debug for any subset of systems as needed: more
  versatile; but can be harder to code, and the number of subsystems/bits
  can grow quite large (e.g., IBM GPFS, Linux Tracepoints, Solaris dtrace,
  BSD ktrace).

  $ ./myprog -D net,nogui # enable debug in "net" component; turn off in "gui"

- sometimes you combine the two: you have several subsystems, each
  represented by, say, 2 bits (00 off, then you have 1, 2, or 3).

- In some cases, complex apps may define a hierarchy of debug subsystems, or
  even offer "aliases" like "allnet" to mean enable all networking
  components' debugging.

- sometimes when there's a lot of subsystems, some utility let's you turn
  on/off a number of them related to a certain activity.


* NEXT

#include "8.c"

#include "redirecting.txt"
#include "crypto.txt"