Debugging requires understanding of program flow as well as techniques to isolate the issues (bugs).
Firmware flow consists of:
The first few instructions are typically written in assembler or possibly a series of hex values. These instructions provide enouth of an environment so that the next portion of code may be written in another language, typically 'C'. This code creates the kernel stack. The last assembly instruction is a jump to the 'C' code.
The higher language code finishes the CPU initialization.
Next the memory controller is initialized so that the external memory may be accessed. In some systems, the memory itself needs initialization to wake it up and get it behaving as memory.
Now that memory is available, the kernel library is initialized. The kernel library provides the services for the kernel including low level I/O.
The boot loader is invoked to locate the application code and create the environment to start the application. The boot loader transfers control to the application.
The first instructions in the application initialize the runtime library. Each high level language has a runtime library to support high level operations needed by the compiler and the application. The runtime library must be initialized before any of the application code is run to ensure that library calls by the compiler or application are properly handled.
Next class constructors of any globals or static locals are called to ensure that they are properly initialized before they are referenced by the application code.
Note: This is application code and may fail before setup is called!
In the Arduino environment the setup subroutine is called. This is typically considered the starting point of the sketch (application or program). This code typically initializes the serial port and performs the one time initialization for the sketch.
The loop subroutine is repeatedly called after the setup subroutine exits. Its function is to handle any runtime operations that are not handled by interrupts or events.
There are several debugging techniques:
For your own sanity, remember that the most likely place for the bug is in your own code! Probability of the bug decreases significantly the further you go down this list:
One of the fastest ways of debugging the code is to be very familiar with the code's operation. Know what subroutines are performing what operations and how they work. When an issue's symptoms arise mapping these symptoms onto the code's operation typically will narrow down the failure to a specific routine or line of code.
When you don't know the code, one of the best techniques is the binary search. Put a debugging statement or breakpoint halfway through the execution path of the code and see if you reach it before the bug occurs. Continue repeating the process by splitting the piece containing the bug in half again until you get down to a very small portion of code, typically one line of code.
Serial output provides a way to see variable values and describe the expected behavior. For C++ code, placing Serial.printf statements in routines provides a way to both determine if the code is getting executed as well as what values are being used during the execution. This will typically help in isolating and solving the issue.
The Arduino environment provides some log macros that display serial output. There are several levels of debug output that may be displayed. In the tool bar of the Arduino IDE or on the command line of the Arduino CLI you may select the log level for the sketch at build time. These messages are typically disabled when the application is delivered.
Alternatively, you may printf instead of the log macros and use a variable that may be changed during runtime to display debug messages. This allows you to perform debugging when an issues is detected without requiring another build and then forcing you to reproduce the issue.
Random failures are typically associated with an uninitialized variable! Please note that all globals and static variables are initialized to zero (0) if an initialization value was not specified. The uninitialized variable must be on the stack, in the heap, or referenced by a pointer to some other portion of the system. The randomness occurs because stack values change depending on what has executed before the code with the issue executes. The variables overlay a portion of the stack that contains random garbage from a previous routine's execution.
Similarily a variable that is uninitialized in the heap sees the previous contents left by block of memory freed by another user of the heap. Tracking down these types of errors is difficult because they are not very repeatable because the execution path changes due to elapsed time and external events.
Know how your variables are getting initialized!