The importance of completely testing software applications, and achieving it cost effectively is currently one of the major growth areas in the software development tools industry. There are many tools that offer solutions for automating the unit and unit integration test process for our application code. These tools can parse our code, and from a few simple options automatically create test drivers to stimulate our code under test, and test stubs for our dependencies. While these tools vary in capability and automation, they all offer an improvement on current manual testing processes that are in place.
One of the areas that is relatively poorly understood, and perhaps glossed over in the selection of these tools, is the process they go through to parse the code under test and automatically build the test framework. Every unit test automation tool in the market today needs to parse the source code under test to determine types, and interfaces for functions to be tested, and implement stubs for its dependencies. As a result all of these parsers understand standard C or C++ code, however every compiler not only supports the standard C or C++ language, they have extensions.
These extensions or keywords are generally processor specific directives to more precisely control the binary output from the compiler. For example "_interrupt_handler" which would be placed in front of a function that will be attached to an interrupt vector. The keyword tells the compiler to disable stack checking. Perhaps if the code is written for some kind of high performance microprocessor like a DSP, then it may also have custom sized types, like a 16 bit float type, etc. The challenge for the parsers in a unit test automation tool is that they understand the standard C or C++ language, but when presented with these compiler specific keywords, they don't understand it. So for these parsers to process these keywords, the vendor or end user introduces MACROs, i.e. '#define _interrupt_handler ", or "#define float16 float". The act of doing this modifies the code under test, and produces a completely different binary to what is running in the real system. This could potentially mean there are unchecked boundary conditions in the code (due to different sized types used), or even worse the meaning of the code has been changed due to the dropping of the keyword.
Here are some examples of the impact of using MACROs
Case 1 - compiler keyword __packed - typically used to stop the compiler building data structures on word/long word alignments for more efficient data transmission / storage. The effect of the __packed keyword on the size of the struct is that the size of the struct is reduced to 5 or, if __packed were to be macroed out, then it increases to 8 (due to long word alignment).
Case 2: compiler keyword __irq using the IAR ARM microprocessor compiler.
You can see that both functions use R1. in the case of an irq function, R1 is saved and restored. In the case of the non-irq function, R1 is used without being restored. The compiler knows not to rely on R1 for a code section that makes a call out to a dependent function.
As a result, when using test automation tools, it is important to work with tools that not only support the standard language set for C & C++, but also support the keyword extensions that specific compilers have. This would ensure that when the parser of the test automation tool parses the code under test, it will leave the code unmodified, and as a result produce binary code that is identical to what would run in the real system. This means as a tester you have the assurance that the code under test is actually the code that is being tested.
