What is meant by Structural (Code) Coverage?

par

Put simply structural, or code, coverage is the amount of code that is covered in execution by a single test or collection of tests.

For a procedural language like C, you can identify a function of interest, run some test cases on this function, and then measure what proportion (expressed as a percentage) of the code has been executed. The general rule is that the higher the coverage achieved, then the higher the confidence that it has been thoroughly tested.

Measurement of structural coverage of code is an objective means of assessing the thoroughness of testing. There are various industry-standard metrics available for measuring structural coverage, these can be gathered easily with support from software tools. Such metrics do not constitute testing techniques, but a measure of the effectiveness of testing techniques.

A coverage metric is expressed in terms of a ratio of the code construct items executed or evaluated at least once, to the total number of code construct items. This is usually expressed as a percentage.

There is significant overlap between the benefits of many of the structural code coverage metrics.

Structural code coverage is a measure of the completeness of software testing showing which areas of the source code are exercised in the application during the test. This provides a convenient way to ensure that software is not released with untested code.

The list below identifies reasons why some code has been found to be untested using structural code coverage and the resulting actions which may be taken.

Reason for unexecuted code:

Code is ‘Dead’

(i.e. dynamically unreachable)

Resulting Action:

Code can be removed to reduce:

> possibility of it becoming inadvertently executable after future code changes

> future maintainability costs with clearer code

Code can be left but commented out to make it non-executable.

Reason for unexecuted code:

Code is ‘de-activated’ or ‘infeasible’

(i.e. not supposed to be executed in a particular context, e.g. certain states, threads or system configurations)

Resulting Action:

An explanation of why the code is de-activated or infeasible to execute in a particular context can be documented (either internally using comments or externally)

Reason for unexecuted code:

Code is ‘Untested’ and is unnecessary

(e.g. code from previous versions/variants of the SUT has been carried into the code base unnecessarily)

Resulting Action:

Code can be removed to reduce:

> possibility of it being used in un-tested scenarios.

> future maintainability costs with clearer code

Code can be left but commented out to make it un-executable.

Reason for unexecuted code:

Code is ‘Untested’ but is necessary

(e.g. code is indirectly related to or ‘derived’ from a requirement such as code is added for defensive programming, may not have been explicit enough for requirements-driven test cases to be created)

Resulting Action:

Additional test cases can be added to exercise the ‘untested’ code.

Requirements can also be refined to make them more explicit for ‘derived’ requirements, depending on the need for and granularity of requirements traceability.

Code Coverage Metric Definitions

Different code coverage metrics measure the execution of different syntax constructs within the code. The most common code coverage metrics are:

1. Function / Method Entry Points

2. Function / Method Calls (and their Returns)

3. Lines (of executable code)

4. Statements

5. Basic Blocks (of sequential Statements)

6. Decisions

7. Conditions (Boolean operands)

8. Relational Operators

9. Loops

10. MC/DC (Modified Condition / Decision Coverage), both Masking & Unique Cause forms

The fundamental strategic question of how much testing you should do is generally driven by available resources, both time and budget. If you are not required to measure tests against a specific set of structural code coverage metrics by a software safety standard, then the choice of which metrics and which thresholds to set as acceptable can be determined by your own software quality policy. For more information on the advantages and disadvantages of different code coverage metrics see the QA Systems white paper “Which Code Coverage Metrics to Use”.

For all the main software safety standards the required code coverage metrics (depending on integrity level) are Entry-point, Function Call, Statement, Decision and MC/DC coverage. These are explained in more detail below.

Entry-Point Coverage

Entry-Point coverage measures the proportion of functions in the source code that have been executed at least once. It is the easiest metric to achieve 100% coverage in tests.

Call-Return Coverage

Call-Return coverage measures the proportion of function or method calls in the source code made and completed at least once. It is the most commonly used metric to measure integration level testing.

Decision Coverage

Decision Coverage measures the proportion of decision outcomes in the source code which have been evaluated at least once. It can sometimes be referred to by these alternate names: C2, Branch Coverage, TER2, TER-B coverage. Decisions include constructs such as ‘if… else…’, ‘switch… case…’ and loops such as ‘while’ and ‘for’. Decision coverage contains Statement coverage but ignores the complexities of conditions within decisions.

MC/DC — Modified Condition / Decision Coverage

Modified Condition Decision Coverage measures the proportion of operand conditions which could independently affect the true/false outcome of the decision expression that have been effective in doing so at least once. It can sometimes be referred to as a combination of Decision coverage and Boolean Operand Effectiveness coverage. MC/DC coverage demonstrates that every sub-condition can affect the outcome of the decision, independent of the other sub-condition values.

There are two methods for measuring MC/DC coverage: Unique Cause and Masking. The latter was created by Boeing to accommodate the short-circuiting evaluation of true/false expressions in C/C++.

MC/DC is the hardest metric to achieve 100% coverage in tests requiring the most test cases.

Do you want to to know more?

If you found this introductory article useful, QA-Systems have a detailed paper available for download for free here. This expands on the principles touched upon in this short article.

Download the full 26 page whitepaper as a PDF.

You are also welcome to sign up to the QA-Systems newsletter… You will receive notifications of other useful software development content straight to your inbox.

SAFETY CRITICAL AND EMBEDDED SOFTWARE DEVELOPMENT

QA Systems experts provide industry insights. Discussing software quality for embedded systems, safety critical software development, testing techniques and automation of development processes.

RELATED RESOURCES