GCOV/LCOV Code Coverage Tool

General

Name: GCOV/LCOV Code Coverage Tool
Tool Provider:

GNU, LTP
Supported Standards: ISO26262
Description:

GCOV and LCOV are C/C++ code coverage tools

GCOV is a test coverage program provided by GNU (https://gcc.gnu.org/onlinedocs/gcc/Gcov.html).

Use it in concert with GCC to find out some basic performance statistics, such as:

* how often each line of code executes

* what lines of code are actually executed

LCOV and GENHTML are extensions of GCOV provided by LTP (https://github.com/linux-test-project/lcov)

The extension consists of a set of scripts which build on the textual GCOV output to generate HTML based output.
Use Cases:

Content

Identifications

g++ and lcov

Description:

g++ version 8.4.0 and lcov version 1.15

gcc and lcov

Description:

gcc version 9.3.0 and lcov version 1.15

Use Cases

Code coverage measurement using gcov and genhtml

Safety-Impact: yes
Inputs:
  • C/C++ compiled code:

    The C or C++ compiled code files considered for code coverage measurement
  • C/C++ source code - AUTOSAR-compliant (version 18-10):

    The C or C++ source code files considered for code coverage measurement

    The C or C++ source code files shall be compliant with AUTOSAR C++14 (18-10)
Outputs:
  • HTML coverage report:

    Generated coverage report in HTML format
  • gcov Coverage Report Output files:

    *.c.gcov files

    The documents having extension *.c.gcov are produced by Gcov as a result of its execution and are artifacts which are examined during the qualification.
Features:
  • Execution:

    The execution of the instrumented program is not directly a gcov/lcov feature, but it is required in order to generate the coverage data files.
  • gcov Core functionalities:

    Requirements:

    - Report what lines of C/C++ code are actually executed

    - Report what lines of C/C++ code are not executed
  • gcov Options:

    Container for the options accepted by gcov

    [Detailled information has been removed here. More options are contained in the QKit]
  • genhtml Core functionalities:

    Requirements:

    - Create an HTML report file from lcov/gcov data files (convert data files)

    - Report a summary of the coverage data and coverage ratio (collect and convert log information)

  • genhtml Options:

    Container for the options accepted by genhtml

    [Detailled information has been removed here. More options are contained in the QKit]
Potential errors:
# Error Safety-Impact Mitigations for detections/avoidance Final detection/avoidance probability
1 [Execution] The instrumented executable exhibits a different functional behavior than the uninstrumented executable

During the execution, the instrumented executable exhibits a different functional behavior than the uninstrumented executable.


Comment:

This error has a safety impact as the tests and the code coverage analysis are based on the instrumented version of an executable.

 impact_true HIGH
2 [Execution] The instrumented executable is not executed (not starting or not terminating)

The instrumented executable may not be successfully running.


Comment:

This error has no safety impact as no coverage report will be generated. Therefore, the end-user will detect it with a high probability when checking the list of coverage reports.

 impact_false
/
 HIGH
3 [gcov Core functionalities] Reported code coverage too low

The reported code coverage may be too low.


Comment:

This error has no impact since if the reported code coverage is lower than expected, the developer/tester will have to argue why some code is not covered.

 impact_false
/
 HIGH
4 [gcov Core functionalities] When reporting code coverage as percentage, gcov is rounding up to 100%

This error corresponds to the violation of the following requirement:

* When printing percentages, 0% and 100% are only printed when the values are exactly 0% and 100% respectively. Other values which would conventionally be rounded to 0% or 100% are instead printed as the nearest non-boundary value.

 impact_true HIGH
5 [gcov Options] Wrong (too high) reporting of branch executions using Gcov

A branch may wrongly be reported as covered when it was not.


Comment:

Reporting a branch as not covered when it was has no impact; only reported a branch as covered when it was not has a safety impact.

 impact_true HIGH
6 [gcov Options] Wrong (too high) reporting of line executions using Gcov

A line may wrongly be reported as covered when it was not.


Comment:

Reporting a line as not covered when it was has no impact; only reported a line as covered when it was not has a safety impact.

 impact_true HIGH
7 [gcov Options] Wrong format transformation

With this option, the gcov report is saved in the plain text intermediate format. This format is more condensed than the one in the standard form. Yet, all the relevant information should be reported.


Comment:

The intermediate format actually does not report exceptional lines that are not covered. This applies to C++ source code (see the corresponding Known Bug).

 impact_true HIGH
8 [genhtml Core functionalities] Report file corrupted

The HTML coverage report may be corrupted and therefore, unreadable.


Comment:

This error has no impact since if the reported code coverage is corrupted, the developer/tester will have to argue with some code is not covered.

 impact_false
/
 HIGH
9 [genhtml Core functionalities] The HTML summary does not match the actual coverage information

The summary information displayed in the HTML report may not match the actual coverage information reported in this HTML report.


Comment:

This error has a safety impact because the summary is often the primary source of information used to decide if the report should be investigated in detail or not.

 impact_true HIGH
10 [genhtml Options] Wrong (too high) reporting of branch executions using Genhtml

A branch may wrongly be reported as covered when it was not.


Comment:

Reporting a branch as not covered when it was has no impact; only reported a branch as covered when it was not has a safety impact.

 impact_true HIGH
11 [genhtml Options] Wrong (too high) reporting of line executions using Genhtml

A line may wrongly be reported as covered when it was not.


Comment:

Reporting a line as not covered when it was has no impact; only reported a line as covered when it was not has a safety impact.

 impact_true HIGH

Code coverage measurement using lcov and genhtml

Safety-Impact: yes
Inputs:
  • C/C++ compiled code:

    The C or C++ compiled code files considered for code coverage measurement
  • C/C++ source code - AUTOSAR-compliant (version 18-10):

    The C or C++ source code files considered for code coverage measurement

    The C or C++ source code files shall be compliant with AUTOSAR C++14 (18-10)
Outputs:
  • HTML coverage report:

    Generated coverage report in HTML format
  • lcov Coverage Report Output files:

    *.dat files

    The documents having extension *.dat are produced by Lcov as a result of its execution and are artifacts which are examined during the qualification.
Features:
  • Execution:

    The execution of the instrumented program is not directly a gcov/lcov feature, but it is required in order to generate the coverage data files.
  • genhtml Core functionalities:

    Requirements:

    - Create an HTML report file from lcov/gcov data files (convert data files)

    - Report a summary of the coverage data and coverage ratio (collect and convert log information)

  • genhtml Options:

    Container for the options accepted by genhtml

    [Detailled information has been removed here. More options are contained in the QKit]
  • lcov Core functionalities:

    Requirements:

    - Report what lines of C/C++ code are actually executed

    - Report what lines of C/C++ code are not executed
  • lcov Options:

    Container for the options accepted by lcov

    [Detailled infromation has been removed here. More options are contained in the QKit]
Potential errors:
# Error Safety-Impact Mitigations for detections/avoidance Final detection/avoidance probability
1 [Execution] The instrumented executable exhibits a different functional behavior than the uninstrumented executable

During the execution, the instrumented executable exhibits a different functional behavior than the uninstrumented executable.


Comment:

This error has a safety impact as the tests and the code coverage analysis are based on the instrumented version of an executable.

 impact_true HIGH
2 [Execution] The instrumented executable is not executed (not starting or not terminating)

The instrumented executable may not be successfully running.


Comment:

This error has no safety impact as no coverage report will be generated. Therefore, the end-user will detect it with a high probability when checking the list of coverage reports.

 impact_false
/
 HIGH
3 [genhtml Core functionalities] Report file corrupted

The HTML coverage report may be corrupted and therefore, unreadable.


Comment:

This error has no impact since if the reported code coverage is corrupted, the developer/tester will have to argue with some code is not covered.

 impact_false
/
 HIGH
4 [genhtml Core functionalities] The HTML summary does not match the actual coverage information

The summary information displayed in the HTML report may not match the actual coverage information reported in this HTML report.


Comment:

This error has a safety impact because the summary is often the primary source of information used to decide if the report should be investigated in detail or not.

 impact_true HIGH
5 [genhtml Options] Wrong (too high) reporting of branch executions using Genhtml

A branch may wrongly be reported as covered when it was not.


Comment:

Reporting a branch as not covered when it was has no impact; only reported a branch as covered when it was not has a safety impact.

 impact_true HIGH
6 [genhtml Options] Wrong (too high) reporting of line executions using Genhtml

A line may wrongly be reported as covered when it was not.


Comment:

Reporting a line as not covered when it was has no impact; only reported a line as covered when it was not has a safety impact.

 impact_true HIGH
7 [lcov Core functionalities] Reported code coverage too low

The reported coverage may be too low.


Comment:

This error has no impact since if the reported code coverage is lower than expected, the developer/tester will have to argue with some code is not covered.

 impact_false
/
 HIGH
8 [lcov Core functionalities] When reporting code coverage as percentage, lcov is rounding up to 100%

This error corresponds to the violation of the following requirement:

* When printing percentages, 0% and 100% are only printed when the values are exactly 0% and 100% respectively. Other values which would conventionally be rounded to 0% or 100% are instead printed as the nearest non-boundary value.

 impact_true HIGH
9 [lcov Options] Branch coverage is not reported

lcov does not report the branch coverage information (but only the line information).


Comment:

This potential error has no impact because the end-user will notice it in the generated HTML reports: the reports will not contain any information about the branch coverage.

 impact_false
/
 HIGH
10 [lcov Options] Coverage reporting of files from the wrong directory

lcov may report the coverage information from the wrong directory.

 impact_true HIGH
11 [lcov Options] Wrong (too high) reporting of branch executions using Lcov

A branch may wrongly be reported as covered when it was not.


Comment:

Reporting a branch as not covered when it was has no impact; only reported a branch as covered when it was not has a safety impact.

 impact_true HIGH
12 [lcov Options] Wrong (too high) reporting of line executions using Lcov

A line may wrongly be reported as covered when it was not.


Comment:

Reporting a line as not covered when it was has no impact; only reported a line as covered when it was not has a safety impact.

 impact_true HIGH

Qualifications

GCOV, LCOV and GENHTML QKit

Description:

The QKit is a standard Validas QKit using QST and TAU that automatically executes the qualification tests in the user environment.

This qualification kit covers the following tools: GCOV, LCOV (including GENHTML) as well as the following helper scripts (when applicable) - BAZEL CoverageOutputGenerator and BMW LcovMerger.

Two use cases are considered:

- Code coverage measurement using gcov and genhtml: This use case uses gcov, a helper script and genhtml

- Code coverage measurement using lcov and genhtml: This use case uses lcov and genhtml

Qualfication method:

VALIDAS_QKIT
Identifications:
Use Cases:

Test suites and test cases

TestSuite for gcov , lcov and genhtml

Description:

This testsuite contains tests for use cases described in the model

This TestSuites element is a container for the following test suites:

- AUTOSAR_compliant: Test cases based on the constructs allowed by AUTOSAR 18-10

- Tool_Perspective: Test cases to cover the potential errors at the interface between the tools and the artifacts

- and some other tests [details are removed for confidentiality reasons]
Flags: Has Requirements Based Tests , Has Negative Tests , Has Boundary Tests , Has Equivalence Tests , Has Error Guessing Tests
# tests: 1712
Errors:
  • Coverage reporting of files from the wrong directory
  • The HTML summary does not match the actual coverage information
  • When reporting code coverage as percentage, gcov is rounding up to 100%
  • When reporting code coverage as percentage, lcov is rounding up to 100%
  • Wrong (too high) reporting of branch executions using Gcov
  • Wrong (too high) reporting of branch executions using Genhtml
  • Wrong (too high) reporting of branch executions using Lcov
  • Wrong (too high) reporting of line executions using Gcov
  • Wrong (too high) reporting of line executions using Genhtml
  • Wrong (too high) reporting of line executions using Lcov
  • Wrong format transformation

Mitigations

# Mitigation Detection/Avoidance probability Detected/Avoided errors
1 [Check] BackToBack Test

Compare the generated results of the instrumented executable to the those of the uninstrumented executable


Comment:

This mitigation has a high detection probability because a modification of the behavior of a program due to the instrumentation will be systematically reproducible.

 HIGH
  • The instrumented executable exhibits a different functional behavior than the uninstrumented executable