Program Instrumentation Enabling Coverage Measurement in SW Testing

Requirements

• Ubuntu 20.04
• Python 3.6+
• Python modules from requirements.txt
• install: llvm llvm-dev clang make python3 python3-pip

Structure

• example/ - example project compiled with tforcov with coverage criteria measurement
• src/ - source files
  • src/tforcov.py - main script to be run
• test/ - tests
  • test/e2e - end-to-end tests
  • test/unit - unit tests
• Makefile - targets:
  • docker_run - run container with prepared environment for this tool locally
  • docker_connect - connect to container created with docker_run so as not to create it from scratch
  • test_unit - run unit tests
  • test_e2e - run end-to-end tests
  • python_black - format python code with Black
• requirements.txt - python modules dependencies
• requirements-test.txt - python modules dependencies for test only

CFG

Imported fields in dumped CFG hierarchy:

• 'module'
  • 'module file' - base file that is compiled, used as a name
  • 'function'
    • 'function filename' - filename of source C++ file
    • 'function directory' - directory of the source C++ file
    • 'bb' - basic block
      • 'bb id' - basic block unique id within module
      • 'bb succ' - successor of basic block
      • 'bb start' - number of line basic block starts at
      • 'bb end' - number of line basic block ends at
      • 'bb load' - use of data
      • 'bb store' - definition of data

Testing

Unit tests

python3 -m pip install --user -r requirements-test.txt -r requirements.txt
cd src && python3 -m pytest ../test/unit -v --tb=short --cov . --cov-branch

End-to-end tests

End-to-end tests also require requirements mentioned above

python3 -m pip install --user -r requirements-test.txt -r requirements.txt
cd src && python3 -m pytest ../test/e2e --tb=native --capture=tee-sys -v

Example

Run example as follows:

./run.sh

To see how the use of tforcov tools works, inspect run.sh first.

TFORCOV=../src/tforcov.py

# instrument program
$TFORCOV instrument start
make
$TFORCOV instrument stop

# run instrumented program
./executable

# evaluate coverage criteria
$TFORCOV coverage

Output will be present in example/tforcov_output folder in the end.

Coding style

Tforcov coding style is validated with flake8 and its configuration file .flake8 is written to be compatible with default settings of black.

Python code is documented with reST (reStructuredText) PEP 287