Tutorial: build a coverage tracer

In this tutorial you’ll build a working statement-level code-coverage tool from scratch. Along the way you’ll meet raw handlers, file filtering, sandbox detection, and the tracing lifecycle hooks — the same ingredients that power the coverage.py example shipped with Pyccolo.

The idea is simple: every time a statement is about to run, record it. At the end, compare the set of statements we saw against the total number of statements in each file.

Step 1: record every statement

Start with a pyccolo.BaseTracer subclass that handles before_stmt. We only need to know which statement fired — its identity — so we register a raw handler with pyc.register_raw_handler. A raw handler receives the integer node id instead of a resolved ast.AST node, which is exactly what we want to use as a dictionary/set key (and it’s the fastest path). See Registering handlers for the difference.

from collections import Counter

import pyccolo as pyc


class CoverageTracer(pyc.BaseTracer):
    bytecode_caching_allowed = False

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.seen_stmts = set()
        self.stmt_count_by_fname = Counter()

    @pyc.register_raw_handler(pyc.before_stmt)
    def handle_stmt(self, _ret, stmt_id, frame, *_, **__):
        fname = frame.f_code.co_filename
        if stmt_id not in self.seen_stmts:
            self.stmt_count_by_fname[fname] += 1
            self.seen_stmts.add(stmt_id)

Why bytecode_caching_allowed = False? Pyccolo can cache the instrumented bytecode it compiles. For most tracers that’s a welcome speedup, but a coverage tool must see every statement on every run — a stale or partially-cached module could omit statements from the count. Turning caching off keeps the picture honest. (See Configuration attributes for the full list of these class-level switches.)

Step 2: choose which files to cover

By default, imported modules are not instrumented. You opt files in by overriding should_instrument_file(), which is called with each candidate module’s filename. Return True for the files you want to measure:

def should_instrument_file(self, filename: str) -> bool:
    if "test/" in filename or "examples" in filename:
        return False
    return "pyccolo" in filename

Here we cover the pyccolo package itself while skipping tests and examples. See Tracing real programs for the mechanics of import instrumentation.

Step 3: ignore sandbox frames

Code you run through pyc.exec / pyc.eval is compiled under a synthetic “sandbox” filename. If you only care about coverage of real files on disk, filter those frames out by testing the frame’s filename against pyc.SANDBOX_FNAME_PREFIX:

@pyc.register_raw_handler(pyc.before_stmt)
def handle_stmt(self, _ret, stmt_id, frame, *_, **__):
    fname = frame.f_code.co_filename
    if fname.startswith(pyc.SANDBOX_FNAME_PREFIX):
        return
    if stmt_id not in self.seen_stmts:
        self.stmt_count_by_fname[fname] += 1
        self.seen_stmts.add(stmt_id)

Step 4: report at the end

To emit a report when tracing finishes, override exit_tracing_hook(). We need the total number of statements per file to compute a ratio; a small ast.NodeVisitor that counts ast.stmt nodes does the job:

import ast


class CountStatementsVisitor(ast.NodeVisitor):
    def __init__(self):
        self.num_stmts = 0

    def generic_visit(self, node):
        if isinstance(node, ast.stmt) and not isinstance(node, ast.Raise):
            self.num_stmts += 1
        super().generic_visit(node)


def count_statements(path: str) -> int:
    with open(path) as f:
        visitor = CountStatementsVisitor()
        visitor.visit(ast.parse(f.read()))
    return visitor.num_stmts

def exit_tracing_hook(self) -> None:
    total = 0
    for fname in sorted(self.stmt_count_by_fname):
        if fname.startswith(pyc.SANDBOX_FNAME_PREFIX):
            continue
        seen = self.stmt_count_by_fname[fname]
        in_file = count_statements(fname)
        total += in_file
        print(f"{fname}: seen={seen}, total={in_file}, ratio={seen / in_file:.3f}")
    if total:
        print(f"overall ratio: {len(self.seen_stmts) / total:.3f}")

Step 5: run it

Activate the tracer as a context manager and run some code under it. Coverage of imported modules works because the tracer is active while the import happens:

with CoverageTracer.instance():
    import some_package  # its statements are now counted as they execute

To measure coverage of a whole test suite, drive the test runner from inside the tracing context and install the import hook so freshly-imported modules get instrumented. The shipped example does exactly this with pytest:

from pyccolo.import_hooks import patch_meta_path

with CoverageTracer.instance():
    with patch_meta_path(pyc._TRACER_STACK):
        pytest.console_main()

That’s a complete, if minimal, coverage tool — a couple of handlers and two lifecycle hooks. The full version lives in pyccolo/examples/coverage.py.

Where to next

• Coverage adds real overhead. Guards and performance shows how to deactivate instrumentation once you’ve seen a statement for the first time.

• The BaseTracer page catalogs every hook and method you can override.