Syntax augmentation

Pyccolo can go beyond instrumenting existing Python: a tracer can define new surface syntax. It does this with an pyccolo.AugmentationSpec, which declares a source-level token → replacement rewrite. Pyccolo remembers where the rewrite happened, so a handler can attach to the resulting AST node.

Note

Syntax augmentation requires Python >= 3.8.

A first example: optional chaining

JavaScript-style optional chaining rewrites ?. down to a plain ., then resolves the access to None whenever the receiver is None. The augmentation spec that drives it is a single declaration:

import pyccolo as pyc

optional_chaining_spec = pyc.AugmentationSpec(
    aug_type=pyc.AugmentationType.dot_suffix, token="?.", replacement="."
)

A complete, tested implementation of optional chaining and nullish coalescing (??) ships in pyccolo/examples/optional_chaining.py:

import pyccolo as pyc
from pyccolo.examples.optional_chaining import ScriptOptionalChainer

with ScriptOptionalChainer:
    pyc.exec("bar = None\nprint(bar?.foo)")  # -> None

Augmentation types

pyccolo.AugmentationType enumerates where a token may be anchored:

  • prefix / suffix — before / after an identifier or expression;

  • dot_prefix / dot_suffix — around an attribute-access dot (as in the ?. example);

  • binop — a binary-operator position (as with the |> pipeline operator);

  • custom — a context-sensitive rewrite expressed via pyccolo.CustomRewrite (below).

The mechanism works by rewriting an illegal token span (e.g. ?. or |>) into a legal one (. or bitwise-or |), parsing the now-valid Python, and then associating the resulting AST node with the augmentation so the corresponding handler runs. Because a single event-emission transform is shared by every handler that cares about it, features compose without conflicting AST rewrites (see Composing tracers).

A larger showcase: pipescript

The most complete demonstration of syntax augmentation is pipescript, which layers a whole pipe-and-placeholder dialect on top of Python:

# in IPython / Jupyter, after `%load_ext pipescript`
result = arrays |> map[$
  |> $array[np.isfinite($array)]
  |> np.abs
  |> np.max($, initial=1.0)
] |> max

Under the hood, pipescript rewrites illegal token spans like |> to legal ones (here, bitwise-or |), then uses Pyccolo to associate the resulting ast.BinOp with the |> operator and run the corresponding handler.

Beyond single-token replacement

An AugmentationSpec also supports paired-delimiter (brace-block) augmentation via its close_token / close_replacement fields (its is_paired property reports whether they are set). This powers statement-bodied name{ ... } blocks — see the block_lambda, func_block, and brace_subscript entries in the Example gallery.

For rewrites the token and paired passes cannot express, the custom type and pyccolo.CustomRewrite provide a context-sensitive extension point.

Full coverage of the augmentation surface (including transform/untransform round trips and position remapping) lives in test/test_syntax_augmentation.py. See also Source-to-source: transform, untransform, and pure mode for turning augmented syntax into plain Python source and back.