Core¶

The scheme-agnostic half. A new mangling scheme is written against these and nothing else; Architecture has the layering rules.

The builder protocol¶

demangle.core.builder ¶

The contract between a parser and its output.

Every parser in this package is written against Builder and never constructs its own strings or nodes. It recognises a grammar production and reports it -- builder.pointer( inner), builder.template(base, args) -- leaving what that becomes to the builder.

The reason is that the two things callers want from a demangler have opposite cost profiles. Labelling every symbol in a binary wants a string and nothing else, several hundred thousand times. Inspecting one symbol's template arguments wants a tree. Writing the parser twice to serve both would guarantee the two copies disagree, and the ways they would disagree are exactly the subtle ones -- declarator placement, substitution numbering -- that are hardest to notice.

So: one parser, two builders (SpellingBuilder, AstBuilder), and a third is a class rather than a fork.

Values passed between these methods are handles. A parser may hold one, pass it back in, and ask the builder to spell it, but must never inspect it -- a handle is a Spelling under one builder and a Node under another. spell() exists for the few places where a grammar genuinely requires the text of something already built (naming the class in Foo::Foo, for instance), and is the only legal way to look inside.

Handle `module-attribute` ¶

Handle = Any

An opaque builder product. Its concrete type is the builder's business.

Builder ¶

Bases: Protocol

What a parser may ask for. Implementations choose what to produce.

builtin ¶

builtin(spelling)

A primitive type: int, void, char.

Source code in src/demangle/core/builder.py

def builtin(self, spelling: str) -> Handle:
    """A primitive type: `int`, `void`, `char`."""

name ¶

name(text)

A single identifier, already decoded from its length-prefixed form.

Source code in src/demangle/core/builder.py

def name(self, text: str) -> Handle:
    """A single identifier, already decoded from its length-prefixed form."""

raw ¶

raw(text)

Pre-rendered text with no further structure.

The escape hatch for productions whose spelling a scheme fixes outright, such as std::nullptr_t. Reach for a specific method first: text passed here is invisible to structured consumers.

Source code in src/demangle/core/builder.py

def raw(self, text: str) -> Handle:
    """Pre-rendered text with no further structure.

    The escape hatch for productions whose spelling a scheme fixes outright, such
    as `std::nullptr_t`. Reach for a specific method first: text passed here is
    invisible to structured consumers.
    """

literal ¶

literal(kind, value)

A constant appearing in a type, such as a non-type template argument.

Source code in src/demangle/core/builder.py

def literal(self, kind: Handle | None, value: str) -> Handle:
    """A constant appearing in a type, such as a non-type template argument."""

expression ¶

expression(form, parts)

A constant expression: sizeof (T), a + b, x ? y : z.

parts interleaves the fixed text of the production with the handles of its operands, in output order, so a builder that wants text concatenates and one that wants a tree keeps the boundaries. form names the shape -- binary, call, sizeof, conditional -- so a consumer can ask what an expression is without matching on its spelling.

One method rather than one per operator, because the alternative is fifteen methods that every future builder has to implement and that still would not cover the next operator someone mangles. The parser already owns operator spelling: it comes from the tables in tables.py, which exist to be checked against the ABI. What the builder decides is structure.

Brackets are a part like any other. Whether an operand needs them is a question about precedence that only the parser can answer, so it arrives here already settled, as "(" and ")" around the operand it applies to.

Source code in src/demangle/core/builder.py

def expression(self, form: str, parts: Sequence["Handle | str"]) -> Handle:
    """A constant expression: `sizeof (T)`, `a + b`, `x ? y : z`.

    `parts` interleaves the fixed text of the production with the handles of its
    operands, in output order, so a builder that wants text concatenates and one
    that wants a tree keeps the boundaries. `form` names the shape -- `binary`,
    `call`, `sizeof`, `conditional` -- so a consumer can ask what an expression *is*
    without matching on its spelling.

    One method rather than one per operator, because the alternative is fifteen
    methods that every future builder has to implement and that still would not
    cover the next operator someone mangles. The parser already owns operator
    spelling: it comes from the tables in `tables.py`, which exist to be checked
    against the ABI. What the builder decides is structure.

    Brackets are a part like any other. Whether an operand needs them is a question
    about precedence that only the parser can answer, so it arrives here already
    settled, as `"("` and `")"` around the operand it applies to.
    """

qualified ¶

qualified(parts)

A scoped name: the parts of a::b::c, outermost first.

Source code in src/demangle/core/builder.py

def qualified(self, parts: Sequence[Handle]) -> Handle:
    """A scoped name: the parts of `a::b::c`, outermost first."""

template ¶

template(base, arguments)

A template specialisation, base<arguments...>.

Source code in src/demangle/core/builder.py

def template(self, base: Handle, arguments: Sequence[Handle]) -> Handle:
    """A template specialisation, `base<arguments...>`."""

qualify ¶

qualify(inner, qualifiers)

Apply cv-qualifiers, in the canonical order the caller has already imposed.

Source code in src/demangle/core/builder.py

def qualify(self, inner: Handle, qualifiers: Sequence[str]) -> Handle:
    """Apply cv-qualifiers, in the canonical order the caller has already imposed."""

pointer ¶

pointer(inner)

inner*.

Source code in src/demangle/core/builder.py

def pointer(self, inner: Handle) -> Handle:
    """`inner*`."""

reference ¶

reference(inner)

inner&.

Source code in src/demangle/core/builder.py

def reference(self, inner: Handle) -> Handle:
    """`inner&`."""

rvalue_reference ¶

rvalue_reference(inner)

inner&&.

Source code in src/demangle/core/builder.py

def rvalue_reference(self, inner: Handle) -> Handle:
    """`inner&&`."""

member_pointer ¶

member_pointer(owner, inner)

inner Owner::* -- a pointer to member.

Source code in src/demangle/core/builder.py

def member_pointer(self, owner: Handle, inner: Handle) -> Handle:
    """`inner Owner::*` -- a pointer to member."""

array ¶

array(inner, dimension)

inner [dimension]. An empty dimension means an unbounded array.

Source code in src/demangle/core/builder.py

def array(self, inner: Handle, dimension: str) -> Handle:
    """`inner [dimension]`. An empty dimension means an unbounded array."""

function ¶

function(returns, parameters, suffix='', name=None)

A function type or a whole function declaration.

returns is None where the scheme encodes no return type -- constructors, destructors, conversion operators, and plain non-template functions under the Itanium ABI. suffix carries what trails the parameter list and binds to the function rather than the return type: cv-qualifiers, a ref-qualifier, noexcept. name is the declarator, absent for a bare function type.

Source code in src/demangle/core/builder.py

def function(
    self,
    returns: Handle | None,
    parameters: Sequence[Handle],
    suffix: str = "",
    name: Handle | None = None,
) -> Handle:
    """A function type or a whole function declaration.

    `returns` is None where the scheme encodes no return type -- constructors,
    destructors, conversion operators, and plain non-template functions under the
    Itanium ABI. `suffix` carries what trails the parameter list and binds to the
    function rather than the return type: cv-qualifiers, a ref-qualifier,
    `noexcept`. `name` is the declarator, absent for a bare function *type*.
    """

pack ¶

pack(inner)

An unexpanded parameter pack expansion, inner....

Source code in src/demangle/core/builder.py

def pack(self, inner: Handle) -> Handle:
    """An *unexpanded* parameter pack expansion, `inner...`."""

parameter_pack ¶

parameter_pack(members)

A pack of concrete arguments.

Must stay a sequence rather than collapse to joined text: a declarator applied to a pack applies to every member, so Dp O T_ over three arguments is three rvalue references.

Source code in src/demangle/core/builder.py

def parameter_pack(self, members: Sequence[Handle]) -> Handle:
    """A pack of concrete arguments.

    Must stay a sequence rather than collapse to joined text: a declarator applied
    to a pack applies to every member, so `Dp O T_` over three arguments is three
    rvalue references.
    """

vendor_qualify ¶

vendor_qualify(inner, qualifier)

A vendor extended qualifier, spelled after the type it applies to.

Source code in src/demangle/core/builder.py

def vendor_qualify(self, inner: Handle, qualifier: str) -> Handle:
    """A vendor extended qualifier, spelled after the type it applies to."""

special ¶

special(label, inner)

A symbol that is about an entity: vtable for Foo, typeinfo for Bar.

Source code in src/demangle/core/builder.py

def special(self, label: str, inner: Handle) -> Handle:
    """A symbol that is *about* an entity: `vtable for Foo`, `typeinfo for Bar`."""

decorated ¶

decorated(inner, decoration)

A symbol carrying a symbol-table decoration: an ELF version, a clone suffix.

Kept as structure rather than folded into text so a caller can ask which copy of a function a symbol names, or strip the version and compare.

Source code in src/demangle/core/builder.py

def decorated(self, inner: Handle, decoration: str) -> Handle:
    """A symbol carrying a symbol-table decoration: an ELF version, a clone suffix.

    Kept as structure rather than folded into text so a caller can ask which copy of
    a function a symbol names, or strip the version and compare.
    """

spell ¶

spell(handle, declarator='')

Render a handle, placing declarator in the type's declarator position.

Source code in src/demangle/core/builder.py

def spell(self, handle: Handle, declarator: str = "") -> str:
    """Render a handle, placing `declarator` in the type's declarator position."""

size ¶

size(handle)

How many characters handle would render to, without rendering it.

Must be O(1). This is what lets a parser enforce an output bound while building: the substitution scheme allows each component to be assembled from two copies of an earlier one, so output can double every few input bytes. Checking the bound by measuring spell() would mean materialising the very string the bound exists to prevent -- a few hundred bytes of input can reach gigabytes.

Source code in src/demangle/core/builder.py

def size(self, handle: Handle) -> int:
    """How many characters `handle` would render to, without rendering it.

    Must be O(1). This is what lets a parser enforce an output bound *while*
    building: the substitution scheme allows each component to be assembled from two
    copies of an earlier one, so output can double every few input bytes. Checking
    the bound by measuring `spell()` would mean materialising the very string the
    bound exists to prevent -- a few hundred bytes of input can reach gigabytes.
    """

Spelling¶

demangle.core.spelling ¶

The fast builder: C++ declaration syntax, built directly as text.

C++ does not write a type before the name, it writes it around the name. The type "pointer to function taking char and returning int" is spelled int (*)(char), and giving it a name puts that name in a hole in the middle: int (*f)(char). Arrays do the same: int a[10], and a pointer to one is int (*)[10].

So a partially built type is a pair of strings, left and right, and rendering is left + declarator + right. Every function here exists to keep that hole in the right place as declarators nest. It is the part of a demangler most often subtly wrong, which is why it is one small module with its own tests rather than logic spread through three parsers.

Spellings are immutable. Nothing here mutates a handle it was given, so a builder product can be stored in a substitution table and reused without defensive copying -- which matters, because that is exactly what the Itanium back-reference scheme does.

Spelling ¶

Spelling(
    left,
    right="",
    is_function=False,
    is_array=False,
    ref_kind="",
    members=None,
)

One type or name, split around its declarator position.

Source code in src/demangle/core/spelling.py

def __init__(self, left, right="", is_function=False, is_array=False, ref_kind="", members=None):
    self.left = left
    self.right = right
    self.is_function = is_function
    self.is_array = is_array
    #: "", "&" or "&&", for reference collapsing.
    self.ref_kind = ref_kind
    #: For a parameter pack, its members; None otherwise. Kept as a sequence because
    #: a declarator applies to every member: `Dp O T_` over three arguments is three
    #: rvalue references.
    self.members = members

spell ¶

spell(declarator='')

This type with declarator -- a name -- written where a declarator goes.

Both references space the name off the type unless the type has a right part to put after it, which is what NonTypeTemplateParamDecl::printLeft asks hasRHSComponent in LLVM's ItaniumDemangle.h: int* $N and int A::* $N against int$N [3] and int (*$N) [3]. A function type's left half already ends in a space, so int $N() comes out with exactly one either way. Verified against both references over every declarator shape; int*$N, which is what a rule about the last character gives, is what neither writes.

Source code in src/demangle/core/spelling.py

def spell(self, declarator=""):
    """This type with `declarator` -- a name -- written where a declarator goes.

    Both references space the name off the type unless the type has a *right* part
    to put after it, which is what `NonTypeTemplateParamDecl::printLeft` asks
    `hasRHSComponent` in LLVM's `ItaniumDemangle.h`: `int* $N` and `int A::* $N`
    against `int$N [3]` and `int (*$N) [3]`. A function type's left half already
    ends in a space, so `int $N()` comes out with exactly one either way. Verified
    against both references over every declarator shape; `int*$N`, which is what a
    rule about the last character gives, is what neither writes.
    """
    if not declarator:
        return self.left + self.right if self.right else self.left
    left = self.left
    if not left:
        return declarator + self.right
    joiner = "" if self.right or left.endswith(_TIGHT_ENDINGS) else " "
    return left + joiner + declarator + self.right

SpellingBuilder ¶

SpellingBuilder(
    legacy_angle_spacing=False,
    gnu_clone_suffix=False,
    collapse_duplicate_qualifiers=False,
    tight_after_star=False,
    tight_member_array=True,
)

Bases: Builder

Builds C++ declaration text. The backend behind demangle().

Stateless, so one instance is shared by every parse rather than allocated per name.

Source code in src/demangle/core/spelling.py

def __init__(
    self,
    legacy_angle_spacing=False,
    gnu_clone_suffix=False,
    collapse_duplicate_qualifiers=False,
    tight_after_star=False,
    tight_member_array=True,
):
    #: Write `Foo<Bar<int> >` (pre-C++11 `>>` lexing) and `operator<< <int>` rather
    #: than `operator<<<int>`. GNU c++filt does; llvm-cxxfilt does not.
    self.legacy_angle_spacing = legacy_angle_spacing
    #: Write a clone suffix as `[clone .cold]` (GNU c++filt) rather than `(.cold)`.
    self.gnu_clone_suffix = gnu_clone_suffix
    #: Write `void (*(*)())()` rather than `void (* (*)())()`, as GNU c++filt does.
    self.tight_after_star = tight_after_star
    #: Write `int(A::*) [3]` rather than `int (A::*) [3]`, as llvm-cxxfilt does.
    self.tight_member_array = tight_member_array
    #: Print `int const` for `K K i`, rather than llvm-cxxfilt's `int const const`
    #: ([basic.type.qualifier]: at most one of each). The outer one wins, so `K V K i`
    #: and `K K V i` both read `volatile const`, as c++filt prints them (checked over
    #: all 39 sequences of one to three). An array passes the qualifiers through to
    #: its element; every other declarator stops them.
    self.collapse_duplicate_qualifiers = collapse_duplicate_qualifiers

qualify ¶

qualify(inner, qualifiers, cv=True)

Apply qualifiers to inner, on the right, the way C++ writes them.

cv says whether these are cv-qualifiers, which is what decides whether a repeat collapses. _Complex and _Imaginary go through here too and are not cv-qualifiers: [basic.type.qualifier] folds a duplicate const, and nothing folds a duplicate _Imaginary. c++filt 2.42 writes signed char _Imaginary _Imaginary for _Z1fGGa; collapsing it to one would lose a word of the name, and only the gnu style collapses at all.

Source code in src/demangle/core/spelling.py

def qualify(self, inner, qualifiers, cv=True):
    """Apply `qualifiers` to `inner`, on the right, the way C++ writes them.

    `cv` says whether these are cv-qualifiers, which is what decides whether a
    repeat collapses. `_Complex` and `_Imaginary` go through here too and are not
    cv-qualifiers: `[basic.type.qualifier]` folds a duplicate `const`, and nothing
    folds a duplicate `_Imaginary`. `c++filt` 2.42 writes `signed char _Imaginary
    _Imaginary` for `_Z1fGGa`; collapsing it to one would lose a word of the name,
    and only the gnu style collapses at all.
    """
    if not qualifiers:
        return inner
    if inner.members is not None:
        return pack_of(self.qualify(member, qualifiers, cv=cv) for member in inner.members)
    if inner.is_function:
        # cv on a function type qualifies the implicit object parameter, so it
        # trails the parameter list rather than the return type.
        return Spelling(inner.left, inner.right + " " + " ".join(qualifiers), is_function=True)
    if not (self.collapse_duplicate_qualifiers and cv):
        # Both `int const` and `int* const` are postfix in C++, as the references agree.
        left = inner.left + " " + " ".join(qualifiers)
        right = _respace_bound(left, inner.right) if inner.is_array else inner.right
        return Spelling(left, right, is_array=inner.is_array)
    # Repeated qualifiers move to the end: the outermost is the one c++filt prints last.
    stem = inner.stem if inner.cv else inner.left
    combined = tuple(q for q in inner.cv if q not in qualifiers) + tuple(qualifiers)
    left = stem + " " + " ".join(combined)
    right = _respace_bound(left, inner.right) if inner.is_array else inner.right
    return QualifiedSpelling(left, right, is_array=inner.is_array, cv=combined, stem=stem)

pack ¶

pack(inner)

An unexpanded pack expansion: the whole type, then ....

The reference's ParameterPackExpansion prints its child -- both halves -- and appends the ellipsis, so a declarator type keeps its shape and the dots follow it: void ()..., void (*)()..., int [3].... Putting them in the left half alone would put them where a declarator's name goes, void (*...)() and int... [3], which is not what any reference prints. A type with no right half is unchanged.

Source code in src/demangle/core/spelling.py

def pack(self, inner):
    """An unexpanded pack expansion: the whole type, then `...`.

    The reference's `ParameterPackExpansion` prints its child -- both halves -- and
    appends the ellipsis, so a declarator type keeps its shape and the dots follow
    it: `void ()...`, `void (*)()...`, `int [3]...`. Putting them in the left half
    alone would put them where a declarator's name goes, `void (*...)()` and
    `int... [3]`, which is not what any reference prints. A type with no right half
    is unchanged.
    """
    return Spelling(inner.left + inner.right + "...")

vendor_qualify ¶

vendor_qualify(inner, qualifier)

A vendor qualifier goes after the whole type, declarator and all.

The reference prints the type and then the extension, so a function type comes out void () block_pointer -- not void block_pointer(), which is what putting the word in the left half alone gives for anything that has a right half.

Distributes over a pack like every other declarator: an empty pack gives nothing rather than a bare enable_if, a qualifier on nothing.

Source code in src/demangle/core/spelling.py

def vendor_qualify(self, inner, qualifier):
    """A vendor qualifier goes after the *whole* type, declarator and all.

    The reference prints the type and then the extension, so a function type comes
    out `void () block_pointer` -- not `void block_pointer()`, which is what putting
    the word in the left half alone gives for anything that has a right half.

    Distributes over a pack like every other declarator: an empty pack gives nothing
    rather than a bare ` enable_if`, a qualifier on nothing.
    """
    if inner.members is not None:
        return pack_of(self.vendor_qualify(member, qualifier) for member in inner.members)
    return Spelling(inner.spell() + " " + qualifier)

members ¶

members(handle)

The members of a parameter pack, or None if this is not one.

Source code in src/demangle/core/spelling.py

def members(self, handle):
    """The members of a parameter pack, or None if this is not one."""
    return handle.members

pack_of ¶

pack_of(members)

A parameter pack, spelled as its members and still addressable as a sequence.

Nested packs are spliced. A pack whose one member is an expansion of an empty pack is empty, not a pack of one empty thing -- otherwise AnalysisManager<T_, J Dp T0_ E> with T0_ bound to nothing prints the separator for an argument that is not there.

Source code in src/demangle/core/spelling.py

def pack_of(members):
    """A parameter pack, spelled as its members and still addressable as a sequence.

    Nested packs are spliced. A pack whose one member is an expansion of an empty pack
    is empty, not a pack of one empty thing -- otherwise `AnalysisManager<T_, J Dp T0_ E>`
    with `T0_` bound to nothing prints the separator for an argument that is not there.
    """
    flattened = []
    for member in members:
        if member.members is not None:
            flattened.extend(member.members)
        else:
            flattened.append(member)
    flattened = tuple(flattened)
    return Spelling(", ".join([member.left + member.right for member in flattened]), members=flattened)

The tree¶

The node classes a tree is made of, and the builder a scheme hands them to. Every class declares __match_args__, so a caller can match on the shape of a subtree rather than compare node.kind against a string:

import demangle
from demangle.core.ast import Builtin, Pointer

match demangle.parse("_Z1fPi").parameters[0]:
    case Pointer(Builtin(spelling)):
        print("pointer to", spelling)

A scheme's own node kinds are under its section on the schemes page, and demangle.node_kinds() lists the kind strings a scheme can produce.

demangle.core.ast ¶

The structured builder: a tree a caller can walk.

demangle() answers "what does this name say". parse() answers "what are its parts" -- which namespace, which template arguments, how many parameters and of what types. Tools that rename functions, group symbols by library, or match call signatures need the second, and recovering it by pattern-matching the string is not workable: C++ declaration syntax nests, and nesting is what regular expressions cannot follow.

Every node re-emits itself into any builder through build(). That is what keeps this module free of rendering logic: turning a tree back into text drives SpellingBuilder over it, so there is exactly one implementation of declarator placement in the package and a tree can never disagree with the fast path. It also means a new output format -- JSON, HTML, a token stream -- is a builder, not a second traversal to keep in sync.

AST_BUILDER `module-attribute` ¶

AST_BUILDER = AstBuilder()

Node ¶

Base class for every parsed component.

__slots__ throughout: a single C++ symbol can be a few hundred nodes and a binary a few hundred thousand symbols, so per-instance dictionaries are not affordable.

children ¶

children()

Direct children, in source order. Leaves return an empty tuple.

Source code in src/demangle/core/ast.py

def children(self):
    """Direct children, in source order. Leaves return an empty tuple."""
    return ()

to_dict ¶

to_dict()

This subtree as plain data -- dicts, lists and strings -- for json.dumps.

Every node becomes a dict with a kind and the fields that node kind has, so the role of each child survives: a function has returns, parameters and name, not three anonymous children. A field holding a node becomes a nested dict and one holding a sequence of them becomes a list; anything else -- a spelling, a qualifier list, a flag -- is carried as it is.

kind is the vocabulary to switch on, and demangle.node_kinds() is all of it.

A node reached more than once is written out once, with an id, and afterwards as {"$ref": id}. This is not a size optimisation, it is the difference between terminating and not: the structure is a graph, not a tree. Itanium's substitutions make a component reachable from several places, and a Rust node names its own children twice over -- parts orders them and base and arguments say what they are -- so writing each occurrence out in full doubles per level. A real symbol from the Rust toolchain costs seconds and hundreds of megabytes that way, and there is no bound on how much worse it can get.

Source code in src/demangle/core/ast.py

def to_dict(self):
    """This subtree as plain data -- dicts, lists and strings -- for `json.dumps`.

    Every node becomes a dict with a `kind` and the fields that node kind has, so the
    *role* of each child survives: a `function` has `returns`, `parameters` and
    `name`, not three anonymous children. A field holding a node becomes a nested
    dict and one holding a sequence of them becomes a list; anything else -- a
    spelling, a qualifier list, a flag -- is carried as it is.

    `kind` is the vocabulary to switch on, and `demangle.node_kinds()` is all of it.

    A node reached more than once is written out once, with an `id`, and afterwards
    as `{"$ref": id}`. This is not a size optimisation, it is the difference between
    terminating and not: the structure is a *graph*, not a tree. Itanium's
    substitutions make a component reachable from several places, and a Rust node
    names its own children twice over -- `parts` orders them and `base` and
    `arguments` say what they are -- so writing each occurrence out in full doubles
    per level. A real symbol from the Rust toolchain costs seconds and hundreds
    of megabytes that way, and there is no bound on how much worse it can get.
    """
    return rendered(lambda: _emit(self, _shared_nodes(self), {}))

build ¶

build(builder)

Re-emit this subtree into builder, returning its handle.

Source code in src/demangle/core/ast.py

def build(self, builder):
    """Re-emit this subtree into `builder`, returning its handle."""
    raise NotImplementedError

walk ¶

walk()

Yield this node and every descendant, depth first.

Iterative rather than recursive, and not for speed: a yield from per level means one interpreter frame per level for the whole traversal, so a tree well inside max_depth -- which bounds the parse, in frames of a different size -- could still exhaust the stack here. An explicit stack cannot.

Source code in src/demangle/core/ast.py

def walk(self):
    """Yield this node and every descendant, depth first.

    Iterative rather than recursive, and not for speed: a `yield from` per level
    means one interpreter frame per level for the whole traversal, so a tree well
    inside `max_depth` -- which bounds the *parse*, in frames of a different size --
    could still exhaust the stack here. An explicit stack cannot.
    """
    stack = [self]
    while stack:
        node = stack.pop()
        yield node
        children = node.children()
        if children:
            stack.extend(reversed(children))

find ¶

find(kind)

Yield every descendant of the given kind, including this node.

Source code in src/demangle/core/ast.py

def find(self, kind):
    """Yield every descendant of the given `kind`, including this node."""
    return (node for node in self.walk() if node.kind == kind)

render ¶

render()

The text of this subtree, under the default spelling policy.

spell() is the method to reach for -- it takes the style the tree was parsed under, which this cannot. A scheme whose spelling rules do not fit C++ declarator syntax carries its fragments as text and renders by concatenating them, so render() is what its own nodes and its own tests use, and spell() delegates to it.

It is defined on the base class so every node answers it: walking a tree and asking each node for its text -- the obvious thing to do with walk() -- must not fail partway through on a Decorated wrapper or an Itanium node. For a node that carries pre-rendered text this is that text; for one spelled from structure it is spell() with no declarator, which is the same string.

Source code in src/demangle/core/ast.py

def render(self):
    """The text of this subtree, under the default spelling policy.

    `spell()` is the method to reach for -- it takes the `style` the tree was parsed
    under, which this cannot. A scheme whose spelling rules do not fit C++
    declarator syntax carries its fragments as text and renders by concatenating
    them, so `render()` is what its own nodes and its own tests use, and `spell()`
    delegates to it.

    It is defined on the base class so every node answers it: walking a tree and
    asking each node for its text -- the obvious thing to do with `walk()` -- must
    not fail partway through on a `Decorated` wrapper or an Itanium node. For a node
    that carries pre-rendered text this is that text; for one spelled from structure
    it is `spell()` with no declarator, which is the same string.
    """
    return self.spell()

spell ¶

spell(declarator='', style=None)

Render as declaration text.

style selects the spelling policy, and should be the style the tree was parsed under: some choices -- whether a std:: abbreviation is expanded, for instance -- are made during parsing and are already baked into the tree, so spelling a gnu-parsed tree with LLVM's builder gives a mixture of the two.

Source code in src/demangle/core/ast.py

def spell(self, declarator="", style=None):
    """Render as declaration text.

    `style` selects the spelling policy, and should be the style the tree was parsed
    under: some choices -- whether a `std::` abbreviation is expanded, for instance --
    are made during parsing and are already baked into the tree, so spelling a
    gnu-parsed tree with LLVM's builder gives a mixture of the two.
    """
    builder = get_style(style).spelling_builder
    return rendered(lambda: builder.spell(self.build(builder), declarator))

Builtin ¶

Builtin(spelling)

Bases: Node

A primitive type.

Source code in src/demangle/core/ast.py

def __init__(self, spelling):
    self.spelling = spelling

Name ¶

Name(text)

Bases: Node

A single identifier.

Source code in src/demangle/core/ast.py

def __init__(self, text):
    self.text = text

Raw ¶

Raw(text)

Bases: Node

Text whose spelling the scheme fixes outright, with no further structure.

Source code in src/demangle/core/ast.py

def __init__(self, text):
    self.text = text

Literal ¶

Literal(type_, value)

Bases: Node

A constant value, such as a non-type template argument.

Source code in src/demangle/core/ast.py

def __init__(self, type_, value):
    self.type = type_
    self.value = value

Expression ¶

Expression(form, parts)

Bases: Node

A constant expression appearing in a type or a template argument.

parts interleaves fixed text with operand subtrees, in output order, exactly as the parser reported them -- so rendering concatenates and cannot spell the expression differently from the text path. form says which shape it is: binary, unary, call, conditional, sizeof, and so on.

Brackets, where the spelling needs them, are plain parts. Precedence is settled by the parser before it reports the production, because only the parser knows what the operand was; a consumer reading the tree sees the operands, not the punctuation.

Source code in src/demangle/core/ast.py

def __init__(self, form, parts):
    self.form = form
    self.parts = tuple(parts)

operands `property` ¶

operands

Just the operands, without the punctuation between them.

Qualified ¶

Qualified(parts)

Bases: Node

A scoped name, a::b::c, outermost part first.

Source code in src/demangle/core/ast.py

def __init__(self, parts):
    self.parts = tuple(parts)

Template ¶

Template(base, arguments, angle_space=True)

Bases: Node

A template specialisation, base<arguments...>.

Source code in src/demangle/core/ast.py

def __init__(self, base, arguments, angle_space=True):
    self.base = base
    self.arguments = tuple(arguments)
    #: Whether a style separating consecutive `>` should do so here. False after an
    #: empty trailing pack, where GNU c++filt writes `A<B<int>>`.
    self.angle_space = angle_space

Qualify ¶

Qualify(inner, qualifiers, cv=True)

Bases: Node

Qualifiers applied to a type, written on the right.

cv says whether they are cv-qualifiers. _Complex and _Imaginary arrive here too and are not, which matters only because a repeated cv-qualifier collapses under the gnu style and a repeated _Imaginary does not. Carried on the node so that a tree spells what the text path spelled.

Source code in src/demangle/core/ast.py

def __init__(self, inner, qualifiers, cv=True):
    self.inner = inner
    self.qualifiers = tuple(qualifiers)
    self.cv = cv

Pointer ¶

Pointer(inner)

Bases: _Unary

A pointer to inner: int*.

Source code in src/demangle/core/ast.py

def __init__(self, inner):
    self.inner = inner

Reference ¶

Reference(inner)

Bases: _Unary

An lvalue reference to inner: int&.

Source code in src/demangle/core/ast.py

def __init__(self, inner):
    self.inner = inner

RValueReference ¶

RValueReference(inner)

Bases: _Unary

An rvalue reference to inner: int&&.

Source code in src/demangle/core/ast.py

def __init__(self, inner):
    self.inner = inner

Pack ¶

Pack(inner)

Bases: _Unary

A parameter pack expansion.

Source code in src/demangle/core/ast.py

def __init__(self, inner):
    self.inner = inner

ParameterPack ¶

ParameterPack(members)

Bases: Node

A pack of concrete template arguments.

Source code in src/demangle/core/ast.py

def __init__(self, members):
    self.members = tuple(members)

MemberPointer ¶

MemberPointer(owner, inner)

Bases: Node

A pointer to member, Type Owner::*.

Source code in src/demangle/core/ast.py

def __init__(self, owner, inner):
    self.owner = owner
    self.inner = inner

Array ¶

Array(inner, dimension)

Bases: Node

An array type. An empty dimension means an unbounded array.

Source code in src/demangle/core/ast.py

def __init__(self, inner, dimension):
    self.inner = inner
    self.dimension = dimension

VendorQualify ¶

VendorQualify(inner, qualifier)

Bases: Node

A vendor extended qualifier, spelled after the type it applies to.

Source code in src/demangle/core/ast.py

def __init__(self, inner, qualifier):
    self.inner = inner
    self.qualifier = qualifier

Function ¶

Function(returns, parameters, suffix='', name=None)

Bases: Node

A function type, or a whole function declaration when name is present.

Source code in src/demangle/core/ast.py

def __init__(self, returns, parameters, suffix="", name=None):
    self.returns = returns
    self.parameters = tuple(parameters)
    self.suffix = suffix
    self.name = name

Decorated ¶

Decorated(inner, decoration)

Bases: Node

A symbol plus what the symbol table appended to it.

decoration keeps its separator: "@@GLIBCXX_3.4", ".cold". A tool that wants the undecorated entity walks to inner; one that wants to know which clone or which library version this is reads decoration.

Source code in src/demangle/core/ast.py

def __init__(self, inner, decoration):
    self.inner = inner
    self.decoration = decoration

Special ¶

Special(label, inner)

Bases: Node

A symbol about an entity: vtable for Foo, typeinfo for Bar.

Source code in src/demangle/core/ast.py

def __init__(self, label, inner):
    self.label = label
    self.inner = inner

AstBuilder ¶

AstBuilder(style=None)

Bases: Builder

Builds Node trees. The backend behind parse().

One instance serves every call. The only state is the leaf table below, which is a cache and does not make an answer depend on what was parsed before it.

Each method records on the node it builds an upper bound on that subtree's rendered length, which is what makes size() constant time. Over-estimating is deliberate -- the number enforces max_output, where erring high is the safe direction -- and it is carried rather than measured because the tree is a graph with shared subtrees, and walking it would be exponential in exactly the cases the bound exists to stop.

Leaves are interned; nothing else is. Measured over the 5,913 trees of the shipped libstdc++: leaves are 55% of all nodes and repeat 49 times over on average -- 15,654 builtin instances hold 32 distinct spellings, 8,807 raw instances hold 16. Interning them costs nothing because a leaf is keyed by its text, so a lookup is one string hash and no comparison of subtrees; it is 40% less memory over that corpus and, because allocating fewer objects is less work than allocating more, about 6% faster as well.

Composite nodes are not interned: a composite hashes by walking its children, so building that table would cost several times the whole parse to save memory a caller may not be short of. Node.__eq__ still compares composites by value, so a caller wanting to deduplicate a particular result can.

Source code in src/demangle/core/ast.py

def __init__(self, style=None):
    self._builtins = {}
    self._names = {}
    self._raws = {}
    # A registered name where there is one, so re-registering is picked up; else the
    # one-off style object. See `builder_for`.
    self._style = style

members ¶

members(handle)

The members of a parameter pack, or None if this is not one.

Source code in src/demangle/core/ast.py

def members(self, handle):
    """The members of a parameter pack, or None if this is not one."""
    return handle.members if type(handle) is ParameterPack else None

builder_for ¶

builder_for(style)

The tree builder for style.

A registered style is held by name, so one registered again under a name it already had is picked up rather than remembered, and the common path -- parse(name), whose style is the string "llvm" -- is a dictionary lookup.

A style object that is not the registered one under its name gets its own builder, holding the object. Keyed by name it would be wrong in both directions: a composed style like demangle.style("llvm", msvc={...}) keeps the name "llvm" and would silently be served the registered style's builder, and a style whose name is not registered at all would make parse() raise unknown style from inside the parser while demangle() accepts the very same object. Building one per call matches the bargain demangle() makes for a style object: it is not cached either.

setdefault rather than a lock: two threads arriving together each build one and the loser's is discarded, which costs an allocation and cannot produce two builders in use for one style.

Source code in src/demangle/core/ast.py

def builder_for(style):
    """The tree builder for `style`.

    A *registered* style is held by name, so one registered again under a name it already
    had is picked up rather than remembered, and the common path -- `parse(name)`, whose
    style is the string `"llvm"` -- is a dictionary lookup.

    A style *object* that is not the registered one under its name gets its own builder,
    holding the object. Keyed by name it would be wrong in both directions: a composed
    style like `demangle.style("llvm", msvc={...})` keeps the name `"llvm"` and would
    silently be served the registered style's builder, and a style whose name is not
    registered at all would make `parse()` raise `unknown style` from inside the parser
    while `demangle()` accepts the very same object. Building one per call matches the
    bargain `demangle()` makes for a style object: it is not cached either.

    `setdefault` rather than a lock: two threads arriving together each build one and
    the loser's is discarded, which costs an allocation and cannot produce two builders
    in use for one style.
    """
    if isinstance(style, Style):
        try:
            registered = get_style(style.name)
        except ValueError:
            registered = None
        if registered is not style:
            return AstBuilder(style)
        style = style.name
    found = _BUILDERS.get(style)
    if found is None:
        found = _BUILDERS.setdefault(style, AstBuilder(style))
    return found

Reading¶

demangle.core.reader ¶

A bounds-checked cursor over a mangled name.

Every parser in the package reads through this and never indexes the string directly. That is not ceremony: the schemes are full of places where the next character decides the production, and a bare text[pos] at the end of a truncated name raises IndexError -- which then has to be caught somewhere far away and translated into something meaningful. Reading past the end here is simply "no character", and the parser decides whether that is fatal.

Kept deliberately small and __slots__-ed: an instance exists per name demangled, and its methods are the innermost loop in the package.

Reader ¶

Reader(text)

A cursor over text, with the lookahead and consumption parsers need.

length is the end of input and is not always len(text). A parser may shorten it to bound a nested encoding it has found the end of by other means -- the Itanium scheme does this for ___Z<encoding>_block_invoke, where a regex says where the encoding stops and the parser must not read the literal that follows. Every method here answers against length rather than against the string, because a bound only half the class honours is worse than none: if peek reported the end of input while expect stepped past it, a truncated S would borrow the _ of _block_invoke for its terminator and read as though it were whole.

Source code in src/demangle/core/reader.py

def __init__(self, text):
    self.text = text
    self.pos = 0
    self.length = len(text)
    #: Whether a `length_prefixed()` read had a leading zero. Recorded, not refused:
    #: `c++filt` 2.42 reads `_Z1f01A` as `f(A)` and `llvm-cxxfilt` 18 refuses it.
    #: Read by tools/enumerate.py.
    self.padded_length = False

peek ¶

peek()

The character at the cursor, or "" past the end.

Returning a string rather than raising is what lets a parser write if reader.peek() == "N" without first checking for the end of input.

A parser that has just had a non-empty character out of peek, with nothing between the two, may consume it by advancing pos itself rather than calling take: the bounds test take would repeat has already happened. Nothing weaker licenses it. An intervening call may have moved the cursor, and an empty result means there was no character to consume -- either way pos += 1 would step past the end, which is the one thing this class exists to prevent. The productions that do it say so at the site.

It is called around fifty times per name demangled -- more than any other method in the package -- and CPython charges for a default it then has to bind, so it takes no argument: that is a fifth off the cost of the call. ahead is the offset form.

Source code in src/demangle/core/reader.py

def peek(self):
    """The character at the cursor, or `""` past the end.

    Returning a string rather than raising is what lets a parser write
    `if reader.peek() == "N"` without first checking for the end of input.

    A parser that has just had a *non-empty* character out of `peek`, with nothing
    between the two, may consume it by advancing `pos` itself rather than calling
    `take`: the bounds test `take` would repeat has already happened. Nothing weaker
    licenses it. An intervening call may have moved the cursor, and an empty result
    means there was no character to consume -- either way `pos += 1` would step past
    the end, which is the one thing this class exists to prevent. The productions
    that do it say so at the site.

    It is called around fifty times per name demangled -- more than any other method
    in the package -- and CPython charges for a default it then has to bind, so it
    takes no argument: that is a fifth off the cost of the call. `ahead` is the
    offset form.
    """
    pos = self.pos
    return self.text[pos] if pos < self.length else ""

ahead ¶

ahead(offset)

The character offset past the cursor, or "" past the end.

Source code in src/demangle/core/reader.py

def ahead(self, offset):
    """The character `offset` past the cursor, or `""` past the end."""
    index = self.pos + offset
    return self.text[index] if index < self.length else ""

peek2 ¶

peek2()

The next two characters, for the many two-letter codes in these grammars.

Source code in src/demangle/core/reader.py

def peek2(self):
    """The next two characters, for the many two-letter codes in these grammars."""
    pos = self.pos
    end = pos + 2
    length = self.length
    return self.text[pos : end if end < length else length]

ahead2 ¶

ahead2(offset)

The two characters offset past the cursor, bounded like peek2.

For the lookaheads that decide a branch without consuming anything -- whether an abbreviation is followed by a constructor, whether gs introduces an allocation. Reaching into text directly is the end of input's one blind spot: a bound only half the class honours is worse than none.

Source code in src/demangle/core/reader.py

def ahead2(self, offset):
    """The two characters `offset` past the cursor, bounded like `peek2`.

    For the lookaheads that decide a branch without consuming anything -- whether an
    abbreviation is followed by a constructor, whether `gs` introduces an allocation.
    Reaching into `text` directly is the end of input's one blind spot: a bound only
    half the class honours is worse than none.
    """
    start = self.pos + offset
    end = start + 2
    length = self.length
    return self.text[start : end if end < length else length]

take ¶

take()

Consume and return one character. Raises at the end of input.

Source code in src/demangle/core/reader.py

def take(self):
    """Consume and return one character. Raises at the end of input."""
    if self.pos >= self.length:
        raise TruncatedError(self.text, self.pos)
    char = self.text[self.pos]
    self.pos += 1
    return char

take_exactly ¶

take_exactly(count)

Consume exactly count characters, or raise if they are not all there.

Source code in src/demangle/core/reader.py

def take_exactly(self, count):
    """Consume exactly `count` characters, or raise if they are not all there."""
    end = self.pos + count
    if count < 0 or end > self.length:
        raise TruncatedError(self.text, self.pos)
    chunk = self.text[self.pos : end]
    self.pos = end
    return chunk

eat ¶

eat(literal)

Consume literal if it is next, reporting whether it was.

Source code in src/demangle/core/reader.py

def eat(self, literal):
    """Consume `literal` if it is next, reporting whether it was."""
    pos = self.pos
    if len(literal) == 1:
        if pos < self.length and self.text[pos] == literal:
            self.pos = pos + 1
            return True
        return False
    end = pos + len(literal)
    if end <= self.length and self.text.startswith(literal, pos):
        self.pos = end
        return True
    return False

expect ¶

expect(literal)

Consume literal, or raise.

Source code in src/demangle/core/reader.py

def expect(self, literal):
    """Consume `literal`, or raise."""
    # `eat` inlined: this is on the path of every fixed-opening production.
    pos = self.pos
    if len(literal) == 1:
        if pos < self.length and self.text[pos] == literal:
            self.pos = pos + 1
            return
        raise ParseError(self.text, pos, f"expected {literal!r}")
    end = pos + len(literal)
    if end <= self.length and self.text.startswith(literal, pos):
        self.pos = end
        return
    raise ParseError(self.text, pos, f"expected {literal!r}")

digits ¶

digits()

Consume a run of decimal digits and return it, or raise if there are none.

Bounded: an unbounded run would reach int(), which CPython refuses above 4300 digits with a ValueError -- not this package's error type, and so not something a caller of parse() can be expected to catch.

Source code in src/demangle/core/reader.py

def digits(self):
    """Consume a run of decimal digits and return it, or raise if there are none.

    Bounded: an unbounded run would reach `int()`, which CPython refuses above 4300
    digits with a `ValueError` -- not this package's error type, and so not something
    a caller of `parse()` can be expected to catch.
    """
    start = self.pos
    text, length = self.text, self.length
    pos = start
    while pos < length and text[pos] in DIGITS:
        pos += 1
        if pos - start > MAX_NUMBER_DIGITS:
            raise ParseError(text, start, "number too long")
    if pos == start:
        raise ParseError(text, start, "expected a number")
    self.pos = pos
    return text[start:pos]

length_prefixed ¶

length_prefixed()

A decimal length and the characters it counts, as one production.

<source-name> is the commonest production in these grammars -- every component of every qualified name is one -- and reading it as int(digits()) followed by take_exactly() costs three frames for what is one scan and one slice. Raises the same way each of those does: no digits, a run past the bound, or a length the name is too short for.

Source code in src/demangle/core/reader.py

def length_prefixed(self):
    """A decimal length and the characters it counts, as one production.

    `<source-name>` is the commonest production in these grammars -- every component
    of every qualified name is one -- and reading it as `int(digits())` followed by
    `take_exactly()` costs three frames for what is one scan and one slice. Raises
    the same way each of those does: no digits, a run past the bound, or a length
    the name is too short for.
    """
    text, length = self.text, self.length
    start = pos = self.pos
    while pos < length and text[pos] in DIGITS:
        pos += 1
        if pos - start > MAX_NUMBER_DIGITS:
            raise ParseError(text, start, "number too long")
    if pos == start:
        raise ParseError(text, start, "expected a number")
    if text[start] == "0":
        self.padded_length = True
    count = int(text[start:pos])
    end = pos + count
    if end > length:
        raise TruncatedError(text, pos)
    self.pos = end
    return count, text[pos:end]

number ¶

number(allow_negative=True)

A : an optional leading n meaning negative, then digits.

Source code in src/demangle/core/reader.py

def number(self, allow_negative=True):
    """A <number>: an optional leading `n` meaning negative, then digits."""
    negative = allow_negative and self.eat("n")
    digits = self.digits()
    return "-" + digits if negative else digits

seq_id ¶

seq_id()

A followed by _, returned as a zero-based index.

The encoding is offset by one so that the empty sequence can mean something: S_ is entry 0 and S0_ is entry 1.

Source code in src/demangle/core/reader.py

def seq_id(self):
    """A <seq-id> followed by `_`, returned as a zero-based index.

    The encoding is offset by one so that the empty sequence can mean something:
    `S_` is entry 0 and `S0_` is entry 1.
    """
    start = self.pos
    text, length = self.text, self.length
    pos = start
    while pos < length and text[pos] in _SEQ_ID_VALUES:
        pos += 1
        if pos - start > MAX_SEQ_ID_DIGITS:
            raise ParseError(text, start, "substitution index too long")
    # `expect("_")`, inlined.
    if pos >= length or text[pos] != "_":
        self.pos = pos
        raise ParseError(text, pos, "expected '_'")
    self.pos = pos + 1
    if pos == start:
        return 0
    raw = text[start:pos]
    value = 0
    for char in raw:
        value = value * 36 + _SEQ_ID_VALUES[char]
    return value + 1

Plugins¶

A distribution can add a scheme of its own by calling register or by advertising a demangle.languages entry point. Advertised plugins load only when a program calls demangle.load_plugins() (the demangle command does); register always takes effect. Adding a scheme is the walk-through.

demangle.core.plugin ¶

The contract a mangling scheme implements.

A language is a LanguagePlugin. Nothing in core imports a language module at import time, and no language imports another, so a scheme can be developed, tested, replaced or shipped separately from everything else here.

Adding one means providing three things and registering them. It does not mean understanding the rest of the codebase, which is the point: the project should be approachable to a contributor who knows one ABI well and nothing else about it.

LanguagePlugin `dataclass` ¶

LanguagePlugin(
    name,
    detect,
    parse,
    parse_type=None,
    node_kinds=(),
    description="",
    aliases=tuple(),
    options_type=None,
    symbol_table_decorations=False,
    first_characters="",
    priority=100,
)

One mangling scheme.

name `instance-attribute` ¶

name

Stable identifier, used in the API and on the command line: itanium, msvc.

detect `instance-attribute` ¶

detect

Cheap test for "is this plausibly mine".

Called on every symbol a caller offers, including the great majority that are not mangled at all, so it must not allocate or parse. A prefix comparison is the expected shape. False negatives lose symbols; false positives only cost a failed parse, so err towards accepting.

The exception is a scheme whose names carry no marker, gnuv2, codewarrior and ada among them. Its detect screens with a substring test and then parses the whole name, and it must not err towards accepting: it is ordered after the marked schemes, and what justifies it is a measured false-positive count of zero over the other schemes' corpora and over real symbol tables.

parse `instance-attribute` ¶

parse

parse(mangled, builder, limits=..., options=...) -> handle.

Raises NotMangledError or ParseError on failure; never returns a partial result.

parse_type `class-attribute` `instance-attribute` ¶

parse_type = None

parse_type(mangled, builder, limits=..., options=...) -> handle, or None.

A type encoding rather than a whole symbol -- Pi for int*, an MSVC .PEAX, a Swift type string. This is what typeinfo names, RTTI descriptors and vtable entries carry, and it is a separate entry point because a type encoding carries no marker saying which scheme it belongs to: i is a valid Itanium type and a valid anything-else, so it can only be read when a caller says which scheme to read it as.

None for a scheme with no type grammar of its own, which is most of them: Go, Nim, Free Pascal and JNI names carry types only as text they already spell.

node_kinds `class-attribute` `instance-attribute` ¶

node_kinds = ()

Every kind string a tree from this scheme can hold.

The published vocabulary. parse() returns a walkable tree, and without this a caller building on it has to read the source to learn what to switch on -- which is what swift-demangle's kind= dumps and libiberty's hundred DEMANGLE_COMPONENT_* enumerators exist to avoid. demangle.node_kinds() is the whole of it, and tests/test_api_surface.py::TestThePublishedVocabulary checks it against every corpus so it cannot drift.

description `class-attribute` `instance-attribute` ¶

description = ''

One line, shown by demangle --list-languages.

aliases `class-attribute` `instance-attribute` ¶

aliases = field(default_factory=tuple)

Other names callers may use: gnu and gcc both mean itanium.

options_type `class-attribute` `instance-attribute` ¶

options_type = None

The dataclass this language accepts as options, if it takes any.

symbol_table_decorations `class-attribute` `instance-attribute` ¶

symbol_table_decorations = False

Whether names in this scheme may carry symbol-table decorations.

True for schemes that appear in ELF and Mach-O symbol tables, where the linker appends version suffixes and the compiler appends clone suffixes. It must stay False for MSVC, whose decorated names use @ as their own scope separator and would be truncated at the first one.

first_characters `class-attribute` `instance-attribute` ¶

first_characters = ''

The characters a name of this scheme may begin with, if the set is small.

An optimisation, and one the registry checks rather than trusts: a name whose first character is not in here is never offered to detect. Leave it empty -- the default -- for a scheme whose names have no fixed start, which costs nothing but a call.

Getting it wrong loses symbols silently, so the rule is narrow: put a character here only if detect returns False for every name that does not begin with one of them. tests/test_core.py checks the built-in schemes against their own corpora.

priority `class-attribute` `instance-attribute` ¶

priority = 100

Detection order, lower first.

Matters only where two schemes share a prefix. Rust's legacy scheme is Itanium mangling, so Rust must be offered the name first or every Rust symbol demangles as a C++ one.

demangle.core.registry ¶

Discovery of mangling schemes.

Built-in languages are imported lazily, and not only when named: the registry knows each one's detection order and first characters without importing it, so a scheme's module is imported only once a name reaches its detect -- a name starting ? never costs the Swift parser. Third-party languages are found through the demangle.languages entry-point group, so a separate distribution can add a language without a patch here -- but only once a caller asks, with load_plugins(). Nothing is discovered on its own: an installed distribution cannot replace a built-in scheme unnoticed, and two installs read a name alike.

The registry is process-global and populated once. Registration is idempotent, so importing a plugin module twice is harmless.

register ¶

register(plugin)

Add a plugin to the registry, replacing any earlier one of the same name.

Source code in src/demangle/core/registry.py

def register(plugin):
    """Add a plugin to the registry, replacing any earlier one of the same name."""
    global _by_first, _ordered
    if not isinstance(plugin, LanguagePlugin):
        raise TypeError(f"expected a LanguagePlugin, got {type(plugin).__name__}")
    with _lock:
        builtin = _registers_itself(plugin)
        if builtin:
            # A built-in's module registering its own `PLUGIN` as it is imported, which
            # can happen at any time -- a style's options import it mid-parse -- so it
            # never displaces a plugin a caller registered under its name, whether that
            # came before or after the registry loaded.
            current = _plugins.get(plugin.name)
            if current is not None:
                if current is _stand_ins.get(plugin.name):
                    # The languages on offer are unchanged, so no callback.
                    _plugins[plugin.name] = plugin
                    _ordered = None
                    _by_first = None
                return plugin
        else:
            # Check every alias before recording any, so a rejected plugin leaves no
            # trace. A built-in's name counts before the registry has loaded it too.
            for alias in plugin.aliases:
                if (alias in _plugins or alias in _BUILTIN_NAMES) and alias != plugin.name:
                    raise ValueError(f"alias {alias!r} collides with a registered language name")
        _plugins[plugin.name] = plugin
        if builtin:
            _offer_builtin_aliases(plugin.name, plugin.aliases)
        else:
            for alias in plugin.aliases:
                _aliases[alias] = plugin.name
        # A re-registered name wins back over any alias that had shadowed it.
        _aliases.pop(plugin.name, None)
        _ordered = None
        _by_first = None
    for callback in _on_change:
        callback()
    return plugin

get ¶

get(name)

Look up a plugin by name or alias. Raises KeyError if unknown.

Source code in src/demangle/core/registry.py

def get(name):
    """Look up a plugin by name or alias. Raises KeyError if unknown."""
    _load()
    try:
        with _lock:
            plugin = _plugins[name] if name in _plugins else _plugins[_aliases[name]]
    except TypeError:
        raise KeyError(name) from None
    if plugin is _stand_ins.get(plugin.name):
        return _materialise(plugin.name, _module_of(plugin.name))
    return plugin

available ¶

available()

Every registered plugin, in detection order.

Source code in src/demangle/core/registry.py

def available():
    """Every registered plugin, in detection order."""
    _load()
    with _lock:
        waiting = [name for name, stand_in in _stand_ins.items() if _plugins.get(name) is stand_in]
    for name in waiting:
        _materialise(name, _module_of(name))
    return _order()

names ¶

names()

Registered language names, sorted.

Source code in src/demangle/core/registry.py

def names():
    """Registered language names, sorted."""
    _load()
    with _lock:
        return sorted(_plugins)

aliases ¶

aliases()

Every alias, mapped to the plugin name it resolves to.

Source code in src/demangle/core/registry.py

def aliases():
    """Every alias, mapped to the plugin name it resolves to."""
    _load()
    with _lock:
        return dict(_aliases)

Symbol-table decorations¶

demangle.core.decorations ¶

Decorations a symbol table adds around a mangled name.

A symbol in a real binary is often not just a mangled name. The linker and the compiler both append things to it, and none of those additions belong to any mangling scheme -- they are conventions of the object format and the toolchain:

_ZNSt10moneypunctIcLb0EE2idE@@GLIBCXX_3.4     ELF symbol versioning
_ZN3Foo3barEv.cold                             a cold-path clone
_ZN3Foo3barEv.part.0                           a partial inlining clone
_ZN3Foo3barEv.llvm.1234567890                  an LLVM internalisation suffix
_ZN3Foo3barEv.constprop.0                      a constant-propagation clone

Handling these in core rather than in each parser matters because they are orthogonal to the grammar: a scheme should not have to know about ELF, and a new scheme gets this for free. The reference demanglers pass them through verbatim, and so do we -- the suffix identifies which copy of a function a symbol is, which is information a caller

split_decorations ¶

split_decorations(name)

Split name into (mangled, version-suffix).

Only the ELF version suffix is split here, because only it can be recognised before parsing: @ appears in no scheme this applies to, so the first one is necessarily the start of the suffix.

Clone suffixes deliberately are not. A . is not reserved: GCC names a coroutine's frame type _ZN6modern9coroutineEi.Frame and writes that as a length-prefixed identifier, so the coroutine's actor is _ZN6modern9coroutineEPZNS_9coroutineEiE28_ZN6modern9coroutineEi.Frame.actor -- where the first . is inside the identifier and only the second begins the clone suffix. Cutting at the first . truncates the name mid-production. A clone suffix can only be recognised as what is left over once the grammar has consumed everything it can, which means the parser has to do it.

The suffix is returned with its separator intact, so joining is concatenation.

Source code in src/demangle/core/decorations.py

def split_decorations(name):
    """Split `name` into (mangled, version-suffix).

    Only the ELF version suffix is split here, because only it can be recognised
    *before* parsing: `@` appears in no scheme this applies to, so the first one is
    necessarily the start of the suffix.

    Clone suffixes deliberately are not. A `.` is not reserved: GCC names a coroutine's
    frame type `_ZN6modern9coroutineEi.Frame` and writes that as a length-prefixed
    identifier, so the coroutine's actor is
    `_ZN6modern9coroutineEPZNS_9coroutineEiE28_ZN6modern9coroutineEi.Frame.actor` --
    where the first `.` is inside the identifier and only the second begins the clone
    suffix. Cutting at the first `.` truncates the name mid-production. A clone suffix
    can only be recognised as what is *left over* once the grammar has consumed
    everything it can, which means the parser has to do it.

    The suffix is returned with its separator intact, so joining is concatenation.
    """
    version = name.find(VERSION_SEPARATOR)
    if version <= 0:
        return name, ""
    return name[:version], name[version:]

describe ¶

describe(decoration, gnu_clone_suffix=False)

A readable spelling of a decoration.

ELF version suffixes are passed through as written by both references. Clone suffixes are not: llvm-cxxfilt writes Foo::bar() (.cold) where GNU c++filt writes Foo::bar() [clone .cold]. Neither is more correct, so it is a style choice.

Source code in src/demangle/core/decorations.py

def describe(decoration, gnu_clone_suffix=False):
    """A readable spelling of a decoration.

    ELF version suffixes are passed through as written by both references. Clone
    suffixes are not: llvm-cxxfilt writes `Foo::bar() (.cold)` where GNU c++filt writes
    `Foo::bar() [clone .cold]`. Neither is more correct, so it is a style choice.
    """
    if not decoration.startswith(CLONE_SEPARATOR):
        return decoration
    if not gnu_clone_suffix:
        return f" ({decoration})"
    # GNU spells each clone separately -- `.actor.cold` is two clones, not one -- while
    # a trailing number belongs to the clone before it, so `.part.0` stays whole.
    spelled = []
    for component in decoration.split(CLONE_SEPARATOR)[1:]:
        if component.isdigit() and spelled:
            spelled[-1] += f".{component}"
        else:
            spelled.append(f".{component}")
    return "".join(f" [clone {clone}]" for clone in spelled)

Styles¶

demangle.core.style ¶

Output styles.

Some spelling questions have more than one correct answer, and the reference implementations answer them differently: llvm-cxxfilt prints std::string and Foo<Bar<int>>, GNU c++filt prints the full basic_string specialisation and Foo<Bar<int> >. Both describe the same types.

A Style bundles those choices so a caller picks one name -- "llvm", "gnu" -- instead of threading half a dozen flags through every call. New languages add their own options to the mapping without changing this type, which is why the per-language policy is an opaque dict rather than a fixed set of fields.

Not every choice is a disagreement between references, though, and the second kind is what with_options is for: how much of a name to print. Every peer tool lets a caller compose those at the call site -- llvm-undname --no-calling-convention, UnDecorateSymbolName's mask, c++filt -p -- because "the same names, with less around them" is what makes a symbol table greppable. A style is immutable, so composing one returns a new style rather than changing the shared default, and a style object handed to demangle() is deliberately not cached: it is one call's policy, not the process's.

Style `dataclass` ¶

Style(name, spelling_builder, language_options=dict())

A named set of output policy choices.

name `instance-attribute` ¶

name

What the style is registered and looked up under -- "llvm", "gnu".

spelling_builder `instance-attribute` ¶

spelling_builder

The builder used when a caller asks for text. Must be a Builder.

language_options `class-attribute` `instance-attribute` ¶

language_options = field(default_factory=dict)

Per-language option objects, keyed by language name.

options_for ¶

options_for(language)

The option object this style specifies for language, or None.

Source code in src/demangle/core/style.py

def options_for(self, language):
    """The option object this style specifies for `language`, or None."""
    return self.language_options.get(language)

with_options ¶

with_options(**languages)

This style with some language's options changed, as a new style.

Each keyword is a language name and each value is either that language's whole options object or a mapping of the fields to change in the one already in force:

>>> import demangle
>>> from demangle.core.style import get_style
>>> narrow = get_style("llvm").with_options(msvc={"calling_convention": False})
>>> demangle.demangle("?f@@YAXH@Z", style=narrow)
'void f(int)'

demangle.style() is the same thing spelled from the package.

The result is a Style object rather than a registered name, which is what keeps it a per-call policy: demangle() does not cache a call that passes one, so one caller's narrower spelling cannot be served to another asking for the default.

Source code in src/demangle/core/style.py

def with_options(self, **languages) -> "Style":
    """This style with some language's options changed, as a new style.

    Each keyword is a language name and each value is either that language's whole
    options object or a mapping of the fields to change in the one already in force:

        >>> import demangle
        >>> from demangle.core.style import get_style
        >>> narrow = get_style("llvm").with_options(msvc={"calling_convention": False})
        >>> demangle.demangle("?f@@YAXH@Z", style=narrow)
        'void f(int)'

    `demangle.style()` is the same thing spelled from the package.

    The result is a `Style` object rather than a registered name, which is what keeps
    it a *per-call* policy: `demangle()` does not cache a call that passes one, so one
    caller's narrower spelling cannot be served to another asking for the default.
    """
    options = self.language_options
    changed = {}
    for language, value in languages.items():
        if isinstance(value, Mapping):
            current = options.get(language)
            if current is None:
                raise ValueError(
                    f"style {self.name!r} carries no options for {language!r} to change; "
                    f"pass that language's options object instead of a mapping"
                )
            value = replace(current, **value)
        elif language not in options:
            # The object form adds rather than changes, so a typo must be caught
            # here. Imported lazily: no import-time dependency on any scheme.
            from .registry import names as _known_languages

            known = _known_languages()
            if language not in known:
                raise ValueError(f"unknown language {language!r}; known languages are {known}")
        changed[language] = value
    if isinstance(options, _LazyOptions):
        # Only the languages changed are imported, not every one the style names.
        return replace(self, language_options=options.with_values(changed))
    return replace(self, language_options={**options, **changed})

register_style ¶

register_style(style)

Add a style, so a downstream project can define its own house spelling.

Source code in src/demangle/core/style.py

def register_style(style):
    """Add a style, so a downstream project can define its own house spelling."""
    table = _table()
    with _lock:
        table[style.name] = style
    for callback in _on_change:
        callback()
    return style

get_style ¶

get_style(name)

Look up a style by name. None gives the default, and a Style is returned as is.

Raises:

Type	Description
`ValueError`	`name` is not a registered style.

Source code in src/demangle/core/style.py

def get_style(name):
    """Look up a style by name. `None` gives the default, and a `Style` is returned as is.

    Raises:
        ValueError: `name` is not a registered style.
    """
    # `_table()`'s fast path inlined: every call into the package resolves a style.
    styles = _STYLES
    if styles is None:
        styles = _table()
    if name is None:
        return styles["llvm"]
    if isinstance(name, Style):
        return name
    try:
        return styles[name]
    except (KeyError, TypeError):
        raise ValueError(f"unknown style {name!r}; known styles are {sorted(styles)}") from None

available_styles ¶

available_styles()

The registered style names, sorted. demangle.styles() is this.

Source code in src/demangle/core/style.py

def available_styles():
    """The registered style names, sorted. `demangle.styles()` is this."""
    return sorted(_table())

Caching¶

demangle.core.cache ¶

Bounded memoisation.

Symbol tables repeat themselves relentlessly: one C++ binary can name std::allocator<char> thousands of times, and every one costs the same parse. Caching is worth more here than almost any micro-optimisation.

It also cannot be unbounded. A tool that walks a corpus of binaries would otherwise accumulate an entry per distinct symbol ever seen, and there is no natural end to that.

BoundedCache ¶

BoundedCache(max_size=8192, max_weight=None, weigh=None)

A dict of at most max_size entries, kept as two generations.

New entries go into the young generation. When it holds half of max_size, it becomes the old one -- the previous old one is dropped whole -- and a new, empty young generation starts. A lookup tries the young generation, then the old; an entry found in the old one moves back to the young, so whatever is still in use survives every turnover while what went cold ages out within two.

A hit in the young generation costs one lookup, and a hit in the old one a second lookup and a move. True LRU would cost a linked-list update on every hit, the operation being optimised, and served a warm name 15% slower for a few points of hit rate.

Adding an entry -- a miss, or a promotion from the old generation -- takes a lock, so under free threading the turnover and the weight are exact and the bound holds. A hit in the young generation takes none, being the path this exists to make cheap. The lock is re-entrant: a signal handler or finalizer that runs demangle() while this thread is inside put proceeds rather than deadlocks, at worst leaving the weight one entry off.

epoch counts clear() calls. A caller that computes a value from state whose change clears the cache reads it before computing and passes it to put, which then drops the value if a clear came in between, rather than keep an answer from before.

hits and misses count lookups: a hit in either generation is a hit. They are not guarded, so under free threading concurrent lookups can lose an increment and the figures are approximate.

weigh(key, value), with max_weight, also bounds what the entries hold.

Source code in src/demangle/core/cache.py

def __init__(self, max_size=8192, max_weight=None, weigh=None):
    """`weigh(key, value)`, with `max_weight`, also bounds what the entries hold."""
    self.max_size = max_size
    self.max_weight = max_weight
    self._weigh = weigh
    #: Entries per generation; at least one, so a `max_size` under 2 still caches (two).
    self._generation = max(max_size // 2, 1)
    #: Weight per generation: a generation turns over once it reaches this, so the
    #: whole holds at most `max_weight` and two entries.
    self._budget = float("inf") if max_weight is None or weigh is None else max(max_weight // 2, 1)
    self._young = {}
    self._old = {}
    self._young_weight = 0
    self._young_weights = {}
    self._lock = threading.RLock()
    self.epoch = 0
    self.hits = 0
    self.misses = 0

get_old ¶

get_old(key)

Finish a lookup whose young generation already missed.

Source code in src/demangle/core/cache.py

def get_old(self, key):
    """Finish a lookup whose young generation already missed."""
    # Read first, so an entry taken from before a `clear()` is not put back.
    epoch = self.epoch
    value = self._old.pop(key, _MISSING)
    if value is _MISSING:
        self.misses += 1
        return _MISSING
    self.put(key, value, epoch)
    self.hits += 1
    return value

put ¶

put(key, value, epoch=None)

Store value and return it; not stored if epoch is given and out of date.

Source code in src/demangle/core/cache.py

def put(self, key, value, epoch=None):
    """Store `value` and return it; not stored if `epoch` is given and out of date."""
    # `acquire` and `release`: `with` costs twice as much, and this runs on every miss.
    # Weighed before the lock is taken, so as little Python as possible runs under it.
    weight = 0 if self._weigh is None else self._weigh(key, value)
    lock = self._lock
    lock.acquire()
    try:
        if epoch is not None and epoch != self.epoch:
            return value
        young = self._young
        replacing = key in young
        if not replacing and (len(young) >= self._generation or self._young_weight >= self._budget):
            self._old = young
            self._young = young = {}
            self._young_weight = 0
            self._young_weights = {}
        young[key] = value
        if self._weigh is not None:
            previous_weight = self._young_weights.get(key, 0)
            self._young_weights[key] = weight
            self._young_weight += weight - previous_weight
    finally:
        lock.release()
    return value

Core¶

The builder protocol¶

demangle.core.builder ¶

Handle module-attribute ¶

Builder ¶

builtin ¶

name ¶

raw ¶

literal ¶

expression ¶

qualified ¶

template ¶

qualify ¶

pointer ¶

reference ¶

rvalue_reference ¶

member_pointer ¶

array ¶

function ¶

pack ¶

parameter_pack ¶

vendor_qualify ¶

special ¶

decorated ¶

spell ¶

size ¶

Spelling¶

demangle.core.spelling ¶

Spelling ¶

spell ¶

SpellingBuilder ¶

qualify ¶

pack ¶

vendor_qualify ¶

members ¶

pack_of ¶

The tree¶

demangle.core.ast ¶

AST_BUILDER module-attribute ¶

Node ¶

children ¶

to_dict ¶

build ¶

walk ¶

find ¶

render ¶

spell ¶

Builtin ¶

Name ¶

Raw ¶

Literal ¶

Expression ¶

operands property ¶

Qualified ¶

Template ¶

Qualify ¶

Pointer ¶

Reference ¶

RValueReference ¶

Pack ¶

ParameterPack ¶

MemberPointer ¶

Array ¶

VendorQualify ¶

Function ¶

Decorated ¶

Special ¶

AstBuilder ¶

members ¶

builder_for ¶

Reading¶

demangle.core.reader ¶

Reader ¶

peek ¶

ahead ¶

peek2 ¶

ahead2 ¶

take ¶

take_exactly ¶

eat ¶

Handle `module-attribute` ¶

AST_BUILDER `module-attribute` ¶

operands `property` ¶

LanguagePlugin `dataclass` ¶

name `instance-attribute` ¶

detect `instance-attribute` ¶

parse `instance-attribute` ¶

parse_type `class-attribute` `instance-attribute` ¶

node_kinds `class-attribute` `instance-attribute` ¶

description `class-attribute` `instance-attribute` ¶

aliases `class-attribute` `instance-attribute` ¶

options_type `class-attribute` `instance-attribute` ¶

symbol_table_decorations `class-attribute` `instance-attribute` ¶

first_characters `class-attribute` `instance-attribute` ¶

priority `class-attribute` `instance-attribute` ¶

Style `dataclass` ¶

name `instance-attribute` ¶

spelling_builder `instance-attribute` ¶

language_options `class-attribute` `instance-attribute` ¶