[Question] Is there any way to prevent pdoc from documenting method objects?

[lhriley]

Problem Description

A clear and concise description of what the bug is.

In my application, I use typer as a CLI framework (https://typer.tiangolo.com/). Part of the framework uses decorators with annotations to manage flags, help text, etc.

In the example below you can see that the @callback decorator uses a class CommonOptions which provide a reusable set of options. When I run pdoc the typer.Option() method (typer.models.OptionInfo) is found to be an in-memory object, and as a result the id value changes on every run of pdoc.

This is problematic as the documentation related to the files which use this decorator are never "correct" (the id values are always different). I would like to use pdoc in a pre-commit hook, but this behavior makes that infeasible.

My question is: Is there any way to prevent this from happening?

There are a few possible "solutions" that would be acceptable if possible, for example:

1. Is my code written "incorrectly" with relation to how pdoc works that isn't obvious to me?
2. Is there a flag that I can add to pdoc to write a static reference to the in-memory object? (I don't believe so)
3. Is there a decorator that will skip "following" the object, and just leave a static reference to the method? I have tried @private in the docstring, however I would really prefer to have the methods documented. Also, since typer uses the docstring for help text, this will show up in the help text and cause confusion.
4. Is there a way to tell pdoc to ignore / exclude an entire directory tree from documentation? So I don't have to add @private to all of the docstrings individually.
5. Is there some change that could implement one or more of the above solutions within pdoc?

Including example code below in case there's something I can do to fix my code. I can try to produce a simpler example for reproducibility, if requested.

code:

@callback(app, CommonOptions)
def callback(ctx: typer.Context):
    """
    Callback help text
    """

    if ctx.invoked_subcommand is None:
        typer.help(ctx)

documentation:

@callback(app, CommonOptions)
def callback(
	ctx: typer.models.Context,
	*,
	log_level: Annotated[Optional[str], <typer.models.OptionInfo object at 0x7ff4362c79b0>] = None,
	version: Annotated[Optional[bool], <typer.models.OptionInfo object at 0x7ff4362c4a10>] = False
):

Steps to reproduce the behavior:

1. See example code below.

System Information

Paste the output of "pdoc --version" here.

❯ pdoc --version
pdoc: 14.3.0
Python: 3.12.1
Platform: Linux-6.2.0-1017-lowlatency-x86_64-with-glibc2.35

Example source

CommonOptions source:

"""
source: https://github.com/tiangolo/typer/issues/153#issuecomment-1771421969
"""
from dataclasses import dataclass, field
from typing import Annotated, Optional

import click
import typer

from lbctl.utils.helpers.version import Version, VersionCheckErrors


@dataclass
class CommonOptions:
    """
    Dataclass defining CLI options used by all commands.

    @private - hide from pdoc output due to some dynamic objects
    """

    instance = None

    ATTRNAME: str = field(default="common_params", metadata={"ignore": True})

    def __post_init__(self):
        CommonOptions.instance = self

    @classmethod
    def from_context(cls, ctx: typer.Context) -> "CommonOptions":
        if (common_params_dict := getattr(ctx, "common_params", None)) is None:
            raise ValueError("Context missing common_params")

        return cls(**common_params_dict)

    def callback_log_level(cls, ctx: typer.Context, value: str):
        """Callback for log level."""
        if value:
            from lbctl.utils.config import config

            config.configure_logger(console_log_level=value)

    def callback_version(cls, ctx: typer.Context, value: bool):
        """Callback for version."""
        if value:
            try:
                ver = Version()
                ver.version(show_check=True, suggest_update=True)
            except (KeyboardInterrupt, click.exceptions.Abort):
                raise VersionCheckErrors.Aborted

            raise VersionCheckErrors.Checked

    log_level: Annotated[
        Optional[str],
        typer.Option(
            "--log-level",
            "-L",
            help="Set log level for current command",
            callback=callback_log_level,
        ),
    ] = None

    version: Annotated[
        Optional[bool],
        typer.Option("--version", "-V", help="Show version and exit", callback=callback_version),
    ] = False

decorator source

"""
source: https://github.com/tiangolo/typer/issues/153#issuecomment-1771421969
"""

from dataclasses import fields
from functools import wraps
from inspect import Parameter, signature
from typing import TypeVar

import typer

from lbctl.common.options import CommonOptions

OptionsType = TypeVar("OptionsType", bound="CommonOptions")


def callback(typer_app: typer.Typer, options_type: OptionsType, *args, **kwargs):
    def decorator(__f):
        @wraps(__f)
        def wrapper(*__args, **__kwargs):
            if len(__args) > 0:
                raise RuntimeError("Positional arguments are not supported")

            __kwargs = _patch_wrapper_kwargs(options_type, **__kwargs)

            return __f(*__args, **__kwargs)

        _patch_command_sig(wrapper, options_type)

        return typer_app.callback(*args, **kwargs)(wrapper)

    return decorator


def command(typer_app, options_type, *args, **kwargs):
    def decorator(__f):
        @wraps(__f)
        def wrapper(*__args, **__kwargs):
            if len(__args) > 0:
                raise RuntimeError("Positional arguments are not supported")

            __kwargs = _patch_wrapper_kwargs(options_type, **__kwargs)

            return __f(*__args, **__kwargs)

        _patch_command_sig(wrapper, options_type)

        return typer_app.command(*args, **kwargs)(wrapper)

    return decorator


def _patch_wrapper_kwargs(options_type, **kwargs):
    if (ctx := kwargs.get("ctx")) is None:
        raise RuntimeError("Context should be provided")

    common_opts_params: dict = {}

    if options_type.instance is not None:
        common_opts_params.update(options_type.instance.__dict__)

    for field in fields(options_type):
        if field.metadata.get("ignore", False):
            continue

        value = kwargs.pop(field.name)

        if value == field.default:
            continue

        common_opts_params[field.name] = value

    options_type(**common_opts_params)
    setattr(ctx, options_type.ATTRNAME, common_opts_params)

    return {"ctx": ctx, **kwargs}


def _patch_command_sig(__w, options_type) -> None:
    sig = signature(__w)
    new_parameters = sig.parameters.copy()

    options_type_fields = fields(options_type)

    for field in options_type_fields:
        if field.metadata.get("ignore", False):
            continue
        new_parameters[field.name] = Parameter(
            name=field.name,
            kind=Parameter.KEYWORD_ONLY,
            default=field.default,
            annotation=field.type,
        )
    for kwarg in sig.parameters.values():
        if kwarg.kind == Parameter.KEYWORD_ONLY and kwarg.name != "ctx":
            if kwarg.name not in new_parameters:
                new_parameters[kwarg.name] = kwarg.replace(default=kwarg.default)

    new_sig = sig.replace(parameters=tuple(new_parameters.values()))
    setattr(__w, "__signature__", new_sig)

[mhils]

• Is my code written "incorrectly" with relation to how pdoc works that isn't obvious to me?
• Is there a flag that I can add to pdoc to write a static reference to the in-memory object? (I don't believe so)

Not really, that all looks reasonable. We normally try to detect memory addresses and remove them from output, not sure why this is not happening here:

pdoc/pdoc/doc.py

Line 1183 in 3e93213

formatted = re.sub(r" at 0x[0-9a-fA-F]+(?=>$)", "", str(param))

Could you please try to remove the dollar sign from the regex and see if that fixes things?

• Is there a way to tell pdoc to ignore / exclude an entire directory tree from documentation? So I don't have to add @private to all of the docstrings individually.

Yes, see https://pdoc.dev/docs/pdoc.html#exclude-submodules-from-being-documented

[lhriley]

Okay, so removing the $ as suggested seems to have resolved the id being included. In fact removing the entire (?=>$) lookahead seems to have no obvious negative effect. I wrote a bit of perl (sed should also work) to workaround this in the meantime:

#!/usr/bin/env perl

use strict;
use warnings;
use Tie::File;

@ARGV == 1
    or usage("Incorrect number of arguments\n");

tie my @file, "Tie::File", $ARGV[0] or die $!;

for (@file) {
    $_ =~ s/(<span[^<]+object<\/span>)\s*<span[^<]+at<\/span>\s*<span[^<]+0x[a-f0-9]+<\/span>/$1/gx;
}

Which runs like this:

find docs/ -type f -exec "${SCRIPT_DIR}/documentation-cleanup.pl" {} \;

[lhriley]

Thanks for the quick turnaround on the fix @mhils . Any idea on when you plan to make the next release?

[mhils]

Happening now! :)

[lhriley]

Thanks @mhils ! 🏅