Project architecture¤

This document describes how the project is architectured, both regarding boilerplate and actual code. We start by giving an overview of the project's contents:

📁 .git/
📁 .github/ # (1)!
📁 .venv/ # (2)!
📁 .venvs/ # (3)!
📁 .vscode/ # (4)!
📁 config/ # (5)!
📁 docs/ # (6)!
📁 htmlcov/ # (7)!
📁 packages/ # (8)!
📁 scripts/ # (9)!
📁 site/ # (10)!
📁 tests/ # (11)!
 .copier-answers.yml # (12)!
 .envrc # (13)!
 .gitignore
 CHANGELOG.md
 CODE_OF_CONDUCT.md
 CONTRIBUTING.md
 LICENSE
 Makefile # (14)!
 README.md
 duties.py # (15)!
 logo.svg
 mkdocs.yml # (16)!
 pyproject.toml # (17)!
 uv.lock

1. GitHub workflows, issue templates and other configuration.

   📁 ISSUE_TEMPLATE/ # (1)!
   📁 workflows/ # (2)!
    FUNDING.yml
    pull_request_template.md
   

   1.  1-bug.md
       2-feature.md
       3-docs.md
       4-change.md
       config.yml
      

   2.  ci.yml
       release.yml
       sponsors.yml
      

2. The default virtual environment (git-ignored). See make setup command.

3. The virtual environments for all supported Python versions (git-ignored). See make setup command.

4. The configuration for VSCode (git-ignored). See make vscode command.

    configurationCache.log
    dryrun.log
    launch.json
    settings.json
    targets.log
    tasks.json
   

5. Contains our tooling configuration. See Scripts, configuration.

   📁 vscode/ # (1)!
    coverage.ini
    git-changelog.toml
    pytest.ini
    ruff.toml
    ty.toml
   

   1.  launch.json
       settings.json
       tasks.json
      

6. Documentation sources (Markdown pages). See make docs task.

   📁 .overrides/ # (1)!
   📁 css/ # (2)!
   📁 extensions/ # (3)!
   📁 guide/ # (4)!
   📁 img/ # (5)!
   📁 js/ # (6)!
   📁 reference/ # (7)!
    alternatives.md
    changelog.md
    code-of-conduct.md
    community.md
    contributing.md
    credits.md
    downstream-projects.md
    extensions.md
    getting-help.md
    getting-started.md
    guide.md
    index.md
    installation.md
    introduction.md
    license.md
    logo.svg
    playground.md
    reference.md
    schema-docstrings-options.json
    schema.json
   

   1. Customization of Material for MkDocs' templates.

      📁 partials/ # (1)!
       main.html
      

      1.  comments.html
          path-item.html
         

   2.  custom.css
       material.css
       mkdocstrings.css
      

   3. 📁 built-in/ # (1)!
      📁 official/ # (2)!
      📁 third-party/ # (3)!
       built-in.md
       official.md
       third-party.md
      

      1.  dataclasses.md
          unpack-typeddict.md
         

      2.  autodocstringstyle.md
          inherited-docstrings.md
          public-redundant-aliases.md
          public-wildcard-imports.md
          pydantic.md
          runtime-objects.md
          sphinx.md
          typingdoc.md
          warnings-deprecated.md
         

      3.  docstring-inheritance.md
          fastapi.md
          fieldz.md
          generics.md
          inherited-method-crossrefs.md
          modernized-annotations.md
         

   4. 📁 contributors/ # (1)!
      📁 users/ # (2)!
       contributors.md
       users.md
      

      1.  architecture.md
          commands.md
          setup.md
          workflow.md
         

      2. 📁 how-to/ # (1)!
         📁 recommendations/ # (2)!
          checking.md
          extending.md
          loading.md
          navigating.md
          serializing.md
         

         1.  parse-docstrings.md
             selectively-inspect.md
             set-docstring-styles.md
             set-git-info.md
             support-decorators.md
            

         2.  docstrings.md
             public-apis.md
             python-code.md
            

   5.  favicon.ico
       gha_annotations_1.png
       gha_annotations_2.png
      

   6.  feedback.js
      

   7. 📁 api/ # (1)!
       api.md
       cli.md
       docstrings.md
      

      1. Python API reference, injected with mkdocstrings.

         📁 docstrings/ # (1)!
         📁 models/ # (2)!
          agents.md
          checks.md
          cli.md
          docstrings.md
          exceptions.md
          expressions.md
          extensions.md
          finder.md
          helpers.md
          loaders.md
          loggers.md
          models.md
          serializers.md
         

         1.  models.md
             parsers.md
            

         2.  alias.md
             attribute.md
             class.md
             function.md
             module.md
             type_alias.md
            

7. HTML report for Python code coverage (git-ignored), integrated in the Coverage report page. See make coverage task.

8. 📁 griffecli/ # (1)!
   📁 griffelib/ # (2)!
   

   1. 📁 src/ # (1)!
       pyproject.toml
      

      1. 📁 griffecli/ # (1)!
         

         1. 📁 _internal/ # (1)!
             __init__.py
             __main__.py
            

            1.  __init__.py
                cli.py
               

   2. 📁 src/ # (1)!
       pyproject.toml
      

      1. 📁 griffe/ # (1)!
         

         1. 📁 _internal/ # (1)!
             __init__.py
             __main__.py
            

            1. Our internal API, hidden from users. See Program structure.

               📁 agents/ # (1)!
               📁 docstrings/ # (2)!
               📁 extensions/ # (3)!
                __init__.py
                c3linear.py
                collections.py
                debug.py
                diff.py
                encoders.py
                enumerations.py
                exceptions.py
                expressions.py
                finder.py
                git.py
                importer.py
                loader.py
                logger.py
                merger.py
                mixins.py
                models.py
                py.typed
                stats.py
                tests.py
               

               1. 📁 nodes/ # (1)!
                   __init__.py
                   inspector.py
                   visitor.py
                  

                  1.  __init__.py
                      assignments.py
                      ast.py
                      docstrings.py
                      exports.py
                      imports.py
                      parameters.py
                      runtime.py
                      values.py
                     

               2.  __init__.py
                   auto.py
                   google.py
                   models.py
                   numpy.py
                   parsers.py
                   sphinx.py
                   utils.py
                  

               3.  __init__.py
                   base.py
                   dataclasses.py
                   unpack_typeddict.py
                  

9. Our different scripts. See Scripts, configuration.

    gen_credits.py
    gen_griffe_json.py
    gen_structure_docs.py
    get_version.py
    make
    make.py
   

10. Documentation site, built with make run mkdocs build (git-ignored).

11. Our test suite. See Tests.

    📁 fixtures/
    📁 test_docstrings/ # (1)!
    📁 test_extensions/ # (2)!
     __init__.py
     conftest.py
     helpers.py
     test_api.py
     test_cli.py
     test_diff.py
     test_encoders.py
     test_expressions.py
     test_finder.py
     test_functions.py
     test_git.py
     test_inheritance.py
     test_inspector.py
     test_loader.py
     test_merger.py
     test_mixins.py
     test_models.py
     test_nodes.py
     test_public_api.py
     test_stdlib.py
     test_visitor.py
    

    1.  __init__.py
        conftest.py
        helpers.py
        test_google.py
        test_numpy.py
        test_sphinx.py
        test_warnings.py
       

    2.  __init__.py
        test_base.py
        test_dataclasses.py
        test_unpack_typeddict.py
       

12. The answers file generated by Copier. See Boilerplate.

13. The environment configuration, automatically sourced by direnv. See commands.

14. A dummy makefile, only there for auto-completion. See commands.

15. Our project tasks, written with duty. See Tasks.

16. The build configuration for our docs. See make docs task.

17. The project metadata and production dependencies.

Boilerplate¤

This project's skeleton (the file-tree shown above) is actually generated from a Copier template called copier-uv. When generating the project, Copier asks a series of questions (configured by the template itself), and the answers are used to render the file and directory names, as well as the file contents. Copier also records answers in the .copier-answers.yml file, allowing to update the project with latest changes from the template while reusing previous answers.

To update the project (in order to apply latest changes from the template), we use the following command:

copier update --trust --skip-answered

Scripts, configuration¤

We have a few scripts that let us manage the various maintenance aspects for this project. The entry-point is the make script located in the scripts folder. It doesn't need any dependency to be installed to run. See Management commands for more information.

The make script can also invoke what we call "tasks". Tasks need our development dependencies to be installed to run. These tasks are written in the duties.py file, and the development dependencies are listed in devdeps.txt.

The tools used in tasks have their configuration files stored in the config folder, to unclutter the root of the repository. The tasks take care of calling the tools with the right options to locate their respective configuration files.

Sources¤

Sources are located in the packages/ subfolders, following the src-layout. We use Hatch to build source and wheel distributions, and configure it in pyproject.toml.

Tests¤

Our test suite is located in the tests folder. It is located outside of the sources as to not pollute distributions (it would be very wrong to publish a tests package as part of our distributions, since this name is extremely common), or worse, the public API. The tests folder is however included in our source distributions (.tar.gz), alongside most of our metadata and configuration files. Check out our pyproject.toml files to get the full list of files included in our source distributions for every individual package within the packages/ folder.

The test suite is based on pytest. Test modules reflect our internal API structure, and except for a few test modules that test specific aspects of our API, each test module tests the logic from the corresponding module in the internal API. For example, test_finder.py tests code of the griffe._internal.finder internal module, while test_functions tests our ability to extract correct information from function signatures, statically. The general rule of thumb when writing new tests is to mirror the internal API. If a test touches to many aspects of the loading process, it can be added to the test_loader test module.

Program structure¤

Griffe is split into two pieces: the library and the CLI.

Each of them has an internal API contained within an _internal folder:

• packages/griffelib/src/griffe/_internal for the library,
• packages/griffecli/src/griffecli/_internal for the CLI.

Griffe can be installed in library-only mode, which means that the CLI package from packages/griffecli is not present. Library-only mode can be preferred if the user does not utilize the CLI functionality of Griffe and does not want to incorporate its dependencies.

The top-level packages/griffelib/src/griffe/__init__.py module exposes all the public API available: it always re-exports internal objects from various submodules of griffe._internal and, if the CLI is installed, it re-exports the public API of griffecli as well.

Users then import griffe directly, or import objects from it. If they don't have griffecli installed, they cannot import the CLI-related functionality, such as griffecli.check.

We'll be honest: our code organization is not the most elegant, but it works Have a look at the following module dependency graph, which will basically tell you nothing except that we have a lot of inter-module dependencies. Arrows read as "imports from". The code base is generally pleasant to work with though.

You can zoom and pan all diagrams on this page with mouse inputs.

The following sections are generated automatically by iterating on the modules of our public and internal APIs respectively, and extracting the comment blocks at the top of each module. The comment blocks are addressed to readers of the code (maintainers, contributors), while module docstrings are addressed to users of the API. Module docstrings in our internal API are never written, because our module layout is hidden, and therefore modules aren't part of the public API, so it doesn't make much sense to write "user documentation" in them.

CLI entrypoint¤

griffe.__main__¤

Public API¤

griffe¤

This top-level module imports all public names from the package, and exposes them as public objects. We have tests to make sure no object is forgotten in this list.

Internal API¤

The internal API layout doesn't follow any particular paradigm: we simply organize code in different modules, depending on what the code is used for.

agents¤

These modules contain the different agents that are able to extract data.

inspector.py¤

This module contains our dynamic analysis agent, capable of inspecting modules and objects in memory, at runtime.

nodes¤

These submodules contain utilities for working with AST and object nodes.

assignments.py¤

This module contains utilities for extracting information from assignment nodes.

ast.py¤

This module contains utilities for navigating AST nodes.

docstrings.py¤

This module contains utilities for extracting docstrings from nodes.

exports.py¤

This module contains utilities for extracting exports from __all__ assignments.

imports.py¤

This module contains utilities for working with imports and relative imports.

parameters.py¤

This module contains utilities for extracting information from parameter nodes.

runtime.py¤

This module contains utilities for extracting information from runtime objects.

values.py¤

This module contains utilities for extracting attribute values.

visitor.py¤

This module contains our static analysis agent, capable of parsing and visiting sources, statically.

c3linear.py¤

This module contains a single function, c3linear_merge. The function is generic enough to be in its own module.

• Copyright (c) 2019 Vitaly R. Samigullin
• Adapted from https://github.com/pilosus/c3linear
• Adapted from https://github.com/tristanlatr/pydocspec

collections.py¤

This module contains collection-related classes, which are used throughout the API.

debug.py¤

This module is here to help users report bugs. It provides a function to print environment information, which is called from the public griffe.debug module (when called with python -m griffe.debug) or thanks to the --debug-info CLI flag.

diff.py¤

This module exports "breaking changes" related utilities. The logic here is to iterate on objects and their members recursively, to yield found breaking changes.

The breakage class definitions might sound a bit verbose, but declaring them this way helps with (de)serialization, which we don't use yet, but could use in the future.

docstrings¤

These submodules define models and parsers for docstrings.

auto.py¤

This module defines functions to parse docstrings by guessing their style.

google.py¤

This module defines functions to parse Google-style docstrings into structured data.

models.py¤

This module contains the models for storing docstrings structured data.

numpy.py¤

This module defines functions to parse Numpy-style docstrings into structured data.

Based on https://numpydoc.readthedocs.io/en/latest/format.html, it seems Numpydoc is a superset of RST. Since fully parsing RST is a non-goal of this project, some things are stripped from the Numpydoc specification.

Rejected as non particularly Pythonic or useful as sections:

• See also: this section feels too subjective (specially crafted as a standard for Numpy itself), and there are may ways to reference related items in a docstring, depending on the chosen markup.

Rejected as naturally handled by the user-chosen markup:

• Warnings: this is just markup.
• Notes: again, just markup.
• References: again, just markup.

parsers.py¤

This module imports all the defined parsers and provides a generic function to parse docstrings.

sphinx.py¤

This module defines functions to parse Sphinx docstrings into structured data.

utils.py¤

This module contains utilities for docstrings parsers.

encoders.py¤

This module contains data encoders/serializers and decoders/deserializers. We only support JSON for now, but might want to add more formats in the future.

enumerations.py¤

This module contains all the enumerations of the package.

exceptions.py¤

This module contains all the exceptions specific to Griffe.

expressions.py¤

This module contains the data classes that represent resolvable names and expressions. First we declare data classes for each kind of expression, mostly corresponding to Python's AST nodes. Then we declare builder methods, that iterate AST nodes and build the corresponding data classes, and two utilities _yield and _join to help iterate on expressions. Finally we declare a few public helpers to safely get expressions from AST nodes in different scenarios.

extensions¤

These submodules contain our extension system, as well as built-in extensions.

base.py¤

This module contains the base class for extensions and the functions to load them.

dataclasses.py¤

Built-in extension adding support for dataclasses.

This extension re-creates __init__ methods of dataclasses during static analysis.

unpack_typeddict.py¤

finder.py¤

This module contains the code allowing to find modules.

Note: It might be possible to replace a good part of this module's logic with utilities from importlib (however the util in question is private):

>>> from importlib.util import _find_spec
>>> _find_spec("griffe.agents", _find_spec("griffe", None).submodule_search_locations)
ModuleSpec(
    name='griffe.agents',
    loader=<_frozen_importlib_external.SourceFileLoader object at 0x7fa5f34e8110>,
    origin='/media/data/dev/griffelib/packages/griffe/src/griffe/agents/__init__.py',
    submodule_search_locations=['/media/data/dev/griffelib/packages/griffe/src/griffe/agents'],
)

git.py¤

This module contains Git utilities, used by our load_git function, which in turn is used to load the API for different snapshots of a Git repository and find breaking changes between them.

importer.py¤

This module contains utilities to dynamically import objects. These utilities are used by our Inspector to dynamically import objects specified as Python paths, like package.module.Class.method.

loader.py¤

This module contains all the logic for loading API data from sources or compiled modules.

logger.py¤

This module contains the logger used throughout Griffe. The logger is actually a wrapper around the standard Python logger. We wrap it so that it is easier for other downstream libraries to patch it. For example, mkdocstrings-python patches the logger to relocate it as a child of mkdocs.plugins so that it fits in the MkDocs logging configuration.

We use a single, global logger because our public API is exposed in a single module, griffe. Extensions however should use their own logger, which is why we provide the get_logger function.

merger.py¤

This module contains utilities to merge stubs data and concrete data.

mixins.py¤

This module contains some mixins classes that hold shared methods of the different kinds of objects, and aliases.

models.py¤

This module contains our models definitions, to represent Python objects (and other aspects of Python APIs)... in Python.

stats.py¤

This module contains utilities to compute loading statistics, like time spent visiting modules statically or dynamically.

tests.py¤

This module contains helpers. They simplify programmatic use of Griffe, for example to load data from strings or to create temporary packages. They are particularly useful for our own tests suite.